figure table

# Use of Equivalent Relative Utility (ERU) to Evaluate Artificial
Intelligence-Enabled Rule-Out Devices

Kwok Lung Fan  U.S. Food and Drug Administration  Department of Physics,
University of Maryland, College Park  Yee Lam Elim Thompson  U.S. Food and
Drug Administration  Weijie Chen  U.S. Food and Drug Administration  Craig K.
Abbey  Department of Psychological and Brain Sciences, UC Santa Barbara  Frank
W Samuelson  U.S. Food and Drug Administration

###### Abstract

Objective: We investigated the use of equivalent relative utility (ERU) to
evaluate the effectiveness of artificial intelligence (AI)-enabled rule-out
devices that use AI to identify and autonomously remove non-cancer patient
images from radiologist review in screening mammography.

Approach: We reviewed two performance metrics that can be used to compare the
diagnostic performance between the radiologist-with-rule-out-device and
radiologist-without-device workflows: positive/negative predictive values
(PPV/NPV) and equivalent relative utility (ERU). To demonstrate the use of the
two evaluation metrics, we applied both methods to a recent US-based study
that reported an improved performance of the radiologist-with-device workflow
compared to the one without the device by retrospectively applying their AI
algorithm to a large mammography dataset. We further applied the ERU method to
a European study utilizing their reported recall rates and cancer detection
rates at different thresholds of their AI algorithm to compare the potential
utility among different thresholds.

Results: For the study using US data, neither the PPV/NPV nor the ERU method
can conclude a significant improvement in diagnostic performance for any of
the algorithm thresholds reported. For the study using European data, ERU
values at lower AI thresholds are found to be higher than that at a higher
threshold because more false-negative cases would be ruled-out at higher
threshold, reducing the overall diagnostic performance.

Conclusions: Both PPV/NPV and ERU methods can be used to compare the
diagnostic performance between the radiologist-with-device workflow and that
without. One limitation of the ERU method is the need to measure the baseline,
standard-of-care relative utility (RU) value for mammography screening in the
US. Once the baseline value is known, the ERU method can be applied to large
US datasets without knowing the true prevalence of the dataset.

###### keywords:

rule-out, mammography screening, semi-autonomous AI/ML, equivalent relative
utility, evaluation

*Frank W Samuelson<EMAIL_ADDRESS>
##  1 Introduction

Recent advancements in artificial intelligence (AI) using deep learning and
convolution neural networks (CNN) have revolutionized the field of computer
vision for medical imaging. Currently most AI-enabled Computer-Assisted
Detection/Diagnosis (CAD) devices authorized by the U.S. Food and Drug
Administration (FDA) are intended to be used as a second reader or concurrent
aid to radiologists by detecting regions of interests in an image, diagnosing
patients with specific disease conditions based on their images, or triaging
images of patients with time-sensitive diseases. Guidance documents have been
published on the evaluation of such devices in preparation for a regulatory
submission [1, 2]. All these devices require the radiologist review the
patient images and make the final decision. As the AI standalone performance
improves, we anticipate that in the future, a new type of device –which we are
terming “rule-out device”– may autonomously remove patient images deemed free
of disease by the algorithm from the radiologist’s reading queue. Figure 1
compares the standard-of-care workflow without any AI algorithm, a workflow in
which the AI is used as an aided device, and a workflow in which the AI is
used as a rule-out device in which a subset of patient images are autonomously
ruled-out from follow-up exams. In this work, we focus on the comparison
between the radiologist-with-rule-out-device (the bottom workflow in Fig. 1)
and radiologist-without-device workflows (the top workflow in Fig. 1) because
the evaluation of the workflow with an AI-enabled aiding device is well-
established [1, 2].

Figure 1: Comparison between the standard-of-care workflow without AI
algorithms (top), a workflow in which the AI is used to assist radiologists in
decision making (middle), and a workflow in which the AI is used as a rule-out
device where AI-negative patient cases are autonomously excluded from the
radiologist’s reading list (bottom). This work focuses on the comparison of
the workflow with AI as a rule-out device (bottom) against the workflow
without AI (top).

Mammography screening has been shown to reduce breast cancer morbidity and
mortality in randomized trials [3, 4]. The U.S. Preventive Services Task Force
recommends biennial screening mammography for women starting at age 40 instead
of 50 [5], which increases the amount of screening mammograms to be reviewed
by radiologists every year. Due to a shortage of radiologists [6, 7], numerous
studies have investigated the potential application of AI to rule out images
from radiologist’s review in screening mammography [8, 9, 10, 11, 12]. Because
the prevalence of cancer in the mammography screening populatin is low ($<$1%
[13]), most screening mammograms are negative. Therefore, such an rule-out
device can greatly reduce the workload of a radiologist. However, due to the
lack of human supervision on the AI-negative patient cases, a future rule-out
device has a different benefit-risk profile in terms of the overall
radiologist-with-device diagnostic performance and requires different
evaluation methods than currently available aided devices.

Rule-out devices have the potential to both positively and negatively impact
overall screening performance. The positive impact comes from ruling out
patient images that do not contain cancer that would otherwise be recalled
(i.e. reducing false-positives in the radiologist-without-device workflow),
and the negative impact comes from ruling out patient images that contain
cancer and would otherwise have been recalled (reducing true-positives in the
radiologist-without-device workflow). Assessing these impacts requires knowing
more than just the performance of the rule-out device; human performance in
the ruled-out images is needed as well. If the radiologist had a chance to
review AI-false-negative patient images removed by the device, he/she may also
misdiagnose some of these signal-present cases as normal findings. Including
an rule-out device in the workflow does not alter the outcome of these cases
labeled as negative by both AI and the radiologist. Therefore, one may argue
that a more relevant risk factor is associated with the signal-present cases
removed by the device that would have been diagnosed as positive by the
radiologist if they remained in the reading queue. This risk will be lower
than the AI-alone false-negative rate, especially when the AI score and
radiologists’ ratings are expected to be positively correlated for signal-
present cases.

A principled evaluation of diagnostic performance of rule-out devices should
compare population screening performance with and without the device using
accepted performance measures, such as area under the receiver operating
characteristic curve (AUROC), and appropriate modeling of random effects (e.g.
multi-reader, multi-case study [14]). Such a study allows testing for
performance effects across the population of readers and cases. However, the
implementation of a prospective study of this sort in breast-cancer screening
may be prohibitively expensive and time-consuming given the low prevalence
(below 1%) of disease [13]. This motivates an alternative approach that
performs a retrospective “simulated” study in which an rule-out device
analyzes mammograms from large-scale reader studies. All AI-negative cases are
assigned the lowest possible rating for suspicion of disease and are also
considered removed from the reading workload. For example, Rodriguez-Ruiz et
al. [15] found that retrospectively applying their AI algorithm as a rule-out
device to analyze images from nine different reader studies maintained a non-
inferior AUROC compared to the radiologist-without-device scenario, while
reducing 47% of radiologist workload. It should be noted that this
retrospective approach assumes that excluding AI-negative cases from the
workload does not alter the readers’ responses to the remaining images. At
this point, it is not known whether reducing the workload and altering the
distribution of cases will affect reader performance. In particular, for the
AUROC endpoint, maintaining the relative ordering of cases is critical; if a
reader ranks hypothetical Case A as more suspicious than Case B, and Case B as
more suspicious than Case C, then removing Case B from the reader’s queue
should not change the relative ranking that a reader would give Case A and
Case C.

In many observational clinical studies, the only information collected from
radiologists is a binary decision whether to recall each patient. Such studies
yield only a single average pair of diagnostic metrics, such as sensitivity
and specificity when the ground truth is known for both radiologist-positive
and negative patients, or cancer detection rate (number of true positive
divided by total number of patients) and recall rate (number of test positive
divided by total number of patients) when only ground truth for radiologist-
positive patients are known. There is not sufficient information to construct
ROC curves or calculate AUROC. However even with binary data we can “simulate”
studies to estimate the diagnostic performance of readers using rule-out
devices by labeling all AI-negative cases as negative. As in ROC studies we
assume that the radiologist’s ordering of cases remains the same, but we also
assume that the reader does not change their decision threshold or which cases
they call positive or negative even when the rule-out device is implemented.
This may not be a realistic assumption, but the sensitivity and specificity of
the radiologist-with-rule-out-device scenario as obtained from a simulated
study is expected to lie on the ROC curve of the radiologist-with-rule-out-
device scenario.

As an example of such a retrospective simulated study with binary data, Yala
et al. [9] concluded that, when their AI algorithm rules out 19.3% of
mammograms, the radiologist-with-rule-out-device workflow has a significantly
improved specificity (from 93.5% to 94.2%) while observed value of sensitivity
dropped from 90.6% to 90.1% compared to the radiologist-without-device
workflow. A similar European study, where double reading is the standard of
care for breast cancer screening, investigated 11 theoretical workflow
scenarios. When their AI algorithm rules out 50% of cases, they reported that
the recall rate dropped from 3.2% to 2.6% and the cancer detection rate
dropped from 6.2 per thousand to 6.1 per thousand when compared to the
standard-of-care workflow [11]. Laruitzen et al. [12] performed a similar
retrospective study using another dataset from Europe. The AI model is the
same as Larsen et al., but with an additional rule-in component, and a similar
result was reported.

Evaluating the radiologist-with-rule-out-device diagnostic performance solely
based on the change in sensitivity and specificity (or cancer detection and
recall rates) before and after the rule-out device is included in the workflow
is inevitably ambiguous. In “simulated” studies such as those mentioned above,
all positive rates will decrease with the addition of the rule-out device, and
therefore sensitivity _must_ decrease, and specificity _must_ increase.
Likewise both recall rate and detection rate will decrease. Knowing this
trend, one cannot conclude improved performance solely based on the
radiologist-with-rule-out-device sensitivity and specificity in these
retrospective analyses. Therefore, there is a need of an alternative
evaluation method to compare the diagnostic performance of the standard of
care (radiologist-without-rule-out-device) workflow to that of radiologist-
with-rule-out-device workflow.

In Section 2, we review two methods that can be used to compare diagnostic
performance of the two workflows (radiologist with and without an rule-out
device) based on two points in the ROC space. Section 2.1 describes using
positive and negative predictive values, and section 2.2 describes equivalent
relative utility (ERU). Section 3 applies these two methods to evaluate the
performance of two rule-out devices published in literature. The strength and
limitations of using ERU as the evaluation metric for rule-out devices are
discussed in Section 4, followed by a conclusion in Section 5.

##  2 Methods

In this section, we review two general evaluation frameworks (PPV/NPV and ERU)
which can be applied to compare the diagnostic performance between
radiologist-with-rule-out-device and radiologist-without-device workflows. We
define an alternative diagnostic test A as the radiologist-with-rule-out-
device workflow and a reference diagnostic test B as the radiologist-without-
device workflow.

###  2.1 Positive and negative predictive value lines

In general, if test A has a higher sensitivity and higher specificity than
test B, then the diagnostic performance of test A is superior to that of test
B. Graphically in the ROC space, this is equivalent to a point, representing
the true-positive rate ($R_{\text{TP},A}$) and false-positive rate
($R_{\text{FP},A}$) of test A, located in the blue upper-left region in Fig. 2
compared to the ($R_{\text{FP},B}$, $R_{\text{TP},B}$) point of test B (blue
star in Fig. 2).

Figure 2: Regions in the ROC space where the diagnostic performance of a
hypothetical alternative diagnostics test A (orange dot) is superior to that
of a hypothetical reference test B (blue star). Here, test A refers to the
radiologist-with-rule-out-device workflow, and test B refers to the
radiologist-without-device workflow. If the true-positive and false-positive
rates of test A falls in the blue-shaded region, test A has a better
sensitivity (Se) and specificity (Sp), as well as a higher positive predictive
value (PPV) and negative predictive value (NPV), than test B. If they fall in
the purple-shaded region below (or on the right side of) the blue area, test A
has a lower sensitivity (or specificity) but a higher PPV (or NPV). If the
orange dot is outside of the purple region, PPV/NPV lines cannot be used to
decide whether test A has a more superior diagnostic performance than test B.

Furthermore, Biggerstaff [16] demonstrated that test A may have a higher
positive predictive value (PPV) and negative predictive value (NPV) than test
B, even if test A has a lower sensitivity or specificity. By definition, PPV
is the probability of having a truly diseased case given that the test result
is positive i.e. $P(D|\texttt{+})$. Via Bayes’ theorem, PPV is directly
related to the positive likelihood ratio $\rho_{\texttt{+}}$ [16].

$\displaystyle PPV\equiv
P(D|\texttt{+})=\frac{\rho_{\texttt{+}}\omega}{\rho_{\texttt{+}}\omega+1}$ (1)

where $\omega=\pi/(1-\pi)$, and $\pi$ is the prevalence of the disease of
interest. Because $\rho_{\texttt{+}}$ is equal to the true positive rate
divided by the false positive rate ($R_{\text{TP}}/R_{\text{FP}}$),
$\rho_{\texttt{+}}$ represents the slope of a line connecting the
($R_{\text{FP}},R_{TP}$) point of a diagnostic test to (0,0) in the ROC space
(the red dash-dotted line in Fig. 2). Hence, all
($R_{\text{FP}},R_{\text{TP}}$) points along this line have a constant PPV.
Since Eq. 1 monotonically increases with $\rho_{\texttt{+}}$, all points on
the left of the line have a higher PPV. Similarly, Biggerstaff [16] also
related NPV to the negative likelihood ratio $\rho_{\texttt{-}}$.

$\displaystyle NPV\equiv P(ND|\texttt{-})=\frac{1}{\rho_{\texttt{-}}\omega+1}$
(2)

where $\rho_{\texttt{-}}=(1-R_{\text{TP}})/(1-R_{\text{FP}})$ is, by
definition, the slope of a line connecting the operating point of a diagnostic
test to (1,1) (the olive-color dashed line in Fig. 2). Since Eq. 2
monotonically increases as $\rho_{\texttt{-}}$ decreases, all points above the
line have a higher NPV. If the ($R_{\text{FP,A}},R_{\text{TP,A}}$) point of
diagnostic test A is located in the purple region in Fig. 2, then test A has a
superior diagnostic performance in terms of PPV and NPV than the reference
test B. In this scenario Egan [17, Section 2.6.3] demonstrated that test A
must exist on a higher ROC curve than test B if those ROC curves are proper
with a monotonically decreasing slope along the curve from (0,0) to (1,1).

When comparing the diagnostic performance of the alternative “simulated”
radiologist-with-rule-out-device workflow to that of the reference
radiologist-without-device workflow, the simulated operating point of the
radiologist with an rule-out device is expected to have a lower false positive
rate because the device helps exclude signal-absent cases that would have been
mistakenly diagnosed as positive by the radiologist. The radiologist-with-
rule-out-device operating point is guaranteed to have a greater PPV, if the
rule-out device performs better than guessing,
$R_{\text{TP,d}}-R_{\text{FP,d}}>0$. To determine whether the radiologist-
with-rule-out-device workflow also results in a greater NPV, it is necessary
to have the true negative rate and false negative rate. That means we need to
have the ground truth of the negative cases by both the device and
radiologist. This is often very difficult and costly since patients of
negative diagnoses by the radiologists usually do not get follow-up exams.
Therefore, the PPV/NPV method may be less practical in evaluating the combined
diagnostic performance of an rule-out devices for cancer screening.

As a side note, if we were to evaluate a rule-in device that is intended to
autonomously recall AI-positive patients, the PPV/NPV method would be more
practical than using sensitivity and specificity. Since the
($R_{\text{FP}},R_{\text{TP}}$) operating point of a radiologist with a rule-
in device is expected to have a higher false positive rate, the NPV of the
radiologist-with-rule-in-device workflow is guaranteed to be greater than that
of the radiologist-without-device workflow. To determine whether the
radiologist-with-rule-in-device workflow also results in a greater PPV, we
only need to know the ground truth for the positive cases labeled as positive
by both the rule-in device and the radiologist, which is usually less
burdensome to obtain. Therefore, the PPV/NPV method could readily be used to
evaluate a rule-in device.

###  2.2 Relative utility

Abbey et al. [18] showed that it is possible to apply utility theory to
compare the performance of two diagnostic tests, and similar utility-based
approaches were developed by Baker et al. [19] to evaluate risk prediction
models. The utility of a diagnostic test could be expressed as

$\displaystyle
U=U_{\text{TP}}P(\text{TP})+U_{\text{FP}}P(\text{FP})+U_{\text{TN}}P(\text{TN})+U_{\text{FN}}P(\text{FN})$
(3)

where $P$ represents the probability of the corresponding outcome (true-
positive TP, false-positive FP, true-negative TN, and false-negative FN), and
$U$ represents the utility of each outcome. The probability of each outcome
depends on both the sensitivity and specificity as well as the prevalence of
the disease. The probability of each outcome could be expressed by
$P(\text{TP})=R_{\text{TP}}\pi$, $P(\text{FN})=(1-R_{\text{TP}})\pi$,
$P(\text{TN})=(1-R_{\text{FP}})(1-\pi)$, and
$P(\text{FP})=R_{\text{FP}}(1-\pi)$. Rearranging the terms in Eq. 3 gives the
following expression for iso-utility.

$\displaystyle
R_{\text{TP}}=\frac{(U_{\text{TN}}-U_{\text{FP}})(1-\pi)}{(U_{\text{TP}}-U_{\text{FN}})\pi}R_{\text{FP}}+\frac{U-U_{\text{FN}}\pi-
U_{\text{TN}}(1-\pi)}{(U_{\text{TP}}-U_{\text{FN}})\pi}$ (4)

Assuming the utility of each outcome is independent of the rate of each
outcome, the resulting iso-utility curve is a straight line with a positive
slope (assuming the correct decision have greater utiltiy than incorrect
decision) in the ROC space [18]. Moreover, the expected utility $U$ is maximum
when the tangent of an operating point along a ROC curve is equal to the slope
of the iso-utility line.

Wagner et al. [20] defined relative utility $U_{\text{Rel}}$ as the utility
difference between the correct and incorrect diagnoses of a signal-present
patient divided by the utility difference between the correct and incorrect
diagnoses of a signal-absent patient.

$\displaystyle
U_{\text{Rel}}=\frac{(U_{\text{TP}}-U_{\text{FN}})}{(U_{\text{TN}}-U_{\text{FP}})}$
(5)

Since the utility of correct decision is assumed to be higher than incorrect
decision, the numerator and denominator should both be positive. Relative
utility could be interpreted as the importance or trade-off of identifying
signal-present patients compared to identifying signal-absent patients. Hence,
for cancer screening, relative utility is usually much greater than 1. Based
on Eq. 4, the slope of the tangent line at the optimal point on an ROC curve
is inversely proportional to the relative utility.

$\displaystyle\text{slope}_{\text{opt}}=\frac{Q_{\pi}}{U_{\text{Rel}}}$ (6)

where $Q_{\pi}\equiv(1-\pi)/\pi$. With Eq. 6, the iso-utility equation (Eq. 4)
can be re-expressed as

$\displaystyle R_{\text{TP}}=\frac{Q_{\pi}}{U_{\text{Rel}}}R_{\text{FP}}+c$
(7)

where $c$ is a constant.

The value of relative utility is often unknown. However, as Abbey et al. [21]
argued, assuming that an experienced radiologist operates at the optimal
point, one could estimate the relative utility from the tangent line at the
radiologist’s operating point along the ROC curve. As an illustrative example,
Fig. 3 shows a tangent line at a hypothetical radiologist-without-device
operating point. Using a large dataset from the DMIST study [22], Abbey et al.
estimated a relative utility of 162 for a standard-of-care screening
mammography workflow (without any AI-enabled aiding or rule-out devices) in
the US using the tangent of the ROC curve at the radiologist’s operating
point. This means that the utility of identifying a truly signal-present
patient is 162 times more important than that of identifying a signal-absent
patient. In this work, 162 is used as the baseline value of relative utility
for screening mammography in the radiologist-without-device workflow in the
US.

Figure 3: The blue line represents a hypothetical ROC curve in a radiologist-
without-device workflow. The gray dashed lines represent different iso-utility
lines. The slope of the iso-utility lines is $Q_{\pi}/U_{\text{Rel}}$. Iso-
utility lines with a higher y-intercept represent a higher expected utility.
The blue star shows the optimal point along the ROC curve, and the tangent at
the optimal point (maximum utility) is equal to the slope of the iso-utility
line.

Using relative utility, one can determine whether an alternative diagnostic
test A has a better diagnostic performance than a reference diagnostic test B
with an iso-utility line. If test A has an operating point in the ROC space
(orange dot in Fig. 4) that is above the iso-utility line of test B (green
dashed line in Fig. 4), then the expected utility of test A is higher since
its operating point lays in a better iso-utility line. In other words, test A
has a higher expected utility if the slope of the line connecting the
operating points of test A and B (Eq. 8) is smaller than the slope of the iso-
utility line of test B. The green-shaded region in Fig. 4 shows the region
where an alternative diagnostic test A is superior to the reference test B in
terms of expected utility.

$\displaystyle\text{slope}(A,B)=\frac{R_{\text{TP,A}}-R_{\text{TP,B}}}{R_{\text{FP,A}}-R_{\text{FP,B}}}$
(8)

Figure 4: Regions where a simulated radiologist with an rule-out device
(alternative test A) has a superior diagnostic performance compared to that of
a radiologist-alone workflow (reference test B). The blue-shaded region
indicates the alternative test A has a better sensitivity (Se) and specificity
(Sp) than the reference test B. The purple-shaded region indicates that test A
has either a lower sensitivity or specificity but a higher positive predictive
value and a higher negative predictive value. The green-shaded region
indicates test A has a higher utility than test B. The relative utility along
the green dashed line was estimated to be 162 by Abbey et al. [21] for US
screening mammography. The operating points of test A (orange dot) and test B
(blue star) as well as the disease prevalence are based on the paper by Yala
et al. [9].

Equivalent to comparing the slope between point A and B to that of an iso-
utility line, Abbey et al. [18] defined an equivalent relative utility (ERU)
between A and B as

$\displaystyle\text{ERU}(A,B)=\frac{Q_{\pi}}{\text{slope}(A,B)}$ (9)

which is the relative utility needed for test A and B to be on the same iso-
utility line. If the ERU between test A and B is higher than the relative
utility of test B (assuming same prevalence), then test A is superior to test
B since it has a higher expected utility. It should be noted that this
criterion only works when the slope between test A and test B is positive; any
non-positive slope implies that either both tests have the same sensitivity
(slope of 0) or the sensitivity and specificity of one test are both greater
than the other test (negative slope). Using ERU to compare the diagnostic
performance of tests A and B, instead of the slopes, comes with two
advantages.

First, ERU can be estimated from cancer-detection rate $R_{D}$ and recall rate
$R_{R}$ which are often easier to obtain in real-world studies. As discussed
in Abbey et al. [18], true positive and false positive rates are related to
cancer-detection rate and recall rate.

$\displaystyle R_{D}=\pi R_{\text{TP}}$ (10) $\displaystyle R_{R}=\pi
R_{\text{TP}}+(1-\pi)R_{\text{FP}}$ (11)

Substituting them into Eqs. 8 and 9, one can express ERU as a function of
$R_{D}$ and $R_{R}$ of tests A and B.

$\displaystyle\text{ERU}=\frac{R_{\text{R,A}}-R_{\text{R,B}}}{R_{\text{D,A}}-R_{\text{D,B}}}-1$
(12)

Therefore, ERU can be estimated by measuring the difference between the recall
rates of the two tests divided by the difference between their cancer-
detection rates.

Second, ERU gives an intuitive sense of the trade-off between test A and test
B. Assume test A has a higher specificity (lower false positive rate) but a
lower sensitivity than test B such that the ERU is 200. This means that for
every positive case test A misidentifies relative to test B, it will correctly
identify 200 negative cases. To see this, we rearrange the term in Eq. 12

$\displaystyle\text{ERU}=\frac{R_{\text{FR,A}}-R_{\text{FR,B}}}{R_{\text{D,A}}-R_{\text{D,B}}}$
(13)

where $R_{\text{FR}}=(1-\pi)R_{\text{FP}}=R_{\text{R}}-R_{\text{D}}$ is the
false recall rate. We could see that the ERU is the ratio between the drop in
false recall rate (gain in true negatives) and the drop in cancer detection
rate (loss in true positives). If this trade-off is better than test B optimal
trade-off (i.e. ERU $>$ $U_{\text{Rel}}$), then test A is superior to test B.

As pointed out by Abbey et al. [18], the uncertainty of ERU can be large
especially when the prevalence is low. Therefore, it is more useful to compare
the confidence interval to the relative utility baseline (162). Abbey et al.
provided a formula to compute the Bayesian confidence interval for two
independent tests. However, in comparing the radiologist-with-rule-out-device
and radiologist-without-device workflow using retrospectively simulated
studies, the two operating points are highly correlated, and the formula is
not applicable. In our work, we use 5000 bootstrap samples to estimate the 95%
confidence interval of ERU. In addition, we use the same bootstrap samples to
compute the chance of the radiologist-with-rule-out-device workflow having a
higher PPV and NPV, and the chance of the radiologist-with-rule-out-device
workflow having a ERU$>$162\. Note that this bootstrap sampling did not
account for the nesting of cases within readers or reader variability and may
therefore underestimate the confidence intervals.

##  3 Retrospective analyses of published rule-out models

In this section, we apply the PPV/NPV and ERU$>U_{\text{Rel}}$ criteria on two
retrospectively simulated studies. First is the analysis by Yala et al. [9] in
which the potential impact on sensitivity, specificity, and reading workload
due to an rule-out device was investigated for screening US mammography. We
use the relative utility of 162 estimated by Abbey et al. [21] from a US-
enabled DMIST study [22] as our baseline value. Second, we compute the ERU of
a European study by Larsen et al. [11] which reported the cancer-detection
rate and recall rate.

###  3.1 US mammography screening

A recent study done by Yala et al. [9] retrospectively applied a deep-learning
AI algorithm as a potential rule-out device on a large US mammography dataset.
The rule-out device was trained using 212,272 mammograms from 56,831 patients
and was evaluated using 26,540 mammograms from 7,176 patients with a
prevalence of 0.7% (191/26,540). The device’s ROC curve was reported with an
AUROC of 0.85 [95% CI: 0.80, 0.85]. All data was retrospectively collected at
a large tertiary academic medical center in the US in which one radiologist
read (without any AI-enabled aiding or rule-out devices) and determined
whether a patient should be followed up. The radiologist-without-device
workflow has a sensitivity of 90.6% [95% CI: 86.6%, 94.7%] and a specificity
of 93.5% [95% CI: 86.0%, 93.9%]. The authors then retrospectively simulated a
radiologist-with-rule-out-device workflow in which a patient would only get a
follow-up exam when the patient was diagnosed as positive by the radiologist
and also labeled as positive by the AI algorithm. The radiologist-with-rule-
out-device workflow would have a sensitivity of 90.1% [95% CI: 86.0%, 94.3%]
and a specificity of 94.2% [95% CI: 94.0%, 94.6%], and the AI algorithm would
have excluded 19.3% of cases. The authors also provided the sensitivity and
specificity values of the radiologist-with-rule-out-device workflow as a
function of the fraction of excluded mammograms in Table E2 of the appendix.

Using the reported values of prevalence, sensitivity, and specificity for the
radiologist-without-device workflow, we can calculate PPV (Eq. 1) and NPV (Eq.
2) with values of 9.18% and 99.9% respectively. Figure 4 shows the lines of
constant PPV and NPV at the sensitivity and specificity of the radiologist-
without-device workflow. The purple area represents the sensitivity and
specificity region in which radiologist-with-rule-out-device scenario is
superior than the radiologist-without-device workflow. Since the reported
sensitivity and specificity of the radiologist-with-rule-out-device workflow
is located outside the purple region, we cannot conclude based on the method
of PPV/NPV lines that the diagnostic performance of the radiologist-with-rule-
out-device workflow is superior to that of the radiologist-without-device
workflow. Using bootstrap sampling, we estimate the chance of the radiologist-
without-rule-out-device workflow having a greater PPV/NPV value is 36%.

Alternatively, the ERU method with the baseline relative utility of 162 may be
used to compare the diagnostic performance of the two workflows (radiologist-
with-rule-out-device vs radiologist-without-device). With the prevalence,
sensitivity, and specificity of the two workflows, the ERU between the two
scenarios (Eqs. 8 and 9) is 189 [95% CI: 56, inf]. Since the ERU is above 162
(dashed line in Fig. 4), the radiologist-with-rule-out-device operating point
is within the region of higher utility (green area in Fig. 4). However,
because the lower bound of the 95% confidence interval is below 162, one still
cannot conclude with statistical certainty that their potential rule-out
device at the operating threshold provides a significantly better utility
based on the ERU method. The chance of the radiologist-with-rule-out-device
workflow having a greater utility is 72.1%.

Using other AI thresholds reported in Table E2 (Appendix of Yala et al. [9]),
the ERU values, their 95% confidence intervals, and the chances that the
radiologist-with-rule-out-device workflow have greater utility and PPV/NPV are
summarized in Table 1. The ERU values and their 95% confidence intervals are
plotted as a function of the percentage of cases excluded by the potential
rule-out device in Fig. 5. Based on the available operating thresholds, none
of them achieve an ERU higher than 162. Therefore, it remains inconclusive
that the radiologist-with-rule-out-device workflow has a significantly higher
expected utility.

Rule out patient(%) | ERU | ERU(95% CI) | ERU>162(%) |  Greater PPV and NPV(%)
---|---|---|---|---
10% | 123.0 | ( 36,inf ) | 36.4 | 36.4
20% | 114.5 | ( 44.2,inf ) | 41.6 | 13.2
30% | 70.8 | ( 35.6,352 ) | 11.7 | 0.6
40% | 55.6 | ( 32.3,140.5 ) | 1.9 | 0
50% | 44.2 | ( 28.8,83 ) | 0 | 0
60% | 35.8 | ( 24.6,58.5 ) | 0 | 0
70% | 36.5 | ( 26.2,55.8 ) | 0 | 0
80% | 23.5 | ( 18,32.2 ) | 0 | 0
90% | 20.2 | ( 16.3,26.4 ) | 0 | 0
Table 1: Table summarizing the ERU values, their 95% confidence intervals, and
the probabilities that the radiologist-with-rule-out-device workflow have a
greater utility and PPV/NPV as a function of the AI thresholds provided in
Table E2 of the Appendix of Yala et al. [9]. We used 5000 bootstrap samples to
calculate the probabilities and the confidence intervals. The columns are
(from left to right) percentage of total patients excluded by the AI algorithm
as a potential rule-out device, the ERU between radiologist-with-rule-out-
device and radiologist-without-device workflows, 95% confidence interval of
ERU, the chance of ERU $>$ 162, the chance of the radiologist-with-rule-out-
device workflow has a greater PPV and NPV than radiologist-without-device
workflow. Figure 5: Equivalent relative utility (ERU) as a function of the
percentage of patients excluded by the AI algorithm as a potential rule-out
device based on the AI thresholds provided in Table E2 in the Appendix of Yala
et al. [9]. The blue line shows the estimated ERUs for different AI
thresholds, and the vertical error bars represent the 95% confidence
intervals. The orange dashed line represents the baseline relative utility of
162 from Abbey et al. [21].

It should be noted that, due to the limitation of a retrospectively
“simulated” radiologist-with-rule-out-device workflow, this workflow may have
a higher expected utility than radiologist-without-device workflow even though
the estimated ERU is below the baseline value. This is because, when an rule-
out device is deployed in a real-world clinic, radiologists are expected to
adjust their decision thresholds due to a change in prevalence [23]. The
resulting radiologist-with-rule-out-device operating point may have a higher
utility (green region in Fig. 4), PPV and NPV (purple region), and/or even
higher sensitivity and specificity (blue region). However a “simulated”
radiologist-with-rule-out-device workflow, where patients are ruled-out
retrospectively from the reading list, must result in an operating point with
a lower true-positive and false-positive rates compared to the operating point
of the radiologist-without-device workflow. We expect these true- and false-
positive rates to decrease more when the rule-out fraction increases (i.e.
when more patients are ruled-out by the device). However, this operating point
from the “simulated”, radiologist-with-rule-out-device workflow is expected to
land on the radiologist-with-rule-out-device ROC curve. Thus, the utility from
the “simulated”, radiologist-with-rule-out-device workflow can serve as a
lower bound of the expected utility even when the radiologists adjust their
operating points due to the change in prevalence in their worklist. In other
words, the ERU$>U_{\text{Rel}}$ based on a retrospective, simulated analysis
is a sufficient condition but not a necessary condition for the radiologist-
with-rule-out-device workflow to have a higher expected utility than the
radiologist-without-device scenario.

###  3.2 European mammography screening

Several studies found that European mammography radiologists have a lower
recall rate compared to the US radiologists [24, 25, 26]. In addition, double
reading for mammography screening is a common clinical practice in Europe [27]
instead of single reading in the US; mammograms in Europe are reviewed by two
radiologists who would have to reach a consensus if their initial decisions
disagree. Therefore, the ROC curves are expected to differ significantly
between radiologists in Europe and the US, and the trade-off between signal-
present and signal-absent patients is expected to be different. Hence, the
baseline relative utility of 162 is not applicable to European studies and
would need to be measured in future studies. However, despite the lack of the
baseline relative utility, one can still apply the ERU method on European
studies such as that by Larsen et al. [11] to measure the trade-off obtained
by a potential rule-out device.

Larsen et al. [11] investigated the clinical impacts that an rule-out device
can bring to the standard-of-care double-reading mammography screening. Their
AI-enabled lesion detection system developed for mammography screening (with
an AUROC of 0.84 [95% CI: 0.82, 0.86] [15]) is used to retrospectively analyze
122,969 mammograms from 47,877 patients in Norway. From the standard-of-care
(double-reading) workflow, the recall rate and cancer detection rates are 3.2%
and 0.61% respectively. The authors then defined 11 hypothetical workflow
scenarios using the AI score and the ratings from the two radiologists and
calculated their recall rates, cancer detection rates, and sensitivities. Due
to the difference in single vs double reading between the US and Europe, we
focus on Scenarios 3-5 (defined in Table 1 in Larsen et al. [11]) in which the
AI score is hypothetically used to pre-screen cases before they enter
radiologists’ worklist. These three workflows correspond to the potential
rule-out device ruling out 30%, 50%, and 70% of the total patients, and their
estimated recall rates (and cancer detection rates) are 2.6% (0.6%), 2.1%
(0.58%), and 1.2% (0.53%) respectively (Table 2 in Larsen et al.).

Although the measured sensitivities for all scenarios were reported (Fig. 2 in
Larsen et al. [11]), the authors did not provide the estimated specificity.
With only sensitivity, there is not enough information to provide insights on
whether the radiologist-with-rule-out-device workflow have better diagnostic
performance than the radiologist-without-device workflow in terms of PPV and
NPV.

However, with the reported recall rates and cancer detection rates, one can
apply the ERU method from Eq. 12. Summary of the ERU values and their 95%
confidence intervals of the three scenarios are shown in Table 2 and Fig. 6.
Comparing the three hypothetical scenarios, Scenario 5 (with the most number
of excluded patients) has a lower utility likely because of a higher false-
negative rate by the potential rule-out device at that threshold. It should be
noted that the ERU values from European studies are generally much lower than
those from the US since a double-reading workflow recalls much fewer patients
for follow-up exams.

Figure 6: Equivalent relative utility vs percentage of patients ruled out by the AI with data from Larsen et al. [11]. The blue line shows equivalent relative utility for different percentages of patients ruled out (AI decision threshold). The vertical error bar represents the 95% confidence interval. Scenario in Larsen et al. |  Rule out patient(%) | ERU | ERU (95% CI)
---|---|---|---
3 | 30 | 39.9 | ( 26.2, 73.6 )
4 | 50 | 39.4 | ( 28.9, 58.7 )
5 | 70 | 23.8 | ( 19.8, 29.6 )
Table 2: Table summarizing the result of applying ERU methods to Scenarios 3-5
defined in Larsen et al. [11]. For each scenario (row), we give the percentage
of total patients ruled out by the potential rule-out device, the ERU between
the radiologist-without-device (double-reading) and radiologist-with-rule-out-
device workflows, and the corresponding 95% confidence interval of ERU.

In this work, we demonstrated how two evaluation frameworks (PPV/NPV and ERU)
can be applied to determine whether the radiologist-with-rule-out-device
workflow has a superior diagnostic performance than the standard-of-care,
radiologist-without-device workflow using two recently published retrospective
analyses. We first analyzed the use of these methods to a US mammography study
published by Yala et al. [9]. The method using PPV/NPV values cannot
definitively conclude or exclude that the radiologist-with-rule-out-device
workflow has a superior diagnostic performance, whereas the ERU method (using
the baseline relative utility value estimated by Abbey et al. [21]) suggests a
potential improvement, though statistically insignificant, in terms of
expected utility. We also applied the ERU method to a European study. Given
the difference in clinical practice between Europe and the US, the baseline
relative utility value is unknown. However, the ERU method can still be
applied to compare the expected utility at different AI-operating thresholds.

##  4 Discussion

The two evaluation methods investigated (PPV/NPV and ERU) have their strengths
and limitations.

One main advantage of both methods is that they are readily applied to any
large observational clinical study. These observational studies may collect
only radiologists’ binary decisions and lack finer radiologist ratings that we
may measure in a controlled, prevalence-enriched reader study [2]. However,
these observational studies are collected in real clinical environments and
have large numbers of patients and image readings. The use of PPV/NPV values
and ERU can reduce ambiguity in our assessment of rule-out device performance
from large observational clinical studies.

As demonstrated in the European example in Sec. 3.2, another benefit of using
ERU method instead of PPV/NPV lines is that ERU can be calculated using only
recall rate and cancer-detection rate. Given the low prevalence in a screening
setting, a majority of patient cases in large clinical studies are not
recalled by radiologists, and their ground truth cannot be determined. Hence,
it is clinically impractical to estimate specificity of these large studies.
Using recall rate and cancer detection rate in the ERU method eliminates the
needs of knowing the false-positive rates, which is less burdensome compared
to using PPV and NPV.

However, neither of the two methods provide insights on the full radiologist-
with-rule-out-device ROC curve or AUROC. Both PPV/NPV lines and ERU methods
compare the operating points before and after the device is hypothetically
included in the workflow and assume that the radiologists do not adjust their
decision-making in the radiologist-with-rule-out-device workflow. However, in
reality, due to the increase in prevalence, radiologists are expected to
adjust their decision thresholds to obtain a higher expected utility. We would
expect this adjustment in threshold and utility to increase as the rule-out
fraction increases. A reader study that elicits reader confidence ratings
would yield a full ROC curve which would map the reader performance at all
decision thresholds from which a hypothesis test using AUROC or partial AUC
can be made. However, because the retrospectively “simulated”, radiologist-
with-rule-out-device operating point should lie on the full with-device ROC
curve, this simulated operating point is still valuable in providing a lower
bound on the expected utility.

As discussed in the European example in Section 3.2, one disadvantage of the
ERU method is the need of a baseline relative utility value. This work relies
on the value of 162 estimated by Abbey et al. [21], which comes with a large
uncertainty and might change across different patient subgroups. Abbey et al.
found that different ROC models and different truthing criteria resulted in
different relative utility estimates, creating a larger systematic uncertainty
in addition to the statistical uncertainty. Furthermore, the estimated value
did not take into account patient subgroups, including breast density, type of
lesions, patients’ demography information and/or their risk levels. For
example, radiologists might choose to recall more patients with a higher risk
of cancer, resulting in a higher relative utility. This problem may be
overcome by estimating the relative utility baseline values for each subgroup.

It is important to note that both PPV/NPV lines and ERU methods can be
applied, not only to mammography screening, but also to other types of
screening (e.g. ultrasound breast cancer screening, lung cancer screening,
etc.). In terms of utility, if the radiologist-with-rule-out-device workflow
achieves an ERU significantly greater than the relative utility, it is
sufficient to conclude that the radiologist-with-rule-out-device workflow has
a higher expected utility compared to the radiologist-without-device scenario.
However, from the device evaluation perspective, achieving a higher ERU may
not be sufficient to demonstrate safety and effectiveness of the rule-out
device. Other factors, such as shifts in radiologists’ decision thresholds and
correlations between radiologists’ decision and AI outputs, should also be
considered.

##  5 Conclusion

Rule-out devices are intended to help reduce radiologists’ workloads in a
screening setting where prevalence is low. However, collecting large datasets
and conducting prospective studies may be burdensome. The two evaluation
methods (PPV/NPV and ERU) explored in this work potentially provide insights,
without the need of large prospective clinical studies, in comparing the
diagnostic performance between radiologist-with-rule-out-device and standard-
of-care (radiologist-without-device) workflows. By applying the two methods to
two recently published analyses, we gained understanding on the strength and
limitations of these two methods. Future work will be continued to investigate
more potential use cases of ERU method, alternative evaluation methods, and
associated considerations for assessing rule-out devices in a screening
setting.

## Acknowledgement

The authors acknowledge funding by appointment to the Research Participation
Program at the Center for Devices and Radiological Health (CDRH) administered
by the Oak Ridge Institute for Science and Education (ORISE) through an
interagency agreement between the US Department of Energy and the US Food and
Drug Administration (FDA).

## References

  * [1] FDA, “Clinical performance assessment: Considerations for computer-assisted detection devices applied to radiology images and radiology device data in premarket notification (510(k)) submissions,” guidance document, US Food and Drug Administration (2022).
  * [2] FDA, “Computer-assisted detection devices applied to radiology images and radiology device data - premarket notification [510(k)] submissions,” guidance document, US Food and Drug Administration (2022).
  * [3] N. Karssemeijer, A. M. Bluekens, D. Beijerinck, et al., “Breast cancer screening results 5 years after introduction of digital mammography in a population-based screening program,” Radiology 253, 353–358 (2009).
  * [4] C. D. Lehman, R. F. Arao, B. L. Sprague, et al., “National performance benchmarks for modern screening digital mammography: Update from the breast cancer surveillance consortium,” Radiology 283, 49–58 (2017).
  * [5] USPSTF, “Breast cancer: Screening.” https://www.uspreventiveservicestaskforce.org/uspstf/draft-recommendation/breast-cancer-screening-adults#fullrecommendationstart (2022).  Published: 2023-05-09.
  * [6] ACR, “The radiology labor shortage.” https://www.acr.org/Practice-Management-Quality-Informatics/ACR-Bulletin/Articles/March-2022/The-Radiology-Labor-Shortage (2022).  Published: 2022-02-10.
  * [7] RSNA, “Radiology facing a global shortage.” https://www.rsna.org/news/2022/may/global-radiologist-shortage (2022).  Published: 2022-05-10.
  * [8] A. Rodriguez-Ruiz, K. Lång, A. Gubern-Merida, et al., “Can we reduce the workload of mammographic screening by automatic identification of normal exams with artificial intelligence? a feasibility study,” European Radiology 29, 4825–4832 (2019).
  * [9] A. Yala, T. Schuster, R. Miles, et al., “A deep learning model to triage screening mammograms: A simulation study,” Radiology 293(1), 38–46 (2019).  PMID: 31385754.
  * [10] S. E. Hickman, G. C. Baxter, and F. J. Gilbert, “Adoption of artificial intelligence in breast imaging: evaluation, ethical constraints and limitations,” British Journal of Cancer 125, 15–22 (2021).
  * [11] M. Larsen, C. F. Aglen, S. R. Hoff, et al., “Possible strategies for use of artificial intelligence in screen-reading of mammograms, based on retrospective data from 122,969 screening examinations,” Eur Radiol 32, 8238–8246 (2022).
  * [12] A. D. Lauritzen, A. Rodríguez-Ruiz, M. C. von Euler-Chelpin, et al., “An artificial intelligence–based mammography screening protocol for breast cancer: Outcome and radiologist workload,” Radiology 304, 41–49 (2022).
  * [13] A. N. Giaquinto, H. Sung, K. D. Miller, et al., “Breast cancer statistics, 2022,” CA: A Cancer Journal for Clinicians 72, 524–541 (2022).
  * [14] D. P. Chakraborty, Observer Performance Methods for Diagnostic Imaging, CRC Press, Boca Raton (2018).
  * [15] A. Rodriguez-Ruiz, K. Lång, A. Gubern-Merida, et al., “Stand-Alone Artificial Intelligence for Breast Cancer Detection in Mammography: Comparison With 101 Radiologists,” JNCI: Journal of the National Cancer Institute 111, 916–922 (2019).
  * [16] B. J. Biggerstaff, “Comparing diagnostic tests: a simple graphic using likelihood ratios,” Statistics in Medicine 19(5), 649–663 (2000).
  * [17] J. P. Egan, Signal Detection Theory and ROC analysis, Academic Press, New York (1975).
  * [18] C. K. Abbey, M. P. Eckstein, and J. M. Boone, “An equivalent relative utility metric for evaluating screening mammography,” Medical Decision Making 30(1), 113–122 (2010).  PMID: 19706880.
  * [19] S. G. Baker, N. R. Cook, A. Vickers, et al., “Using relative utility curves to evaluate risk prediction,” J. R. Stat. Soc. Ser. A Stat. Soc. 172, 729–748 (2009).
  * [20] R. F. Wagner, C. A. Beam, and S. V. Beiden, “Reader variability in mammography and its implications for expected utility over the population of readers and cases,” Medical Decision Making 24(6), 561–572 (2004).  PMID: 15534338.
  * [21] C. K. Abbey, M. P. Eckstein, and J. M. Boone, “Estimating the relative utility of screening mammography,” Medical Decision Making 33(4), 510–520 (2013).  PMID: 23295543.
  * [22] E. D. Pisano, C. Gatsonis, E. Hendrick, et al., “Diagnostic performance of digital versus film mammography for breast-cancer screening,” New England Journal of Medicine 353, 1773–1783 (2005).
  * [23] D. Green and J. Swets, Signal Detection Theory and Psychophysics, Wiley (1966).
  * [24] R. Smith-Bindman, “Comparison of screening mammography in the United States and the United Kingdom,” JAMA 290, 2129 (2003).
  * [25] S. W. Fletcher and J. G. Elmore, “False-positive mammograms—can the USA learn from europe?,” The Lancet 365, 7–8 (2005).
  * [26] L. Domingo, S. Hofvind, R. A. Hubbard, et al., “Cross-national comparison of screening mammography accuracy measures in u.s., Norway, and Spain,” European Radiology 26, 2520–2528 (2015).
  * [27] S. Taylor-Phillips, D. Jenkinson, C. Stinton, et al., “Double reading in breast cancer screening: Cohort evaluation in the CO-OPS trial,” Radiology 287, 749–757 (2018).

Kwok Lung Fan is an Oak Ridge Institute for Science and Education fellow at
FDA/CDRH/OSEL/DIDSR and PhD candidate at the University of Maryland College
Park. He received his BS degree in physics from the University of Hong Kong in
2019. His current research focuses on the application of machine learning in
particle astrophysics and medical imaging.

Yee Lam Elim Thompson is a staff fellow at FDA/CDRH/OSEL/DIDSR. She received
her PhD in experimental astro-particle physics at University of Maryland
College Park in 2018. Her current research focus at the FDA is on evaluation
of AI/ML-enabled Computer-Aided Triage (CADt) / rule-out devices and spectral
small-angle x-ray scattering. She also helps review medical device
submissions.

Weijie Chen, PhD, is a research physicist in the Division of Imaging,
Diagnostics, and Software Reliability, Office of Science and Engineering
Laboratories, CDRH, US FDA, where he conducts research and provides consults
to regulatory review of medical devices. He earned his PhD in Medical Physics
in 2007 from the University of Chicago. His research interests include
performance characterization and assessment methodologies for medical imaging
and AI/ML/CAD devices.

Craig K. Abbey studies the transfer of diagnostic information to observers in
medical imaging using analysis of image statistics as well as visual
psychophysics. He received his PhD in applied mathematics from the University
of Arizona in 1998 and was a postdoctoral fellow in medical physics at Cedars-
Sinai Medical Center and UCLA from 1998 to 2001. He is currently a researcher
in the Dept. of Psychological and Brain Sciences at UC Santa Barbara.

Frank W. Samuelson is a physicist for the US Food and Drug Administration
where he reviews medical imaging devices and AI/ML/CAD devices. He conducts
research in evaluation methods such as reader studies. He earned his PhD in
Astrophysics in 1999 from Iowa State University in 1999.

###### List of Figures

  1. 1 Comparison between the standard-of-care workflow without AI algorithms (top), a workflow in which the AI is used to assist radiologists in decision making (middle), and a workflow in which the AI is used as a rule-out device where AI-negative patient cases are autonomously excluded from the radiologist’s reading list (bottom). This work focuses on the comparison of the workflow with AI as a rule-out device (bottom) against the workflow without AI (top).
  2. 2 Regions in the ROC space where the diagnostic performance of a hypothetical alternative diagnostics test A (orange dot) is superior to that of a hypothetical reference test B (blue star). Here, test A refers to the radiologist-with-rule-out-device workflow, and test B refers to the radiologist-without-device workflow. If the true-positive and false-positive rates of test A falls in the blue-shaded region, test A has a better sensitivity (Se) and specificity (Sp), as well as a higher positive predictive value (PPV) and negative predictive value (NPV), than test B. If they fall in the purple-shaded region below (or on the right side of) the blue area, test A has a lower sensitivity (or specificity) but a higher PPV (or NPV). If the orange dot is outside of the purple region, PPV/NPV lines cannot be used to decide whether test A has a more superior diagnostic performance than test B.
  3. 3 The blue line represents a hypothetical ROC curve in a radiologist-without-device workflow. The gray dashed lines represent different iso-utility lines. The slope of the iso-utility lines is $Q_{\pi}/U_{\text{Rel}}$. Iso-utility lines with a higher y-intercept represent a higher expected utility. The blue star shows the optimal point along the ROC curve, and the tangent at the optimal point (maximum utility) is equal to the slope of the iso-utility line.
  4. 4 Regions where a simulated radiologist with an rule-out device (alternative test A) has a superior diagnostic performance compared to that of a radiologist-alone workflow (reference test B). The blue-shaded region indicates the alternative test A has a better sensitivity (Se) and specificity (Sp) than the reference test B. The purple-shaded region indicates that test A has either a lower sensitivity or specificity but a higher positive predictive value and a higher negative predictive value. The green-shaded region indicates test A has a higher utility than test B. The relative utility along the green dashed line was estimated to be 162 by Abbey et al. [21] for US screening mammography. The operating points of test A (orange dot) and test B (blue star) as well as the disease prevalence are based on the paper by Yala et al. [9].
  5. 5 Equivalent relative utility (ERU) as a function of the percentage of patients excluded by the AI algorithm as a potential rule-out device based on the AI thresholds provided in Table E2 in the Appendix of Yala et al. [9]. The blue line shows the estimated ERUs for different AI thresholds, and the vertical error bars represent the 95% confidence intervals. The orange dashed line represents the baseline relative utility of 162 from Abbey et al. [21].
  6. 6 Equivalent relative utility vs percentage of patients ruled out by the AI with data from Larsen et al. [11]. The blue line shows equivalent relative utility for different percentages of patients ruled out (AI decision threshold). The vertical error bar represents the 95% confidence interval.

###### List of Tables

  1. 1 Table summarizing the ERU values, their 95% confidence intervals, and the probabilities that the radiologist-with-rule-out-device workflow have a greater utility and PPV/NPV as a function of the AI thresholds provided in Table E2 of the Appendix of Yala et al. [9]. We used 5000 bootstrap samples to calculate the probabilities and the confidence intervals. The columns are (from left to right) percentage of total patients excluded by the AI algorithm as a potential rule-out device, the ERU between radiologist-with-rule-out-device and radiologist-without-device workflows, 95% confidence interval of ERU, the chance of ERU $>$ 162, the chance of the radiologist-with-rule-out-device workflow has a greater PPV and NPV than radiologist-without-device workflow.
  2. 2 Table summarizing the result of applying ERU methods to Scenarios 3-5 defined in Larsen et al. [11]. For each scenario (row), we give the percentage of total patients ruled out by the potential rule-out device, the ERU between the radiologist-without-device (double-reading) and radiologist-with-rule-out-device workflows, and the corresponding 95% confidence interval of ERU.


Physics of Elementary Particles and Atomic Nuclei. Theory

# Mott dissociation and kaon to pion ratio in the EPNJL model

###### Abstract

The behaviour of pseudoscalar mesons within the SU(3)PNJL-like models is
considered for finite T and $\mu_{B}$. We compare the pole approximation
(Breit-Wigner) with the Beth-Uhlenbeck approach. We evaluate the $K/\pi$
ratios along the phase transition line in the T-$\mu_{B}$ plane with constant
and $T/\mu_{B}$-dependent pion and strange quark chemical potentials. Using
the model, we can show that the splitting of kaon and anti-kaon masses appears
as a result of introduction of density and this explains the difference in the
$K^{+}/\pi^{+}$ ratio and $K^{-}/\pi^{-}$ ratio at low $\sqrt{s_{NN}}$ and
their tendency to the same value at high $\sqrt{s_{NN}}$. A sharp "horn"
effect in the $K^{+}/\pi^{+}$ ratio is explained by the enhanced pion
production which can be described by occurrence of a nonequilibrium pion
chemical potential of the order of the pion mass. We elucidate that the horn
effect is not related to the existence of a critical endpoint in the QCD phase
diagram.

a Joint Institute for Nuclear Research, 141980, Dubna, Russia b Institute for
Theoretical Physics, University of Wrocław, 50-204 Wrocław, Poland c National
Research Nuclear University (MEPhI), 115409 Moscow, Russia d Dubna State
University, 141982, Dubna, Russia e Matrosov Institute for System Dynamics and
Control Theory, Irkutsk 664033, Russia

PACS: 12.38.Mh; 25.75.Nq

## Introduction

Our interest to the ’’horn’’ in the $K^{+}/\pi^{+}$ ratio (the peak-like
structure at the collision energy $\sqrt{s_{NN}}\sim 8$ GeV) appears due to
the fact that modern attempts to explain its existence give good intuitive
explanation of the $K/\pi$ bechaviour in high-energy region. The smooth,
almost constant behaviour at high energies (RHIC, LHC) can be explained by the
fact that at high energies during the heavy ion collision the quark-gluon-
plasma is created (it is confirmed by other signals). The creation of the QGP
is accompanied by the constant, independent on the temperature yield of the
strange particles. The jump from the maximum value to the almost constant line
is connected with the increase of the pion yield in comparison with
strangeness yield when deconfinement appears and it could be a signal of
’’onset of deconfinement’’ (the idea was originally predicted by Gazdzicki and
Gorenstein) [1].

The description of the $K^{+}/\pi^{+}$ at energies $<8$ GeV has not such a
clear explanation. From one hand it was shown that the partial chiral symmetry
restoration could be responsible for the quick increase in the $K^{+}/\pi^{+}$
ratio at low energies and its decrease with increasing energy (as a result of
chiral condensate destruction) [2, 3]. From other hand it is not clear still
if the deconfinement appeared during heavy ion collision at such energies and
if it coincided with the chiral phase transition. And at last, the recent data
from the NA61 experiment have shown a strong dependence on the system size
with no horn effect in Ar+Sc collisions [1] for which the reason is not yet
understood.

The main idea of this work is find the model able to reproduce both chiral
phase transition and deconfinement transition at finite T and $\mu_{B}$. From
this point the NIL-like models are promising models, as the models are capable
of describing both the chiral phase transition and deconfinement transition.
The chiral symmetry breaking is referred to the quarks developing
quasiparticle masses by propagating in the chiral condensate and the
confinement of coloured quark states is effectively taken into account by
coupling the chiral quark dynamics to the Polyakov loop and its effective
potential. However, in the model the absolute value of the pseudocritical
temperature of the chiral crossover transition with above 200 MeV is too large
when compared to the lattice QCD result for 2+1 flavors of $T_{c}=156.5\pm
1.5$ MeV [4].

The so-called entanglement PNJL (EPNJL) model, with the modified scalar four-
quark interaction $g_{S}$ could be one of possible solutions of this problem
[5, 6]. Such entanglement leads to a change of the phase diagram: the
(pseudo-)critical temperature of the chiral and deconfinement crossover
transition at low chemical potentials is lowered towards the value of the
Lattice QCD prediction. More physical solution is to go beyond the mean field
approximation and to consider the role of hadronic excitations in the medium
in melting the chiral condensate [7]. First steps in this direction have been
explored, e.g., within a generalized Beth-Uhlenbeck approach [8, 9]. We also
considered the (E)PNJL model with the vector interaction to evaluate the
effect of changing the chiral phase transition of the first order to the soft
crossover in the region of high $\mu_{B}$. The including of the vector
interaction to the model gives such possibility as the position of the
critical end point (CEP) in the phase diagram depends on the value of the
coupling $g_{V}$. When $g_{V}$ reaches the critical value, the chiral phase
transition turns into soft crossover overall [10, 11].

In the work we show, that the energy-dependent behaviour of the $K\pi$ ratio
could be rescaled by new variable $T/\mu_{B}$, where both T and $\mu_{B}$ are
chosen along the line of the chiral phase transition, which is supposed to
correspond to the chemical freeze-out line in the QCD phase diagram. In this
region can be important to introduce the nonequilibrium pion chemical
potential. The reason to consider the nonequilibrium pion distribution
function lies in the behaviour of the pions during the heavy ion collision:
their number is quasi conserved over the time scale of the HIC until freeze-
out. The resulting nonequilibrium pion distribution function with the pion
chemical potential has been successfully used in the phenomenological
description of the transverse momentum spectra of pions produced in heavy-ion
collision experiments, see [12].

## Mass spectrum for mesons at finite temperature and density

We start from the SU(3) PNJL model with the Kobayashi - Maskawa - t’Hooft
(KMT) interaction [13]

$\displaystyle\mathcal{L\,}$ $\displaystyle=$
$\displaystyle\bar{q}\,(\,i\,{\gamma}^{\mu}D_{\mu}-\hat{m}-\gamma_{0}\hat{\mu})\,q+\frac{1}{2}g_{S}\,\sum_{a=0}^{8}\,[\,{(\,\bar{q}\,\lambda^{a}\,q\,)}^{2}+{(\,\bar{q}\,i\gamma_{5}\,\lambda^{a}\,q)}^{2}\,]$
(1)
$\displaystyle+g_{D}\left\\{\mbox{det}\,[\bar{q}\,(1+\gamma_{5})\,q\,]+\mbox{det}\,[\bar{q}\,(1-\gamma_{5})\,q\,]\right\\}-\mathcal{U}(\Phi,\bar{\Phi};T).$

The model describes the dynamical chiral symmetry breaking as a coupling of
quarks to the chiral condensate. The quarks develop a quasiparticle mass even
for vanishing current quark masses $m_{0i}$ (chiral limit) by propagating in
the chiral condensate. In this model with Lagrangian (1) the quarks are
coupled both to the chiral condensate and to the homogeneous gluon background
fields representing by the Polyakov loop dynamics. The confinement properties
are described by the effective potential $\mathcal{U}(\Phi,\bar{\Phi};T)$,
which depends on the complex traced Polyakov loop $\Phi={\rm
Tr}_{c}\exp[i\beta(\lambda_{3}A_{3}^{0}+\lambda_{8}A_{8}^{0})]$ and its
conjugate $\bar{\Phi}$. The potential is constructed on the basis of the
center symmetry $Z_{3}$ of the color SU(3) gauge group. The possible
temperature dependence of its parameters is fitted to lattice QCD results for
the pressure in the pure gauge sector (for details see [14]).

The pseudocritical temperature of the chiral crossover transition at
$\mu_{B}=0$ GeV in the mean field PNJL model is $T_{c}=0.218$ GeV and this
value is higher than the $T_{c}=156.5\pm 1.5$ GeV obtained by the ab-initio
simulations of Lattice QCD [4].

PNJL model with rescaled $T_{0}$ has a problem with correspondence between
critical temperature for the chiral condensate and for the Polyakov loop. To
improve this disagreement with Lattice data, was obtained so-called EPNJL
model [6]. According to this extension of the PNJL model, a phenomenological
dependence of the scalar meson coupling $g_{S}$ on the Polyakov loop is
introduced that does obey the $Z_{3}$-symmetry

$\tilde{g}_{S}(\Phi)=g_{S}(1-\alpha_{1}\Phi\bar{\Phi}-\alpha_{2}(\Phi^{3}+\bar{\Phi}^{3})),$
(2)

where the parameters $\alpha_{1}=\alpha_{2}=0.2$ were chosen in [5] to
reproduce the two-flavor LQCD data. Such rescaling of $g_{S}$ leads to a
rescaling of the pseudocritical temperature in the low-density region [5, 6]
to $T_{c}\sim 0.179$ GeV and now being synchronous with the Polyakov-loop
transition.

We also considered the PNJL model with vector interaction to discuss the
critical end point position on the phase diagram. The position of the CEP is
under debate [10] and can not be settled yet by Lattice QCD simulations. It
was shown in the NJL and in the PNJL modes with vector interaction, that the
CEP can disappear when the vector coupling exceeds a critical value. The
Lagrangian of the PNJL model with vector interaction is obtained from the
Lagrangian (1) by adding the vector interaction contribution
$\displaystyle{-\frac{1}{2}g_{V}\sum_{a=0}^{8}\,\,{(\,\bar{q}\gamma_{\mu}\lambda^{a}\,q\,)}^{2}}$
the vector coupling $g_{V}$ can also be rescaled in the spirit of the EPNJL
model, similar to the scalar coupling (2).

The meson spectral properties are encoded in the meson propagator (M=P, S)

$\mathcal{S}_{ij}^{M}(\omega,\mathbf{q})=\frac{2P_{ij}}{1-4P_{ij}\Pi_{ij}^{M}(\omega,\mathbf{q})}~{},$
(3)

where for non-diagonal pseudo-scalar mesons $\pi\,,K$:

$\displaystyle P_{ij}^{\pi}$ $\displaystyle=$ $\displaystyle
g_{S}+g_{D}\left\langle\bar{q}_{s}q_{s}\right\rangle,\ \ \
P_{ij}^{K}=g_{S}+g_{D}\left\langle\bar{q}_{u}q_{u}\right\rangle$ (4)

and $\Pi_{ij}^{M}(P_{0})$ is the polarization function. Particularly
interesting are the (complex) mass pole solutions $z_{M}=m_{M}-i\Gamma_{M}/2$
in the Breit-Wigner (BW) approximation of the Bethe-Salpeter equation at
vanishing meson momentum $\mathbf{q}=0$,

$1-2P_{ij}\Pi_{ij}^{M}(z=z_{M},\mathbf{q=0})=0,$ (5)

where for the width of the quark-antiquark resonant state $\Gamma_{M}\ll
m_{M}$ should hold. When the mass parameter is below the two-quark threshold
($m_{M}<m_{i}+m_{j}$), the polarization function is real and Eq. (5)
corresponds to the homogeneous Bethe-Salpeter equation in the rest frame of
the meson which defines a true bound state with mass $m_{M}$ and infinite
lifetime ($\Gamma_{M}=0$). $\Gamma_{M}$ becomes finite at the Mott transition,
when the meson becomes unbound and its mass is larger than that of its
constituents $m_{M}>m_{i}+m_{j}$. The temperatures of the Mott transition for
pion and kaon are $T^{\pi}_{\rm Mott}=0.232$ GeV and $T^{K}_{\rm Mott}=0.230$
GeV, respectively.

An adequate account for both, bound and continuum states can be made within
the Beth-Uhlenbeck (BU) approach that is generalized for the case of dense
matter and the Mott dissociation of mesons across the chiral/deconfinement
transition in [15, 16]. For this, the mesonic propagator can be rewritten in
the ’’polar’’ representation:

$\mathcal{S}^{M}_{ij}(\omega,\bar{q})=|\mathcal{S}^{M}_{ij}(\omega,\bar{q})|e^{i\delta_{M}(\omega,\bar{q})}$
(6)

with meson phase shift $\delta_{M}(\omega,\bar{q})$. In the phase shift
representation, the bound state appears at the energy where the phase shift
has a jump to the value $\pi$ [15]. In the rest frame of the meson this energy
corresponds to its mass. For energies above the continuum threshold, the phase
shift drops and asymptotically vanishes, in accordance with the Levinson
theorem.

At zero chemical potential both the BW and the BU approach show the degeneracy
of positive and negative charged meson masses, as for light quarks
$m_{u}=m_{d}$ was chosen. At nonzero chemical potential and low T, the
splitting of mass in charged multiplets appears due to excitation of the Dirac
sea modified by the presence of the medium (see [17, 18, 19]). In BU approach
an anomalous mode appears after $T_{\rm Mott}$ for positively charged kaons.
This is due to the fact that with increasing density the scattering mode of
the polarization function $\Pi_{ij}^{M}(P)$ becomes dominant and induces a
bound state pole in the meson propagator (3) [20].

## Kaon to pion ratio

In this Section we discuss the application of the PNJL model to description of
the kaon to pion ratio. There are some remarks to our calculations in the
frame of the PNJL model.

The first point is that experimental data are shown as a function of the
collision energy $\sqrt{s_{NN}}$ which never appears as a parameter in
effective models. We used the fact that in the statistical model for each
experimental energy of collision the temperature and the baryon chemical
potential of freeze-out can be found using parametrization suggested, e.g., by
Cleymans et al. [21]. Using this parametrization, the $K/\pi$ ratio can be
considered as a function of a new variable $T/\mu_{B}$ instead of $s_{NN}$,
where (T, $\mu_{B}$) are taken along the freeze-out line.

The second point is that according to the nonequilibrium statistical models
the ratio of the yields of mesons, such as the $K/\pi$ ratios which were
obtained in the midrapidity range, can be calculated in terms of the ratio of
the number densities of mesons ($K^{\pm}/\pi^{\pm}=n_{K^{\pm}}/n_{\pi^{\pm}}$)
with

$\displaystyle n_{M}$ $\displaystyle=$ $\displaystyle
d_{M}\int_{0}^{\infty}\frac{\rm d^{3}{\bf q}}{(2\pi)^{3}}g_{M}(E_{M}),$ (7)

where $g_{M}(E)=(e^{(E-\mu_{M})/T}-1)^{-1}$ is the Bose function for a meson
with energy $E$ and chemical potential $\mu_{M}$. In the generalized BU
approach, however, an off-shell generalization of the partial number density
of the mesonic species $M$ holds

$\displaystyle n_{M}(T)$ $\displaystyle=$ $\displaystyle
d_{M}\int\frac{{d}^{3}\mathbf{q}}{(2\pi)^{3}}\int\frac{d\omega}{2\pi}g_{M}(\omega)\frac{d\delta_{M}(\omega,\mathbf{q})}{d\omega},$
(8)

where $\delta_{M}(\omega)=\delta_{M}(\omega,\mathbf{0})$ is the meson phase
shift that was calculated for mesons at rest, but for which Lorentz-boost
invariance $\omega=\sqrt{q^{2}+{m}^{2}}$ was assumed, see [16].

The chemical potential for pions as a parameter characterizing the
nonequilibrium state has been chosen as a constant close to the pion mass,
e.g., $\mu_{\pi}=0.135$ GeV, following the works [12]. The chemical potential
for kaons is defined here by their chemical composition, i.e. by their quark
content, so that $\mu_{K^{+}}=\mu_{u}-\mu_{s}$ and
$\mu_{K^{-}}=\mu_{s}-\mu_{u}=-\mu_{K^{+}}$.

For the Beth-Uhlenbeck approach, lowering the pseudocritical temperature of
the chiral restoration in EPNJL model leads to a more rapid drop above the
horn, than in the BW approximation, when the chemical potential of pion is
constant. The pion chemical potential stems from the nonequilibrium character
of pion production process which depends on the temperature and density of the
system, so in particular in collisions with high pion multiplicities we can
observe a frozen nonequilibrium state of the pions, with the nonequilibrium
chemical potential $\mu_{\pi}$. It motivated us to introduce the specific
functional form for its dependence on $x=T/\mu$. The strange quark chemical
potential is a parameter which reflects the asymmetry in the production of
positive and negative charged kaons. Since this imbalance is maximal at low
energies and vanishes at high energies we adjusted a monotonously falling
function of x, i.e. of $\sqrt{(}s)$. In order to find the x-dependence of
$\mu_{\pi}$ and $\mu_{s}$ the following strategy was used: one can fit the
pion chemical potential $\mu_{p}i$ at large $x$ which defines the overall
normalization of ratio curves and error bar reflect the experimental
uncertainty. Then we supposed that the pion chemical potential should be a
non-decreasing function of $x$, while the ratio of strange chemical potential
to nonstrange one should be decreasing one. Therefore the $\mu_{\pi}(x)$ and
$\mu_{s}(x)$ has a smooth behaviour in the transition region (which could be
fitted with from the experimental points), and can be chosen as a functions of
the Woods-Saxon form

$\displaystyle\mu_{\pi}(x)$ $\displaystyle=$
$\displaystyle\mu_{\pi}^{\mathrm{min}}+\frac{\mu_{\pi}^{\mathrm{max}}-\mu_{\pi}^{\mathrm{min}}}{1+\exp\left(-(x-x_{\pi}^{\mathrm{th}})/\Delta
x_{\pi}\right)},$ (9) $\displaystyle\mu_{s}(x)$ $\displaystyle=$
$\displaystyle\frac{\mu_{s}^{\mathrm{max}}}{1+\exp((x-x_{s}^{\mathrm{th}})/\Delta
x_{s})}.$ (10)

The best values of the parameters for the PNJL (EPNJL) model are
$\mu_{\pi}^{\mathrm{max}}=147.6\pm 10~{}(107\pm 10)$ MeV,
$\mu_{\pi}^{\mathrm{min}}=120~{}(92)$ MeV,
$x_{\pi}^{\mathrm{th}}=0.370~{}(0.409)$, $\Delta x_{\pi}=0.015~{}(0.00685)$.
The parameter values $\mu_{s}^{\mathrm{max}}/\mu_{u}^{\mathrm{crit}}=0.205$,
$x_{s}^{\mathrm{th}}=0.223$, $\Delta x_{s}=0.06$ hold for both models.

Figure 1: Left panel: The ratios $K^{+}/\pi^{+}$ (black lines) and
$K^{-}/\pi^{-}$ (red lines) are shown as function of $T/\mu_{B}$ along the
chemical freezeout line for the EPNJL model with $g_{V}=0$ within the BU
approach. Thin lines correspond to the case with fixed $\mu_{\pi}=147.6$ MeV.
The thick lines are obtained when $\mu_{s}$ and $\mu_{\pi}$ vary with
$T/\mu_{B}$ as shown in right panel.

In Fig. 1 we show the $K^{+}/\pi^{+}$ and $K^{-}/\pi^{-}$ for the EPNJL model
as a function of the specific entropy variable $x=T/\mu_{B}$ when the BU
approach is applied and the pion and strange quark chemical potentials are
either constant (thin lines) or given as functions of $x$ (right panel) so
that the experimental data can be fitted (thick lines). To show the behaviour
of $K/\pi$ ratios on the T-$\mu_{B}$, we choose the B.-Uh. approach with
nonequilibrium pion chemical potential in EPNJL model with $g_{V}=0.6g_{S}$
(when there are no the first order chiral phase transition in the system). The
dot on the phase diagram corresponds to the maximum of the $K^{+}/\pi^{+}$
ratio. It is clearly seen from the Fig. 2 the smooth increasing of the value
$K^{-}/\pi^{-}$ along the phase transition line (or with the increasing of
energy, if we make a rescaling back). For the $K^{+}/\pi^{+}$ ratio the
picture is more complicated: it shows the increase of the ratio in the middle
part of the phase transition line.

Figure 2: $K^{+}/\pi^{+}$ (left) and $K^{-}/\pi^{-}$ (right) on the
$T-\mu_{B}$ plane for the EPNJL model with $g_{V}=0.6~{}g_{S}$ (no CEP). The
black dot indicates the maximum of $K^{+}/\pi^{+}$ ratio on the line of
pseudocritical temperatures for the chiral transition (our proxy for chemical
freeze out).

## Result and discussion

In this work, we used the SU(3) PNJL model to describe the QCD matter at
finite temperature and density. In the frame of the model, we paid an
attention to the $K/\pi$ ratio as a function of $T/\mu_{B}$, where T and
$\mu_{B}$ are chosen along the phase transition line on the phase diagram. In
our works we replaced the $\sqrt{s_{NN}}$ by the variable $T/\mu_{B}$ choosing
$T/\mu_{B}$ along the line of the phase transition motivating that by the
observation that along the straight lines of constant $x$ in the phase diagram
the value of the specific entropy $S/N_{B}=s/n$ is constant and belongs to a
conserved quantity in the hydrodynamic evolution of the hadronizing fireball
created in central nucleus-nucleus collisions.

In the mean field PNJL model we considered the cases with different values of
$g_{V}$ which moves the CEP to lower T till it disappears and shifts the
crossover pattern to higher chemical potentials. According to our analysis,
the key to understanding the horn lies in the position and the slope of the
almost straight lines of constant $K^{+}/\pi^{+}$ and $K^{-}/\pi^{-}$ ratios
relative to the curved freezeout line. Both ratios are almost unaffected by
the appearance of a change from crossover to first order transition.

In the present work we have suggested that the sharpness of the horn effect in
the $K^{+}/\pi^{+}$ ratio is well explained by a Bose-enhanced pion production
for $x>x_{\rm horn}$, i.e. for heavy-ion collisions with $\sqrt{s_{NN}}>8$
GeV, in the region of meson dominance, where also a low-momentum enhancement
of pion production has been observed. Such an effect is best described by a
nonequilibrium pion distribution function, which according to Zubarev’s
concept of the nonequilibrium statistical operator requires an additional
Lagrange multiplier, the pion chemical potential, for a consistent description
[22]. Within the Beth-Uhlenbeck approach we have provided fit functions for
the $x$-dependence of the pion and strange quark chemical potentials that lead
to a simultaneous description of the $x$-dependence of both kaon-to-pion
ratios in accordance with the experimental data, with a strong horn effect for
the $K^{+}/\pi^{+}$ ratio. In the present paper only the result of partial fit
is presented. After finding the error bar of the maximal value of the pion
nonequilibrium chemical potential due to overall normalization to the
large-$x$ points (with almost zero chemical potential) the other parameters is
fitted. The detailed description of the fit and error bar estimation of all
parameters procedure has been given in [19].

This work was supported by the Russian Fund for Basic Research (RFBR) under
grant no. 18-02-40137. Numerical calculations are partially performed on
computing cluster "Ac. V.M. Matrosov".

## References

  * [1] _M. Gazdzicki, M. Gorenstein and P. Seyboth_ Brief history of the search for critical structures in heavy-ion collisions // Acta Phys. Polon. B —- 2020\. —- V. 51, no.5 —- P. 1033–1066. —- arXiv:2004.02255 [hep-ph].
  * [2] _A. Palmese, W. Cassing, E. Seifert, T. Steinert, P. Moreau and E. L. Bratkovskaya_ Chiral symmetry restoration in heavy-ion collisions at intermediate energies // Phys. Rev. C —- 2016\. —- V. 94, no.4.—- P. 044912. —- arXiv:1607.04073 [nucl-th].
  * [3] _P. Moreau, A. Palmese, W. Cassing, E. Seifert, T. Steinert and E. L. Bratkovskaya_ Evidence for chiral symmetry restoration in heavy-ion collisions // Nucl. Phys. A —- 2017\. —- V. 967. —- P. 836-839.—- arXiv:1704.05946 [nucl-th].
  * [4] _A. Bazavov et al._ [HotQCD] Chiral crossover in QCD at zero and non-zero chemical potentials // Phys. Lett. B —- 2019\. —- 795\. —- P. 15.-21.—- arXiv:1812.08235 [hep-lat].
  * [5] _Y. Sakai, T. Sasaki, H. Kouno and M. Yahiro_ Entanglement between deconfinement transition and chiral symmetry restoration // Phys. Rev. D —- 2010 —- V. 82. —- P. 076003. —- arXiv:1006.3648 [hep-ph].
  * [6] _M. C. Ruivo, P. Costa and C. A. de Sousa_ Effects of entanglement and instanton suppression at finite T in a SU(2) EPNJL model with anomaly // Phys. Rev. D —- 2012\. —- V.86. —- P. 116007. —- arXiv:1211.2272 [hep-ph].
  * [7] _J. Jankowski, D. Blaschke and M. Spalinski_ Chiral condensate in hadronic matter // Phys. Rev. D —- 2013—- V. 87, no.10.—- P. 105018. —- arXiv:1212.5521 [hep-ph].
  * [8] _D. Blaschke, A. Dubinin and L. Turko_ Generalized Beth-Uhlenbeck approach to the equation of state for quark-hadron matter // Acta Phys. Polon. Supp. —- 2017\. —- V. 10. —- P. 473 –480.—- arXiv:1612.09556 [hep-ph].
  * [9] _D. Blaschke, A. Dubinin, D. Ebert and A. Friesen_ Effects of composite pions on the chiral condensate within the PNJL model at finite temperature // Phys. Part. Nucl. Lett. —- 2018 —- V. 15, no.3. —- (2018). —- P. 230–235. —- arXiv:1712.09322 [hep-ph].
  * [10] _M. A. Stephanov_ QCD phase diagram: An Overview // PoS LAT2006 —- 2006\. —- P. 024. —- arXiv:hep-lat/0701002 [hep-lat].
  * [11] _A. V. Friesen, Y. L. Kalinovsky and V. D. Toneev_ Vector interaction effect on thermodynamics and phase structure of QCD matter // Int. J. Mod. Phys. A —- 2015.—- V. 30, no.16. —- P. 1550089. —- arXiv:1412.6872 [hep-ph].
  * [12] _V. Begun, W. Florkowski and M. Rybczynski_ Explanation of hadron transverse-momentum spectra in heavy-ion collisions at $\sqrt{s}_{NN}=$ 2.76 TeV within chemical non-equilibrium statistical hadronization model // Phys. Rev. C —- 2014.—- V. 90, no.1.—- P. 014906.—- arXiv:1312.1487 [nucl-th].
  * [13] _K. Fukushima_ Phase diagrams in the three-flavor Nambu-Jona-Lasinio model with the Polyakov loop // Phys. Rev. D —- 2008.—- V. 77.—- P. 114028. —- arXiv:0803.3318 [hep-ph].
  * [14] _C. Ratti, M. A. Thaler and W. Weise_ Phases of QCD: Lattice thermodynamics and a field theoretical model //  Phys. Rev. D —- 2006.—- V. 73. —- P. 014019.—- arXiv:hep-ph/0506234 [hep-ph].
  * [15] _A. Wergieluk, D. Blaschke, Y. Kalinovsky and A. Friesen_ Pion dissociation and Levinson‘s theorem in hot PNJL quark matter // Phys. Part. Nucl. Lett. —- 2013\. —- V. 10.—- P. 660–668.—- arXiv:1212.5245 [nucl-th].
  * [16] _D. Blaschke, M. Buballa, A. Dubinin, G. Röpke and D. Zablocki_ Generalized Beth–Uhlenbeck approach to mesons and diquarks in hot, dense quark matter // Annals Phys. —- 2014.—- V. 348.—- P. 228–255.—- arXiv:1305.3907 [hep-ph].
  * [17] _A. Friesen, Y. Kalinovsky and V. Toneev_ Strange matter and kaon to pion ratio in the SU(3) Polyakov?Nambu?Jona-Lasinio model // Phys. Rev. C —- 2019.—- V. 99, no.4. —- P. 045201.—- arXiv:1808.04179 [hep-ph].
  * [18] _P. Costa, M. C. Ruivo, Y. L. Kalinovsky and C. A. de Sousa_ Pseudoscalar mesons in hot, dense matter // Phys. Rev. C —- 2004.—- V. 70. —- P. 025204.—- arXiv:hep-ph/0304025 [hep-ph].
  * [19] _D. Blaschke, A. V. Friesen, Y. L. Kalinovsky and A. Radzhabov_ Chiral phase transition and kaon-to-pion ratios in the entanglement SU(3) PNJL model // [arXiv:2009.03284 [hep-ph]].
  * [20] _D. Blaschke, A. Dubinin, A. Radzhabov and A. Wergieluk_ Mott dissociation of pions and kaons in hot, dense quark matter // Phys. Rev. D —- 2017.—- V. 96, no.9. —- P. 094008.—- arXiv:1608.05383 [hep-ph].
  * [21] _J. Cleymans, H. Oeschler, K. Redlich and S. Wheaton_ Comparison of chemical freeze-out criteria in heavy-ion collisions // Phys. Rev. C —- 2006.—- V. 73. —- P. 034905.—- arXiv:hep-ph/0511094 [hep-ph].
  * [22] _D. Blaschke, G. Röpke, D. N. Voskresensky, V. G. Morozov_ Nonequilibrium Pion Distribution within the Zubarev Approach // Particles —- 2020.—- V. 3. —- P. 380–393.—- arXiv:1501.01599 [hep-ph].


large overlap between three classes. FCR (left column), resolution-adjusted
power (middle column) and the expected fraction of covering prediction sets
(right column) versus SNR for: balanced classes for minimally informative
prediction sets (first row) and for prediction sets excluding a null class
(second row); unbalanced classes for minimally informative prediction sets
(third row) and for prediction sets excluding the largest class (fourth row).
Each class is a mixture of a common component and a unique component, see §
F.1 for details. Based on 2000 data generations, 1000 data points were used
for training, and $n=m=500$.

Figure 14: The setting is that of unbalanced classes, with each class having
probability half of being in the same mixture component, and the SNR is 8. In
the first row, the data points from each of the three classes, where the null
group is in blue: top left panel for the entire test sample, top right panel
the remaining examples after the initial selection step. In the second row,
left panel, their $p$-values using the entire calibration set (black circles)
and using the examples from the calibration set remaining after the initial
selection step (blue triangles). In the second row, right panel, the empirical
CDF of the nonconformity scores for true classes in the entire calibration set
(gray) and using the examples from the calibration set remaining after the
initial selection step for the of the (blue).