AUC-based Selective Classification

Andrea Pugnana Salvatore Ruggieri

Scuola Normale Superiore<EMAIL_ADDRESS>University of Pisa
<EMAIL_ADDRESS>
###### Abstract

Selective classification (or classification with a reject option) pairs a
classifier with a selection function to determine whether or not a prediction
should be accepted. This framework trades off coverage (probability of
accepting a prediction) with predictive performance, typically measured by
distributive loss functions. In many application scenarios, such as credit
scoring, performance is instead measured by ranking metrics, such as the Area
Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a
selection function to a given probabilistic binary classifier. The approach is
specifically targeted at optimizing the AUC. We provide both theoretical
justifications and a novel algorithm, called AUCross, to achieve such a goal.
Experiments show that AUCross succeeds in trading-off coverage for AUC,
improving over existing selective classification methods targeted at
optimizing accuracy.

##  1 Introduction

The predictive performance of a classifier is typically not homogeneous over
the data distribution. Identifying sub-populations with low performance is
helpful, e.g. for debugging and monitoring purposes. In many socially
sensitive application scenarios, we can consider not predicting at all for
these sub-populations as an alternative to poor or even harmful predictions.
Such applications include credit scoring, curriculum screening, access to
public benefits, and medical diagnoses. Selective classification (or
classification with a reject option) Chow, (1970); Pietraszek, (2005) pairs a
classifier with a selection function to determine whether a prediction should
be accepted or the classifier should abstain. The selection function assesses
the trustworthiness of a prediction. The literature on selective
classification has mainly considered distributive loss functions. However, in
many real-world scenarios, such as credit scoring Engelmann, (2011), the
quality of a binary probabilistic classifier is concerned with the
discriminative power of the ranking induced by its scores Huang and Ling,
(2005). This is measured by the Area Under the ROC Curve (AUC) Fawcett, 2006a
or, equivalently, by the Gini coefficient Wu and Lee, (2014). Algorithms
designed to minimize, e.g., the error rate may not lead to the best possible
AUC values Cortes and Mohri, (2003).

In this paper, we first introduce the problem of AUC-based selective
classification, which consists of inducing a selective classifier which
optimizes the AUC over the sub-population of accepted predictions while
guaranteeing a minimum probability mass (called coverage) of such sub-
population. We then focus on an instance of the AUC-selective classification
problem, where we assume that: (i the classification algorithm is given,
including its hyper-parameters, and (ii the selection function abstains on a
range of the scores (score-based selective functions). Assumption (i allows
lifting an existing classifier to a selective classifier. In this respect, our
approach is model-agnostic, as we do not make assumptions about the
classification algorithm. Assumption (ii is widespread in selective
classification, supported by theoretical results El-Yaniv and Wiener, (2010).
Such an assumption allows for a theoretical analysis of the AUC-based
selective classification problem. We devise AUCross, a model-agnostic
algorithm for estimating the bounds of score-based selective functions that
maximize the AUC for target coverage. The approach is based on a cross-fitting
strategy over the training set. Bound estimates are supported by a theoretical
analysis of (sufficient) conditions for abstaining that lead to an increase in
the AUC. Experiments show that the approach performs very close to an oracle
score-based selective function (which has access to the true class of
instances), and it outperforms existing selective classification approaches
targeted at optimizing accuracy.

The paper is organized as follows. Section 2 briefly surveys related work.
Section 3 recalls basic concepts of selective classification and it introduces
the AUC-based selective classification problem. Our theoretical approach is
presented in Section 4, while we introduce AUCross algorithm in Section 5.
Section 6 reports experimental results. Finally, we draw conclusions and
outline possible extensions.

##  2 Related Work

The two main models of selective classification are the cost model Chow,
(1970) and the the bounded-improvement model Pietraszek, (2005). In the
former, the goal is to minimize the expected cost, assuming a cost for
misclassification and a cost for abstention, or a more refined cost based on
the confusion matrix Tortorella, (2005). In the Bayes optimal selective
classifier, the selection function abstains when the posterior probability of
the predicted class is below some threshold. Posterior probabilities can be
estimated on a validation set by the plug-in rule Herbei and Wegkamp, (2006).
In the bounded-improvement model, the selection function is evaluated based on
the probability mass of the accepted region (coverage) and the expected loss
over such a region (selective risk). One can maximize coverage for a maximum
target risk or minimize risk for a minimum target coverage Geifman and El-
Yaniv, (2017). Franc and Průša, (2019) establish the equivalence of cost-based
and bounded-improvement models.

Most approaches for selective classification are model-specific in that they
build the classifier and the selection function concurrently. Examples include
methods for Support Vector Machines (SVM) Fumera and Roli, (2002), boosted
decision trees Cortes et al., (2016), Deep Neural Networks (DNNs) Geifman and
El-Yaniv, (2017, 2019); Liu et al., (2019); Huang et al., (2020). See
Hendrickx et al., (2021) for a complete overview of existing methods.
Moreover, the definitions of cost or risk in selective classification have
been provided using distributive loss functions, where the loss is defined for
every prediction in isolation. AUC is a metric about the ranking induced by a
classifier, for which the loss is determined for pairs of instances. To the
best of our knowledge, the only work directly addressing AUC selective
classification is Shen et al., (2020). However, the selection function is used
here to accept or to abstain from ranking pairs of instances. The AUC to be
optimized is defined as the mean correct order of accepted pairs (an extension
of the Mann-Whitney U-statistics): the same example could appear in both an
accepted and a rejected pair. We target, instead, accepting or rejecting
predictions for every single instance. The AUC we optimize is the mean correct
order of any pair of accepted cases (i.e., the Mann-Whitney U-statistics over
the accepted region).

In this paper, we adhere to the bounded-improvement model, with the AUC as the
metric to optimize by abstaining on single instances, and we take a model-
agnostic view of the problem. Notice that AUC-based selective classification
is orthogonal to the many approaches for optimizing AUC in supervised learning
Yang and Ying, (2023).

##  3 Selective classification for the AUC

Consider random variables $(\mathbf{X},Y)\in\mathcal{X}\times\mathcal{Y}$,
where $\mathcal{X}\subseteq\mathbb{R}^{d}$ is a feature space and
$\mathcal{Y}=\\{0,1,\ldots,n_{\mathcal{Y}}\\}$ a finite label space. The joint
distribution of $(\mathbf{X},Y)\sim\mathcal{D}$ is unknown, but we can observe
one or more datasets of i.i.d. realizations. A classifier is a function
$h:\mathcal{X}\to\mathcal{Y}$ that maps features to classes, computed from an
hypothesis space and a dataset (training set). The expected loss over the
distribution
$R(h):=\int_{\mathcal{X}\times\mathcal{Y}}l(h(\mathbf{x}),y)d\mathcal{D}(\mathbf{x},y)=\mathbb{E}_{\mathcal{D}}[l(h(\mathbf{X}),Y)]$
is called the risk, where $l:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}$ is a
loss function. The risk can be estimated starting from a test set
$S_{n}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ through the empirical risk
$R_{emp}(h):=\frac{1}{n}\sum_{i=1}^{n}l(h(\mathbf{x}_{i}),y_{i})$. Such a
canonical setting is extended to model situations where predictions of
classifiers are not sufficiently reliable, and the option of abstaining is
preferable. A selective classifier is a pair $(h,g)$, where $h$ is a
classifier and $g:\mathcal{X}\to\\{0,1\\}$ is a selection function, which
determines when to accept/abstain from using $h$:

$(h,g)(\mathbf{x})=\begin{cases}h(\mathbf{x}&\text{if}\ g(\mathbf{x})=1\\\
\text{abstain}&\text{otherwise}\end{cases}$

A soft selection approach Geifman and El-Yaniv, (2017) consists of defining
$g$ in terms of a confidence function $k_{h}:\mathcal{X}\to[0,1]$ (the
subscript highlights that $k_{h}$ depends on $h$) and a threshold $\theta$ of
minimum confidence for accepting:

$g(\mathbf{x})=\mathbb{1}(k_{h}(\mathbf{x})>\theta)$ (1)

A good confidence function should order instances based on descending loss,
i.e., if $k_{h}(\mathbf{x}_{i})\leq k_{h}(\mathbf{x}_{j})$ then
$l(h(\mathbf{x}_{i}),y_{i})\geq l(h(\mathbf{x}_{j}),y_{j})$. The coverage of a
selective classifier is $\phi(g)=E_{\mathcal{D}}[g(\mathbf{X})]$, i.e., the
expected probability mass of the accepted region. The selective risk is risk
normalized by coverage:

$R(h,g)=\frac{\mathbb{E}_{\mathcal{D}}[l(h(\mathbf{X}),Y)g(\mathbf{X})]}{\phi(g}=\mathbb{E}_{\mathcal{D}}[l(h(\mathbf{X}),Y)|g(\mathbf{X})]$

Empirical coverage and empirical selective risk are respectively defined as
follows:

$\hat{\phi}(g|S_{n})=\frac{\sum_{i=1}^{n}g(\mathbf{x}_{i}}{n}$
$\hat{r}(h,g|S_{n})=\frac{\frac{1}{n}\sum_{i=1}^{n}l(h(\mathbf{x}_{i}),y_{i})g(\mathbf{x}_{i}}{\hat{\phi}(g|S_{n}}$

By defining $S^{g}_{m}=\\{\mathbf{x}_{i}\in S_{n}\ |\ g(\mathbf{x}_{i})=1\\}$,
the empirical coverage is $\hat{\phi}(g|S_{n})=|S^{g}_{m}|=m$ and the
empirical selective risk reduces to $\hat{r}(h,g|S_{n})=\hat{r}(h|S^{g}_{m})$.
The inherent trade-off between risk and coverage is summarized by the risk-
coverage curve El-Yaniv and Wiener, (2010). The selective classification
problem can be framed by fixing an upper bound to the selective risk and
looking for a selective classifier that maximizes coverage. Geifman and El-
Yaniv, (2017) show how to convert this framing into an alternative one, where
a lower bound $c$ for coverage is fixed (target coverage), and then we look
for a selective classifier that minimizes selective risk. We adhere to such a
formulation. Called $\theta$ the parameter(s) defining $h$ and $g$ (e.g., as
in the soft selection approach), the selective classification problem is
stated as:

$\min_{\theta}R(h,g)\hskip 8.61108pt\text{s.t.}\quad\phi(g)\geq c$

Let us now extend the framework to the AUC metric. We consider binary classes
(0/1, negatives/positives) and probabilistic classifiers, where
$h(\mathbf{x})\in[0,1]$ is an estimate of the probability that $\mathbf{x}$ is
positive. Let $\mathcal{D}_{1}$ and $\mathcal{D}_{0}$ be the conditional
distributions of positives and negatives, respectively. The AUC can be defined
as the probability that a randomly drawn positive receives a higher score than
a randomly drawn negative, conditioned to the fact that both are selected
according to $g$, i.e.

$\displaystyle
AUC(h,g)=\mathbb{E}_{\mathbf{X}_{0}\sim\mathcal{D}_{0},\mathbf{X}_{1}\sim\mathcal{D}_{1}}[\mathbb{1}(h(\mathbf{X}_{1})>h(\mathbf{X}_{0}))|$
$\displaystyle g(\mathbf{X}_{0})=1,g(\mathbf{X}_{1})=1]$

The AUC-selective classification problem can be stated as:

$\max_{\theta}AUC(h,g)\hskip 8.61108pt\text{s.t.}\quad\phi(g)\geq c$ (2)

A good confidence function should order instances based on ascending
contribution to the AUC while allowing for controlling the target coverage.
Empirical AUC given $S_{n}$ can be defined by restricting to the set
$S^{g}_{m}$ directly from the definition above by the Mann-Whitney
U-statistic:
$\widehat{AUC}(h,g|S_{n})=\frac{1}{m^{+}}\frac{1}{m^{-}}\sum_{(\mathbf{x}_{i},1)\in
S^{g}_{m}}\sum_{(\mathbf{x}_{i},0)\in
S^{g}_{m}}\mathbb{1}(h(\mathbf{x}_{i})>h(\mathbf{x}_{j}))$, where $m^{+}$ is
the number of positives in $S^{g}_{m}$, and $m^{-}$ is the number of negatives
in $S^{g}_{m}$. An alternative calculation of $\widehat{AUC}$ is the Area
Under the Receiver Operating Characteristic (ROC) Curve Fawcett, 2006a .

##  4 AUC-selective classification

We consider the AUC-selective classification problem for binary probabilistic
classifiers. We make the further assumption that the parameters of $h$ are
fixed, i.e., $\theta$ in (2) includes only the parameters of the selection
function $g$. Let us denote by $H$ a binary probabilistic classifier induction
algorithm whose hyper-parameters are fixed. The algorithm induces the
classifier $h$ starting from a training set. $h(\mathbf{x})$ is the score
assigned by $h$ to an instance $\mathbf{x}$. We aim to lift $h$ to a selective
classifier $(h,g)$ by calculating a selection function $g$ from the following
hypothesis space (score-bounded selection functions):

$g(\mathbf{x})=\begin{cases}0\quad\text{if }\theta_{l}\leq
h(\mathbf{x})\leq\theta_{u}\\\ 1\quad\text{otherwise}\end{cases}$ (3)

Intuitively, the selective classifier abstains if the score is within the
bounds $\theta_{l}$ and $\theta_{u}$.

To identify the bounds of $g$, we consider the following scenario, where we
have a sample $S_{n}=\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{n}$ and a classifier
$h$. Our objective is to estimate a score-bounded selection function $g$ such
that: (i) $\hat{\phi}(g|S_{n})\geq c$; and (ii): $\widehat{AUC}(h,g|S_{n})$ is
as large as possible. Let us define $n=n^{+}+n^{-}$, with $n^{+}$ (resp.,
$n^{-}$) the number of positive (resp., negative) instances in $S_{n}$. Let
$p^{+}=n^{+}/n$ be the fraction of positives and let $p^{-}=1-p^{+}$ be the
fraction of negatives. Without loss of generality, we can assume that
$\mathbf{x}_{1},\ldots,\mathbf{x}_{n}$ is ordered by non-ascending scores,
i.e., $h(\mathbf{x}_{i})\geq h(\mathbf{x}_{i+1})$ for $i\in[1,n-1]$. The rank
of an instance is $r(\mathbf{x}_{i})=i$, and its relative rank is
$r(\mathbf{x}_{i})/n$. We write $t({\mathbf{x}_{i}})$ to indicate the number
of positive instances occurring in ranks from $1$ to $i$. The true positive
rate (TPR) at $\mathbf{x_{i}}$ is the ratio
$tpr(\mathbf{x}_{i})=t(\mathbf{x}_{i})/n^{+}$. We depart from reasoning
directly on the definition of the AUC. Instead, we consider the linearly
related definition of the Gini coefficient Engelmann, (2011):

$G=2\cdot AUC-1$ (4)

for which the maximization problem is equivalent. The Gini coefficient (also
known as Accuracy Ratio) is defined starting from the Cumulative Accuracy
Profile (CAP), which maps relative rank to TPR. The Gini coefficient can be
written as $G=A/(A+B)$ where $A$ is the area of the CAP between the diagonal
and the line of the classifier $h$, and $B$ is the area between the line of
$h$ and the line of a perfect classifier $h^{\star}$. A perfect classifier
assigns a score of $1$ to positives and a score of $0$ to negatives; hence it
ranks all positives first, then all negatives afterwards. The diagonal line
represents the performance of a random classifier. In summary, the Gini
coefficient measures the discriminative power of a probabilistic classifier as
a fraction of the difference in power between a perfect classifier and a
random classifier. Using the CAP plot and the Gini coefficient will be crucial
in proving our results. Figure 1 shows a sample CAP plot where the axes are
ranks and true positives instead of their relative counterparts. The Gini
coefficient can equivalently be computed as ratio $An^{+}n/(An^{+}n+Bn^{+}n)$.
We observe that $(A+B)n^{+}n$ is $n^{+}n^{+}/2$ (the area of $h^{\star}$ for
positives) plus $n^{-}n^{+}$ (the area of $h^{\star}$ for negatives) minus
$n^{+}n/2$ (the area under the diagonal). Hence $A+B=n^{-}/(2n)=p^{-}/2$.

Let us define $r^{\prime}(\mathbf{x}_{i})=n-r(\mathbf{x}_{i})+1$, i.e.
$r^{\prime}(\mathbf{x}_{i})$ is the rank over the non-descending scores. We
show next a (sufficient) condition to improve $\widehat{AUC}(h,g|S_{n})$ by
abstaining on positive instances.

Figure 1: Sample CAP plot with $n=20$ and $n^{+}=7$. Figure 2: CAP plot before
removing a positive instance.

Figure 3: CAP plot after removing a positive instance.

######  Proposition 1.

For any (number of) positive instance $\mathbf{x}_{i}$ such that
$g(\mathbf{x}_{i})=1$ and:

$\frac{r^{\prime}(\mathbf{x}_{i}}{n}\leq\widehat{AUC}(h,g|S_{n})\cdot p^{-}$
(5)

we have: $\widehat{AUC}(h,g|S_{n})\leq\widehat{AUC}(h,g^{\prime}|S_{n})$,
where $g^{\prime}(\mathbf{x}_{i})=0$ and
$g^{\prime}(\mathbf{x})=g(\mathbf{x})$ otherwise. The inequality is strict if
at least one such $\mathbf{x}_{i}$ exists in $S_{n}$.

###### Proof.

By (4), we can equivalently show the result for the Gini coefficient. Let
$G=A/(A+B)$, and let $\bar{G}=\bar{A}/(\bar{A}+\bar{B})$ be the Gini
coefficient after removing (a.k.a., abstaining on) one positive instance
$\mathbf{x}_{i}$. We have $A+B=n^{-}/(2n)$ and
$\bar{A}+\bar{B}=n^{-}/(2(n-1))$. Then $\bar{G}>G$ iff:

$\bar{A}(n-1)>An$ (6)

As $r^{\prime}(\mathbf{x_{i}})=n-r(\mathbf{x_{i}})+1$, we have that $\bar{A}$
is related to $A$ as follows:

$\bar{A}+\frac{1}{2}=(A+\frac{1}{2}-\left[\frac{(r^{\prime}(\mathbf{x_{i}})-1)+(t(\mathbf{x_{i}})-1)+\nicefrac{{1}}{{2}}}{n^{+}n}\right])\frac{n^{+}n}{(n^{+}-1)(n-1}$

(7)

Using such equality, the condition in (6) can be simplified to (see Appendix
for full derivation):

$\frac{r^{\prime}(\mathbf{x_{i}}}{n}\leq
A+\frac{p^{-}}{2}+\frac{(n^{+}-t(\mathbf{x_{i}})}{n}$

Since $n^{+}\geq t(\mathbf{x_{i}})$, the inequality above is satisfied if:

$\frac{r^{\prime}(\mathbf{x_{i}}}{n}\leq A+\frac{p^{-}}{2}$ (8)

By (4) and $G=A/(A+B)=2A/p^{-}$, we have
$A+\nicefrac{{p^{-}}}{{2}}=\widehat{AUC}(h,g|S_{n})\cdot p^{-}$, and thus the
conclusion (5) holds after removing one positive instance. Let us then
consider the case when we remove $(1-\alpha)n^{+}$ positive instances, with
$\alpha\in[0,1[$. Inequality (8) becomes:

$\frac{r^{\prime}(\mathbf{x_{i}}}{n}\leq A_{\alpha}\frac{(n^{-}+\alpha
n^{+}}{n}+\frac{p^{-}}{2}$ (9)

where $A_{\alpha}$ is the numerator of the Gini fraction after removing
$(1-\alpha)n^{+}$ positives. By repeatedly removing one positive instance at a
time, (6) implies that $A_{\alpha}(n^{-}+\alpha n^{+})>An$. Therefore, if (5)
holds, then (9) holds for any $\alpha$. Therefore, by removing as many as
possible positive instances $\mathbf{x_{i}}$ such that (5) holds, we increase
the Gini coefficient, and, a fortiori, the
$\widehat{AUC}(h,g^{\prime}|S_{n})$. ∎

A key step in the proof is equation (7). Let us give some intuitions.
Abstaining on a positive instance means removing some areas in the CAP plot,
as shown in grey in Figure 2. The vertical grey band consists of
$t(\mathbf{x}_{i})-1$ cells in the $nn^{+}$ grid. The horizontal grey band
consists of $r^{\prime}(\mathbf{x}_{i})-1$ cells. In addition, an area of
$1/2$ is removed from the cell with the increase of the classifier line.
Finally, after removing those areas, the grid is rescaled from $nn^{+}$ to
$(n^{+}-1)(n-1)$, which provides the rescaling factor in (7). Figure 3 shows
the CAP plot after removing a positive instance.

A similar result holds when removing negative examples. The condition is based
on the TPR of the removed instance.

######  Proposition 2.

For any (number of) negative instance $\mathbf{x}_{i}$ such that
$g(\mathbf{x}_{i})=1$ and:

$\frac{t(\mathbf{x_{i}}}{n^{+}}\leq\widehat{AUC}(h,g|S_{n})-\frac{1}{n^{+}}$
(10)

we have: $\widehat{AUC}(h,g|S_{n})\leq\widehat{AUC}(h,g^{\prime}|S_{n})$,
where $g^{\prime}(\mathbf{x}_{i})=0$ and
$g^{\prime}(\mathbf{x})=g(\mathbf{x})$ otherwise. The inequality is strict if
at least one such $\mathbf{x}_{i}$ exists in $S_{n}$.

In general, the selection functions $g^{\prime}$ in Prop. 1 and Prop. 2 are
not score-bounded. Let us assume that $g(\mathbf{x})=1$ for all $\mathbf{x}$,
hence $\widehat{AUC}(h,g|S_{n})=\widehat{AUC}(h|S_{n})$. We can easily lift
condition (5) to scores as follows:

$h(\mathbf{x}_{i})\leq\hat{q}_{n}(\widehat{AUC}(h|S_{n})\cdot
p^{-})=\theta_{u}$

where $\hat{q}_{n}$ is the empirical quantile function over the scores
$\\{h(\mathbf{x}_{i})\ |\ \mathbf{x}_{i}\in S_{n}\\}$. As a consequence,
$g^{\prime}$ is score bounded as it boils down to $g^{\prime}(\mathbf{x})=0$
iff $0\leq h(\mathbf{x})\leq\theta_{u}$. Regarding (10), since TPR is anti-
monotonic with the scores, we can restate the condition as follows:

$\theta_{l}=\hat{q}_{n^{+}}(\widehat{AUC}(h|S_{n})-\nicefrac{{1}}{{n^{+}}})\leq
h(\mathbf{x}_{i})$

where $\hat{q}_{n^{+}}$ is the empirical quantile function over the scores of
the positives $\\{h(\mathbf{x}_{i})\ |\ \mathbf{x}_{i}\in S^{+}_{n^{+}}\\}$,
with $S^{+}_{n^{+}}$ denoting the set of positives in $S_{n}$. In this case,
$g^{\prime}$ is score bounded as $g^{\prime}(\mathbf{x})=0$ iff
$\theta_{l}\leq h(\mathbf{x})\leq 1$.

In summary, if $h(\mathbf{x}_{i})$ is within the bounds $\theta_{l}$ and
$\theta_{u}$, then, irrespective whether $\mathbf{x}_{i}$ is positive or
negative, by abstaining on $\mathbf{x}_{i}$ we obtain an increase in
$\widehat{AUC}$. In summary, we have the following result.

######  Proposition 3.

Called
$\theta_{l}=\hat{q}_{n^{+}}(\widehat{AUC}(h|S_{n})-\nicefrac{{1}}{{n^{+}}})$
and $\theta_{u}=\hat{q}_{n}(\widehat{AUC}(h|S_{n})\cdot p^{-})$, we define
$g(x)$ as:

$g(\mathbf{x})=\begin{cases}0\quad\text{if }\theta_{l}\leq
h(\mathbf{x})\leq\theta_{u}\\\ 1\quad\text{otherwise}.\end{cases}$

Then we have: $\widehat{AUC}(h|S_{n})\leq\widehat{AUC}(h,g|S_{n})$. The
inequality is strict if at least one $\mathbf{x}_{i}$ exists in $S_{n}$ such
that $g(\mathbf{x}_{i})=0$.

##  5 The AUCross algorithm

Alg. 1 shows a procedure to calculate the bounds stated in Prop. 3 starting
from empirical scores and true class labels.

Input : $(\mathbf{s},\mathbf{y}$ \- scores and true class labels

Output : $(\theta_{l},\theta_{u})$ \- bounds for AUC-based selective
classification

1 $n,n^{+},p^{-}\leftarrow|\mathbf{y}|,|\mathbf{y}==1|,1-{n^{+}}/{n}$

2 $\widehat{AUC}\leftarrow AUC.ROC(\mathbf{s},\mathbf{y})$ // compute
empirical AUC

3 $u\leftarrow\lfloor\widehat{AUC}\cdot p^{-}\cdot n\rfloor$ // $\theta_{u}$
position

4 $\mathbf{tpr}\leftarrow 1-cum.sum(\mathbf{y}[order(\mathbf{s})])/n^{+}$ //
compute true positive rates

5 $l\leftarrow{search.sorted}{(\mathbf{tpr},\widehat{AUC}-1/n^{+}}$ //
$\theta_{l}$ position

6 $\mathbf{ss}\leftarrow{sort}(\mathbf{s})$ // sort scores ascending

7 $\theta_{l},\theta_{u}\leftarrow\mathbf{ss}[l],\mathbf{ss}[u]$ // bounds of
$g$

return _$(\theta_{l},\theta_{u})$_

Algorithm 1 EstimateThetasAUC(

We devise an approach for lifting $H$ to an AUC-selective classifier from this
procedure. We call this induction algorithm AUCross. The approach differs from
existing methods in several aspects. First, it aims to determine thresholds
$\theta_{l}$ and $\theta_{u}$ specifically designed for the AUC selective
classification problem (2). Second, we prevent setting apart a validation set
from the training set (as done in state-of-the-art methods) to be able to fit
the classifier $h$ on the whole available training set. Since the goal is to
estimate quantiles over the (unknown) population of scores of the classifier
$h$, we approximate sampling from such a population by using a cross-fitting
strategy, similar to the double ML approach Chernozhukov et al., (2018).
Third, existing state-of-the-art methods calibrate the selection function by
computing the empirical quantile of scores over a validation set. We adopt a
quantile estimator based on subsamples, which improves over the variance of a
full-sample quantile estimator. The theoretical backbone of our approach is
based on the results by Knight and Bassett, (2003).

######  Theorem 4 (Knight and Bassett, (2003), Theorem 3).

Given a random sample distributed according to $F$, satisfying some regularity
conditions, and $K$ non-overlapping subsamples of it, let $\hat{q}(\alpha)$ be
the empirical $\alpha$-quantile estimator of $F$ over the whole sample, and
$\bar{q}(\alpha)$ a weighted average of the empirical quantile estimators of
$F$ over the subsamples. For $t\in[0,1]$, let us define the linear combination
$\tilde{q}(\alpha)=t\hat{q}(\alpha)+(1-t)\bar{q}(\alpha)$. The variance of
$\tilde{q}(\alpha)$ is minimized for $t=\nicefrac{{1}}{{\sqrt{2}}}$, $K=2$ and
equally sized subsamples.

The sample quantile $\hat{q}(\alpha)$ is known to be asymptotically normal. A
weighted averages of subsample quantiles $\bar{q}(\alpha)$ are first-order
equivalent to $\hat{q}(\alpha)$. The above theorem states the conditions for
minimizing the variance of (the second-order term of) any linear combination
of the two estimators.

We report the pseudo-code of AUCross in Alg. 2.

Input : $(\mathbf{X},\mathbf{y}$ \- training set,

$H$ \- binary probabilistic classifier,

$c$ \- target coverage,

$k$ \- number of folds

Output : $(h,g)$ \- selective classifier

1 $\mathit{Lbs},\mathit{Ubs}\leftarrow[],[]$ // empty lists of bounds

2 $S\leftarrow\text{StratifiedKFold}((\mathbf{X},\mathbf{y}),k)$ // stratified
$k$-fold partition

3 for  _$\mathbf{X}_{k},\mathbf{y}_{k}\in S$_ do // for each fold

4
$(\mathbf{X}^{\prime}_{k},\mathbf{y}^{\prime}_{k})=(\mathbf{X}-\mathbf{X}_{k},\mathbf{y}-\mathbf{y}_{k})$
// training data

5 $h_{k}\leftarrow H.fit(\mathbf{X}^{\prime}_{k},\mathbf{y}^{\prime}_{k})$ //
train $k^{th}$ classifier

6 $\mathbf{s}_{k}\leftarrow h_{k}.score(\mathbf{X}_{k})$ // score test data

7 $\mathbf{s}\leftarrow\cup_{k=1}^{K}s_{k}$ // store all the scores

8 $n\leftarrow|\mathbf{y}|$ // number of instances

9
$\theta_{l},\theta_{u}\leftarrow\textit{EstimateThetasAUC}(\mathbf{s},\mathbf{y})$
// bounds for scores over whole training set

10 $\mathcal{I}\leftarrow KFold((\mathbf{s},\mathbf{y}),2)$ // split scores
and actual values in two sub-samples

11 for  _$(\mathbf{s}_{j},\mathbf{y}_{j})\in\mathcal{I}$_ do // for each sub-
sample

12
$\theta_{l_{j}},\theta_{u_{j}}\leftarrow\textit{EstimateThetasAUC}(\mathbf{s}_{j},\mathbf{y}_{j})$

13
$\theta_{u^{*}}\leftarrow\frac{1}{\sqrt{2}}\theta_{u}+(1-\frac{1}{\sqrt{2}})\sum_{i=1}^{2}\theta_{u_{i}}$
// estimate of $\theta_{u}$

14
$\theta_{l^{*}}\leftarrow\frac{1}{\sqrt{2}}\theta_{l}+(1-\frac{1}{\sqrt{2}})\sum_{i=1}^{2}\theta_{l_{i}}$
// estimate of $\theta_{l}$

15 $mid\leftarrow\lfloor n\cdot(\theta_{u^{*}}+\theta_{l^{*}})/2\rfloor$ //
compute mid point position

16 $\delta\leftarrow n\cdot(1-c)/{2}$ // half-width of rejection area

17 $u^{\prime}\leftarrow\min\\{\lfloor mid+\delta\rfloor,n\\}$ // upper bound
position

18 $l^{\prime}\leftarrow\max\\{1,\lfloor mid-\delta\rfloor\\}$ // lower bound
position

19 $\mathbf{ss}\leftarrow{sort}(\mathbf{s})$ // sort scores ascending

20
$\theta_{l^{\prime}},\theta_{u^{\prime}}\leftarrow\mathbf{ss}[l^{\prime}],\mathbf{ss}[u^{\prime}]$
// bounds of $g$

21 $h\leftarrow H.fit(\mathbf{X},\mathbf{y})$ // train classifier

22
$g\leftarrow\text{{lambda}}\;\mathbf{x}:1-\mathbb{1}(\theta_{l^{\prime}}\leq
h.score(\mathbf{x})\leq\theta_{u^{\prime}})$

23 // selection function

return _$(h,g)$_

Algorithm 2 AUCross.fit(

AUCross iterates over a stratified $k$-fold partitioning of the training set
(lines 3-6 of Alg. 2), where, for each of the folds, we use $k-1$ folds to
train a classifier $h_{k}$ (using $H$) and to predict scores
$h_{k}(\mathbf{X}_{k})$ over the remaining fold $k$ (lines 4-5). We store all
these predictions in $\mathbf{s}$ (line 7), and we use them to compute
quantiles according to Prop. 3 (line 9 and Alg. 1). Since $\theta_{l}$ and
$\theta_{u}$ are quantiles estimates, we can improve their second-order
behaviour by randomly splitting the obtained scores and the target variable
$(\mathbf{s},\mathbf{y})$ into two parts (line 9) and repeat the estimation of
quantiles over subsamples (lines 10-12). We combine the estimates according to
Thm. 4 to obtain the final quantiles $\theta_{l^{*}}$ and $\theta_{u^{*}}$
(lines 13-14).

The quantile estimation methods considered so far, and in particular
$\theta_{l}$ and $\theta_{u}$ in Prop. 3, are intended to maximize the AUC
independently of the desired coverage $c$. To address the minimum coverage
constraint $c$, we centre the rejection area at the midpoint between the
instances with score $\theta_{l^{*}}$ and $\theta_{u^{*}}$ (line 15). Such a
mid point $mid$ is the median of the distribution of scores from
$\theta_{l^{*}}$ up to $\theta_{u^{*}}$. We consider a rejection area
$[l^{\prime},u^{\prime}]$ centered in $mid$ and with width $n\cdot(1-c)$, also
checking not to exceed the range $[1,n]$ (lines 16-18). The final selective
classifier $(h,g)$ is then obtained by: (i) fitting $H$ on the whole sample to
get $h$ (line 21); (ii) setting the bounds in the selection function $g$ to
the scores $\theta_{l^{\prime}}$ and $\theta_{u^{\prime}}$ at the boundaries
$l^{\prime}$ and $u^{\prime}$ of the rejection area (line 22).

We point out that the bounds computed by AUCross may be sup-optimal, either
because they do not achieve the minimum target coverage $c$ or do not maximize
the AUC. Even when restricting to the class of score-bounded selection
functions, the reached AUC may be sub-optimal: for instance, we used (8) as a
sufficient but not necessary condition during the proof of Prop. 1. We
investigate the experimental performance of AUCross in the next section, also
concerning such theoretical caveats.

##  6 Experiments

We run experiments on nine real-world datasets (seven tabular datasets and two
image datasets). We split available data instances into 75% for training
selective classifiers and 25% for performance testing. When possible, the
split was based on timestamps to allow for out-of-time validation. Otherwise,
a stratified random split is performed. A summary of the experimental datasets
is reported in Table 1, including the number of features, size for training
and test, and the rate of positives in the test set. For the sake of space, we
detail the pre-processing procedures in the Appendix. We only mention here
that, since we are interested in a binary classification task, we labelled all
the cat images of CIFAR-10-cat as the positive class, while images of other
classes are considered as negatives.

Dataset | # Features | Training Size | Test Size | Positive Rate
---|---|---|---|---
Adult | 55 | 30,162 | 15,060 | .246
LendingClub | 65 | 1,364,697 | 445,912 | .225
GiveMe | 12 | 112,500 | 37,500 | .067
UCICredit | 23 | 22,500 | 7,500 | .221
CSDS1 Barddal et al., (2020) | 155 | 230,409 | 76,939 | .144
CSDS2 Barddal et al., (2020) | 35 | 37,100 | 12,533 | .018
CSDS3 Barddal et al., (2020) | 144 | 71,177 | 23,288 | .253
CatsVsDogs | 64x64 | 20,000 | 5,000 | .500
CIFAR-10-cat | 32x32 | 50,000 | 10,000 | .100

Table 1: Experimental datasets.

Experiments were run on a machine with 96 cores equipped with Intel(R) Xeon(R)
Gold 6342 CPU @ 2.80GHz and two NVIDIA RTX A6000, OS Ubuntu 20.04.4,
programming language Python 3.8.12.

|  | c | Adult | Lending | GiveMe | UCICredit | CSDS1 | CSDS2 | CSDS3 | CatsVsDogs | CIFAR-10-cat
---|---|---|---|---|---|---|---|---|---
.99 | .0003 | .0000 | .0004 | .0008 | .0003 | .0057 | .0001 | .0000 | .0001
.95 | .0011 | .0000 | .0009 | .0008 | .0012 | .0050 | .0005 | .0000 | .0002
.90 | .0022 | .0000 | .0004 | .0007 | .0024 | .0080 | .0011 | .0000 | .0006
.85 | .0033 | .0000 | .0003 | .0013 | .0033 | .0027 | .0017 | .0004 | .0016
.80 | .0033 | .0000 | .0006 | .0007 | .0039 | .0006 | .0020 | .0004 | .0049
.75 | .0037 | .0001 | .0005 | .0001 | .0053 | .0027 | .0023 | .0000 | .0193

Figure 4: AUC by varying bounds: Adult and $c=.90$. |  Table 2: Absolute deviation in empirical AUC of AUCross w.r.t. an Oracle.

####  AUCross vs Oracle

The first experiment evaluates how well the AUCross algorithm approximates the
optimal solution to the problem of maximizing $\widehat{AUC}(h,g|S_{n})$ s.t.
$\hat{\phi}(g|S_{n})\geq c$ over the test set $S_{n}$, considering the family
of score-bounded selection functions. For tabular data, $h$ is fixed to a
LightGBM classifier Ke et al., (2017) with default hyperparameters. For image
data, we used a VGG architecture as a base classifier Simonyan and Zisserman,
(2015), trained with cross-entropy loss. We enumerate all the score-bounded
selection functions $g(\mathbf{x})=0$ iff $\theta_{l}\leq
h(\mathbf{x})\leq\theta_{u}$ by varying the bounds
$\theta_{l},\theta_{u}\in\\{h(\mathbf{x})\ |\ \mathbf{x}\in S_{n}\\}$ such
that $\hat{\phi}(g|S_{n})\geq c$. The Oracle approach chooses the function $g$
which maximizes $\widehat{AUC}(h,g|S_{n})$. It requires knowing the true class
of instances in $S_{n}$; hence it is not feasible in practice. Table 2 shows
the difference between the empirical AUC ($\widehat{AUC}$) achieved by the
Oracle and the one of AUCross on the test sets of the experimental datasets
(the value of the empirical AUC for AUCross is reported in Table 3 and
discussed later on). We notice that AUCross reaches the Oracle performance for
the LendingClub and CatsVsDogs datasets. For the datasets Adult, CSDS1, CSDS3
and CIFAR-10-cat, there are larger violations for smaller $c$’s, while we do
not see such a trend for GiveMe, UCICredit and CSDS2. This suggests that the
strategy of centring the rejection area at a midpoint (line 15 of Alg. 2)
might be sub-optimal. Figure 4 shows the performance over the space of bounds
in a specific case, highlighting the bounds of Oracle and AUCross.

####  AUCross vs baselines

We compare the performance of AUCross to a few baselines. The first one is a
bounded-improvement version of the plug-in rule Herbei and Wegkamp, (2006)
(PlugIn). PlugIn uses softmax as confidence function, i.e., in the binary
case, $g(x)=\mathbb{1}(\max\\{h(x),1-h(x)\\}>\theta)$, and it estimates the
$\theta$ parameter as the $(1-c)$ quantile on a validation set. Such an
approach requires splitting the training set into a dataset for building the
classifier and a validation set for estimating $\theta$ (10% of the original
dataset in our experiments). In contrast, AUCross allows for building the
classifier using the whole training set. Second, we consider two hybrid
versions of AUCross and PlugIn: (1) a plug-in rule specialized for AUC
(PlugInAUC), i.e., that uses a score-bounded selection function by estimating
bounds as in Alg. 1 on a validation set; (2) a cross-fitting version of PlugIn
(SCross), where the selection function is softmax and the estimation of the
bounds exploits the second-order improvement of Thm. 4.

|  | Empirical Coverage | Selective AUC
---|---|---|---
| $c$ |  AUCross |  PlugIn |  PlugInAUC | SCross | SelNet | SAT |  AUCross |  PlugIn |  PlugInAUC | SCross | SelNet | SAT
Adult | .99 |  .989 $\pm$ .001 |  .992 $\pm$ .001 |  .991 $\pm$ .001 | .993 $\pm$ .001 | .985 $\pm$ .001 | .998 $\pm$ .001 |  .929 $\pm$ .003 |  .928 $\pm$ .003 |  .928 $\pm$ .003 | .928 $\pm$ .003 | .901 $\pm$ .003 | .902 $\pm$ .003
.95 |  .950 $\pm$ .002 |  .947 $\pm$ .002 |  .950 $\pm$ .002 | .950 $\pm$ .002 | .954 $\pm$ .002 | .984 $\pm$ .002 |  .935 $\pm$ .003 |  .935 $\pm$ .003 |  .934 $\pm$ .003 | .935 $\pm$ .003 | .907 $\pm$ .003 | .903 $\pm$ .003
.90 |  .896 $\pm$ .003 |  .900 $\pm$ .003 |  .898 $\pm$ .003 | .904 $\pm$ .003 | .897 $\pm$ .003 | .966 $\pm$ .002 |  .943 $\pm$ .002 |  .941 $\pm$ .003 |  .942 $\pm$ .003 | .941 $\pm$ .003 | .917 $\pm$ .003 | .904 $\pm$ .003
.85 |  .849 $\pm$ .003 |  .846 $\pm$ .003 |  .850 $\pm$ .003 | .850 $\pm$ .003 | .772 $\pm$ .004 | .952 $\pm$ .002 |  .950 $\pm$ .002 |  .947 $\pm$ .003 |  .949 $\pm$ .002 | .948 $\pm$ .003 | .848 $\pm$ .005 | .904 $\pm$ .003
.80 |  .797 $\pm$ .004 |  .794 $\pm$ .004 |  .793 $\pm$ .004 | .802 $\pm$ .004 | .793 $\pm$ .004 | .934 $\pm$ .003 |  .958 $\pm$ .002 |  .952 $\pm$ .003 |  .957 $\pm$ .002 | .953 $\pm$ .003 | .924 $\pm$ .004 | .904 $\pm$ .003
.75 |  .750 $\pm$ .004 |  .743 $\pm$ .004 |  .744 $\pm$ .004 | .746 $\pm$ .004 | .736 $\pm$ .004 | .919 $\pm$ .003 |  .963 $\pm$ .002 |  .958 $\pm$ .003 |  .962 $\pm$ .002 | .958 $\pm$ .003 | .917 $\pm$ .004 | .904 $\pm$ .003
Lending | .99 |  .996 $\pm$ .001 |  .995 $\pm$ .001 |  .997 $\pm$ .001 | .994 $\pm$ .001 | .997 $\pm$ .001 | .073 $\pm$ .001 |  .983 $\pm$ .001 |  .984 $\pm$ .001 |  .984 $\pm$ .001 | .983 $\pm$ .001 | .973 $\pm$ .001 | .664 $\pm$ .004
.95 |  .980 $\pm$ .001 |  .970 $\pm$ .001 |  .980 $\pm$ .001 | .969 $\pm$ .001 | .985 $\pm$ .001 | .073 $\pm$ .001 |  .985 $\pm$ .001 |  .985 $\pm$ .001 |  .985 $\pm$ .001 | .984 $\pm$ .001 | .973 $\pm$ .001 | .664 $\pm$ .004
.90 |  .959 $\pm$ .001 |  .939 $\pm$ .001 |  .958 $\pm$ .001 | .938 $\pm$ .001 | .928 $\pm$ .001 | .073 $\pm$ .001 |  .987 $\pm$ .001 |  .986 $\pm$ .001 |  .987 $\pm$ .001 | .986 $\pm$ .001 | .981 $\pm$ .001 | .664 $\pm$ .004
.85 |  .936 $\pm$ .001 |  .909 $\pm$ .001 |  .935 $\pm$ .001 | .908 $\pm$ .001 | .866 $\pm$ .001 | .073 $\pm$ .001 |  .989 $\pm$ .001 |  .988 $\pm$ .001 |  .989 $\pm$ .001 | .987 $\pm$ .001 | .977 $\pm$ .001 | .664 $\pm$ .004
.80 |  .912 $\pm$ .001 |  .880 $\pm$ .001 |  .911 $\pm$ .001 | .879 $\pm$ .001 | .850 $\pm$ .001 | .073 $\pm$ .001 |  .990 $\pm$ .001 |  .989 $\pm$ .001 |  .991 $\pm$ .001 | .989 $\pm$ .001 | .991 $\pm$ .001 | .664 $\pm$ .004
.75 |  .886 $\pm$ .001 |  .850 $\pm$ .001 |  .884 $\pm$ .001 | .849 $\pm$ .001 | .880 $\pm$ .001 | .073 $\pm$ .001 |  .992 $\pm$ .001 |  .990 $\pm$ .001 |  .992 $\pm$ .001 | .990 $\pm$ .001 | .976 $\pm$ .001 | .664 $\pm$ .004
GiveMe | .99 |  .990 $\pm$ .001 |  .990 $\pm$ .001 |  .990 $\pm$ .001 | .990 $\pm$ .001 | .944 $\pm$ .002 | .746 $\pm$ .003 |  .868 $\pm$ .004 |  .861 $\pm$ .004 |  .869 $\pm$ .004 | .859 $\pm$ .004 | .747 $\pm$ .007 | .730 $\pm$ .006
.95 |  .950 $\pm$ .002 |  .949 $\pm$ .002 |  .947 $\pm$ .002 | .950 $\pm$ .002 | .944 $\pm$ .002 | .727 $\pm$ .003 |  .874 $\pm$ .004 |  .824 $\pm$ .006 |  .875 $\pm$ .004 | .826 $\pm$ .006 | .748 $\pm$ .007 | .729 $\pm$ .006
.90 |  .898 $\pm$ .002 |  .903 $\pm$ .002 |  .897 $\pm$ .002 | .902 $\pm$ .002 | .900 $\pm$ .002 | .705 $\pm$ .003 |  .883 $\pm$ .004 |  .784 $\pm$ .007 |  .883 $\pm$ .004 | .784 $\pm$ .007 | .708 $\pm$ .008 | .728 $\pm$ .007
.85 |  .848 $\pm$ .002 |  .856 $\pm$ .002 |  .849 $\pm$ .002 | .852 $\pm$ .002 | .848 $\pm$ .002 | .699 $\pm$ .003 |  .890 $\pm$ .004 |  .759 $\pm$ .008 |  .890 $\pm$ .004 | .757 $\pm$ .008 | .658 $\pm$ .009 | .728 $\pm$ .007
.80 |  .795 $\pm$ .003 |  .807 $\pm$ .003 |  .801 $\pm$ .003 | .800 $\pm$ .003 | .802 $\pm$ .003 | .694 $\pm$ .003 |  .898 $\pm$ .004 |  .744 $\pm$ .009 |  .897 $\pm$ .004 | .743 $\pm$ .009 | .677 $\pm$ .009 | .727 $\pm$ .007
.75 |  .748 $\pm$ .003 |  .756 $\pm$ .003 |  .750 $\pm$ .003 | .748 $\pm$ .003 | .754 $\pm$ .003 | .689 $\pm$ .003 |  .904 $\pm$ .004 |  .738 $\pm$ .010 |  .904 $\pm$ .004 | .734 $\pm$ .010 | .676 $\pm$ .010 | .727 $\pm$ .007
UCICredit | .99 |  .988 $\pm$ .002 |  .993 $\pm$ .002 |  .993 $\pm$ .001 | .993 $\pm$ .001 | .989 $\pm$ .002 | 1.000$\pm$ .000 |  .772 $\pm$ .008 |  .772 $\pm$ .007 |  .774 $\pm$ .007 | .770 $\pm$ .008 | .765 $\pm$ .007 | .768 $\pm$ .007
.95 |  .944 $\pm$ .003 |  .948 $\pm$ .003 |  .946 $\pm$ .003 | .951 $\pm$ .003 | .909 $\pm$ .004 | 1.000$\pm$ .000 |  .778 $\pm$ .008 |  .769 $\pm$ .008 |  .780 $\pm$ .007 | .765 $\pm$ .008 | .718 $\pm$ .009 | .768 $\pm$ .007
.90 |  .896 $\pm$ .004 |  .897 $\pm$ .004 |  .900 $\pm$ .004 | .906 $\pm$ .004 | .904 $\pm$ .004 | .994 $\pm$ .001 |  .785 $\pm$ .008 |  .758 $\pm$ .008 |  .787 $\pm$ .008 | .754 $\pm$ .009 | .707 $\pm$ .009 | .769 $\pm$ .007
.85 |  .840 $\pm$ .005 |  .851 $\pm$ .005 |  .846 $\pm$ .005 | .848 $\pm$ .005 | .855 $\pm$ .005 | .994 $\pm$ .001 |  .792 $\pm$ .008 |  .746 $\pm$ .009 |  .795 $\pm$ .008 | .734 $\pm$ .009 | .668 $\pm$ .009 | .769 $\pm$ .007
.80 |  .789 $\pm$ .005 |  .803 $\pm$ .005 |  .797 $\pm$ .005 | .797 $\pm$ .005 | .810 $\pm$ .005 | .994 $\pm$ .001 |  .800 $\pm$ .008 |  .722 $\pm$ .010 |  .802 $\pm$ .008 | .710 $\pm$ .010 | .644 $\pm$ .010 | .769 $\pm$ .007
.75 |  .737 $\pm$ .006 |  .751 $\pm$ .006 |  .739 $\pm$ .006 | .750 $\pm$ .005 | .759 $\pm$ .005 | .994 $\pm$ .001 |  .809 $\pm$ .008 |  .695 $\pm$ .011 |  .810 $\pm$ .008 | .679 $\pm$ .011 | .620 $\pm$ .011 | .769 $\pm$ .007
CSDS1 | .99 |  .990 $\pm$ .001 |  .980 $\pm$ .001 |  .991 $\pm$ .001 | .980 $\pm$ .001 | .985 $\pm$ .001 | .984 $\pm$ .001 |  .686 $\pm$ .003 |  .675 $\pm$ .003 |  .686 $\pm$ .003 | .675 $\pm$ .003 | .671 $\pm$ .003 | .669 $\pm$ .003
.95 |  .950 $\pm$ .001 |  .918 $\pm$ .001 |  .949 $\pm$ .001 | .917 $\pm$ .001 | .924 $\pm$ .001 | .937 $\pm$ .001 |  .689 $\pm$ .003 |  .654 $\pm$ .004 |  .689 $\pm$ .003 | .655 $\pm$ .004 | .652 $\pm$ .004 | .654 $\pm$ .004
.90 |  .898 $\pm$ .002 |  .853 $\pm$ .002 |  .898 $\pm$ .002 | .849 $\pm$ .002 | .863 $\pm$ .002 | .861 $\pm$ .002 |  .693 $\pm$ .003 |  .640 $\pm$ .004 |  .694 $\pm$ .003 | .641 $\pm$ .004 | .638 $\pm$ .004 | .635 $\pm$ .004
.85 |  .849 $\pm$ .002 |  .792 $\pm$ .002 |  .845 $\pm$ .002 | .790 $\pm$ .002 | .800 $\pm$ .002 | .803 $\pm$ .002 |  .697 $\pm$ .003 |  .631 $\pm$ .004 |  .698 $\pm$ .003 | .631 $\pm$ .004 | .630 $\pm$ .004 | .625 $\pm$ .004
.80 |  .799 $\pm$ .002 |  .737 $\pm$ .002 |  .795 $\pm$ .002 | .736 $\pm$ .002 | .740 $\pm$ .002 | .739 $\pm$ .002 |  .702 $\pm$ .004 |  .624 $\pm$ .004 |  .703 $\pm$ .003 | .620 $\pm$ .004 | .618 $\pm$ .004 | .616 $\pm$ .004
.75 |  .748 $\pm$ .002 |  .681 $\pm$ .002 |  .747 $\pm$ .002 | .682 $\pm$ .002 | .686 $\pm$ .002 | .687 $\pm$ .002 |  .706 $\pm$ .004 |  .613 $\pm$ .005 |  .707 $\pm$ .004 | .616 $\pm$ .005 | .609 $\pm$ .005 | .608 $\pm$ .005
CSDS2 | .99 |  .990 $\pm$ .001 |  .984 $\pm$ .002 |  .991 $\pm$ .001 | .983 $\pm$ .002 | .980 $\pm$ .002 | 1.000$\pm$ .000 |  .616 $\pm$ .020 |  .585 $\pm$ .020 |  .587 $\pm$ .020 | .616 $\pm$ .020 | .607 $\pm$ .020 | .623 $\pm$ .019
.95 |  .952 $\pm$ .002 |  .927 $\pm$ .003 |  .954 $\pm$ .002 | .916 $\pm$ .003 | .894 $\pm$ .003 | 1.000$\pm$ .000 |  .619 $\pm$ .020 |  .572 $\pm$ .021 |  .589 $\pm$ .020 | .581 $\pm$ .022 | .587 $\pm$ .022 | .623 $\pm$ .019
.90 |  .904 $\pm$ .003 |  .851 $\pm$ .004 |  .907 $\pm$ .003 | .834 $\pm$ .004 | .805 $\pm$ .004 | .880 $\pm$ .003 |  .620 $\pm$ .020 |  .566 $\pm$ .023 |  .590 $\pm$ .021 | .570 $\pm$ .023 | .560 $\pm$ .024 | .588 $\pm$ .022
.85 |  .860 $\pm$ .004 |  .773 $\pm$ .004 |  .861 $\pm$ .004 | .761 $\pm$ .004 | .712 $\pm$ .005 | .817 $\pm$ .004 |  .631 $\pm$ .021 |  .552 $\pm$ .024 |  .590 $\pm$ .021 | .572 $\pm$ .025 | .562 $\pm$ .026 | .589 $\pm$ .023
.80 |  .811 $\pm$ .004 |  .715 $\pm$ .005 |  .818 $\pm$ .004 | .687 $\pm$ .005 | .663 $\pm$ .005 | .756 $\pm$ .004 |  .631 $\pm$ .021 |  .532 $\pm$ .025 |  .592 $\pm$ .021 | .555 $\pm$ .026 | .547 $\pm$ .027 | .582 $\pm$ .024
.75 |  .760 $\pm$ .004 |  .651 $\pm$ .005 |  .776 $\pm$ .004 | .620 $\pm$ .005 | .564 $\pm$ .005 | .697 $\pm$ .005 |  .635 $\pm$ .021 |  .536 $\pm$ .027 |  .592 $\pm$ .021 | .541 $\pm$ .028 | .579 $\pm$ .029 | .575 $\pm$ .026
CSDS3 | .99 |  .991 $\pm$ .001 |  .992 $\pm$ .001 |  .992 $\pm$ .001 | .991 $\pm$ .001 | .988 $\pm$ .001 | .992 $\pm$ .001 |  .851 $\pm$ .003 |  .850 $\pm$ .003 |  .850 $\pm$ .003 | .850 $\pm$ .003 | .844 $\pm$ .003 | .844 $\pm$ .003
.95 |  .948 $\pm$ .002 |  .957 $\pm$ .002 |  .960 $\pm$ .002 | .955 $\pm$ .002 | .942 $\pm$ .002 | .951 $\pm$ .002 |  .858 $\pm$ .003 |  .852 $\pm$ .003 |  .855 $\pm$ .003 | .853 $\pm$ .003 | .851 $\pm$ .003 | .849 $\pm$ .003
.90 |  .901 $\pm$ .002 |  .909 $\pm$ .002 |  .909 $\pm$ .002 | .911 $\pm$ .002 | .913 $\pm$ .002 | .902 $\pm$ .002 |  .865 $\pm$ .003 |  .855 $\pm$ .003 |  .863 $\pm$ .003 | .856 $\pm$ .004 | .850 $\pm$ .004 | .855 $\pm$ .004
.85 |  .849 $\pm$ .003 |  .859 $\pm$ .003 |  .856 $\pm$ .003 | .865 $\pm$ .003 | .867 $\pm$ .003 | .856 $\pm$ .003 |  .873 $\pm$ .003 |  .857 $\pm$ .004 |  .871 $\pm$ .003 | .858 $\pm$ .004 | .851 $\pm$ .004 | .858 $\pm$ .004
.80 |  .805 $\pm$ .003 |  .808 $\pm$ .003 |  .811 $\pm$ .003 | .818 $\pm$ .003 | .796 $\pm$ .003 | .819 $\pm$ .003 |  .880 $\pm$ .003 |  .859 $\pm$ .004 |  .878 $\pm$ .003 | .859 $\pm$ .004 | .859 $\pm$ .004 | .858 $\pm$ .004
.75 |  .759 $\pm$ .003 |  .758 $\pm$ .003 |  .765 $\pm$ .003 | .769 $\pm$ .003 | .765 $\pm$ .003 | .776 $\pm$ .003 |  .887 $\pm$ .003 |  .858 $\pm$ .004 |  .884 $\pm$ .003 | .859 $\pm$ .004 | .843 $\pm$ .004 | .853 $\pm$ .004
CatsVsDogs | .99 |  .992 $\pm$ .002 |  .990 $\pm$ .002 |  .991 $\pm$ .002 | .992 $\pm$ .002 | .991 $\pm$ .002 | .993 $\pm$ .002 |  .984 $\pm$ .003 |  .985 $\pm$ .002 |  .985 $\pm$ .002 | .985 $\pm$ .002 | .979 $\pm$ .003 | .985 $\pm$ .002
.95 |  .962 $\pm$ .003 |  .965 $\pm$ .003 |  .954 $\pm$ .004 | .959 $\pm$ .003 | .954 $\pm$ .003 | .959 $\pm$ .003 |  .986 $\pm$ .003 |  .985 $\pm$ .002 |  .987 $\pm$ .002 | .987 $\pm$ .003 | .987 $\pm$ .002 | .987 $\pm$ .002
.90 |  .920 $\pm$ .004 |  .910 $\pm$ .004 |  .907 $\pm$ .005 | .917 $\pm$ .004 | .891 $\pm$ .005 | .910 $\pm$ .005 |  .987 $\pm$ .003 |  .984 $\pm$ .002 |  .988 $\pm$ .002 | .988 $\pm$ .003 | .990 $\pm$ .002 | .989 $\pm$ .002
.85 |  .883 $\pm$ .005 |  .854 $\pm$ .005 |  .858 $\pm$ .005 | .871 $\pm$ .005 | .852 $\pm$ .006 | .862 $\pm$ .005 |  .987 $\pm$ .003 |  .984 $\pm$ .002 |  .989 $\pm$ .002 | .988 $\pm$ .003 | .990 $\pm$ .002 | .990 $\pm$ .002
.80 |  .841 $\pm$ .006 |  .797 $\pm$ .006 |  .805 $\pm$ .006 | .807 $\pm$ .006 | .809 $\pm$ .006 | .818 $\pm$ .006 |  .987 $\pm$ .003 |  .985 $\pm$ .002 |  .990 $\pm$ .002 | .989 $\pm$ .003 | .991 $\pm$ .002 | .990 $\pm$ .002
.75 |  .781 $\pm$ .006 |  .740 $\pm$ .007 |  .749 $\pm$ .007 | .744 $\pm$ .007 | .765 $\pm$ .007 | .768 $\pm$ .007 |  .987 $\pm$ .003 |  .983 $\pm$ .002 |  .990 $\pm$ .002 | .988 $\pm$ .003 | .992 $\pm$ .002 | .990 $\pm$ .002
CIFAR10-CAT | .99 |  1.000$\pm$ .001 |  .991 $\pm$ .001 |  .989 $\pm$ .002 | .992 $\pm$ .001 | .992 $\pm$ .001 | .990 $\pm$ .002 |  .890 $\pm$ .009 |  .915 $\pm$ .007 |  .909 $\pm$ .008 | .922 $\pm$ .007 | .896 $\pm$ .008 | .893 $\pm$ .008
.95 |  .998 $\pm$ .001 |  .950 $\pm$ .003 |  .948 $\pm$ .003 | .953 $\pm$ .003 | .956 $\pm$ .003 | .952 $\pm$ .003 |  .889 $\pm$ .009 |  .895 $\pm$ .009 |  .911 $\pm$ .007 | .906 $\pm$ .009 | .902 $\pm$ .008 | .863 $\pm$ .011
.90 |  .994 $\pm$ .001 |  .899 $\pm$ .004 |  .898 $\pm$ .003 | .903 $\pm$ .003 | .904 $\pm$ .003 | .913 $\pm$ .003 |  .889 $\pm$ .009 |  .896 $\pm$ .009 |  .913 $\pm$ .007 | .851 $\pm$ .013 | .899 $\pm$ .010 | .760 $\pm$ .017
.85 |  .987 $\pm$ .002 |  .849 $\pm$ .004 |  .851 $\pm$ .004 | .886 $\pm$ .004 | .863 $\pm$ .004 | .854 $\pm$ .004 |  .888 $\pm$ .009 |  .898 $\pm$ .009 |  .916 $\pm$ .007 | .815 $\pm$ .016 | .676 $\pm$ .022 | .574 $\pm$ .024
.80 |  .969 $\pm$ .002 |  .796 $\pm$ .005 |  .799 $\pm$ .004 | .882 $\pm$ .004 | .806 $\pm$ .005 | .803 $\pm$ .005 |  .885 $\pm$ .009 |  .906 $\pm$ .008 |  .919 $\pm$ .007 | .804 $\pm$ .017 | .359 $\pm$ .026 | .572 $\pm$ .024
.75 |  .945 $\pm$ .003 |  .749 $\pm$ .005 |  .749 $\pm$ .005 | .879 $\pm$ .004 | .763 $\pm$ .005 | .759 $\pm$ .005 |  .870 $\pm$ .011 |  .905 $\pm$ .008 |  .921 $\pm$ .007 | .797 $\pm$ .018 | .408 $\pm$ .028 | .575 $\pm$ .025
| # |  21/54 |  12/54 |  8/54 | 15/54 | 8/54 | 1/54 |  28/54 |  4/54 |  29/54 | 6/54 | 5/54 | 2/54
| $V$ |  .026 $\pm$ .046 |  .021 $\pm$ .029 |  .013 $\pm$ .026 | .028 $\pm$ .036 | .029 $\pm$ .041 | .171 $\pm$ .251 |  |  |  |  |  |

Table 3: Performance metrics (1,000 bootstrap runs over the test set, results
as mean $\pm$ stdev). $V$ (for violation) is the absolute difference between
the empirical coverage and the target coverage $c$.

These two variants allow us to evaluate the relative contributions of the two
key elements of our approach: the estimation of AUC-specific bounds (Prop. 3),
and the second-order quantile estimation strategy (Thm. 4). We also consider
two state-of-the-art methods for selective classification: SelectiveNet
Geifman and El-Yaniv, (2019) (SelNet), and the Self Adaptive Training method
Huang et al., (2020) (SAT). The method from Shen et al., (2020) is not
included in the baselines since it rejects the comparison of pairs of
instances, while we focus on the rejection of predictions on instancexamples
in isolation.

For tabular data, SelNet and SAT are built using a ResNet structure Gorishniy
et al., (2021), while AUCross, PlugIn, PlugInAUC and SCross are based on a
LightGBM classifier. We report in the Appendix results for other base
classifiers. For CatsVsDogs, a VGG architecture Simonyan and Zisserman, (2015)
is used for all the methods. For all the DNN approaches, we set 300 epochs in
training, Stochastic Gradient Descent as an optimizer, a learning rate of $.1$
decreased by a factor $.5$ every $25$ epochs, as in the original papers. All
the parameters of base classifiers are left as the default ones. See the
Appendix for details.

Table 3 shows performance metrics (mean $\pm$ stdev) evaluated on 1,000
bootstrap runs of the test set: the empirical coverage $\hat{\phi}(g|S_{n})$
and the empirical AUC $\widehat{AUC}(h,g|S_{n})$. Resorting to bootstrap
allows for calculating standard errors of the metrics111Other approaches
include confidence intervals for AUC DeLong et al., (1988); Cortes and Mohri,
(2004); Goncalves et al., (2014) or direct estimation of its variance Cleves,
(2002). Rajkomar et al., (2018). Additional results on the selective accuracy
and the positive rate222Positive rate in the accepted region, compared to the
positive rate in the overall test set, measures fairness of the selection
function w.r.t. the class labels. metrics are reported in the Appendix.
Results in Table 3 show that AUCross achieves an empirical coverage close to
the target $c$ in most of the cases for tabular data. The most significant
violation occurs for LendingClub, where also all the other methods fail to
reach the target coverage. In particular, SAT performs poorly on such a
dataset. The other largest violation occurs for CIFAR-10-cat. Here, also the
other cross-fitting-based algorithm (SCross) fails to reach the target one.
Interestingly, the runner-up method w.r.t. coverage is PlugInAUC, which has
the smallest average violation. A striking case is the CSDS2 dataset, which is
the most imbalanced one with a positive rate of 1.8%. Here, both AUCross and
PlugInAUC have small violations, while all other approaches have larger ones.
We argue that this is due to the bounds determined by Prop. 1 and Prop. 2,
which are specific for positives and negatives, respectively, while the other
methods calibrate the selection function independently of the class labels.
Regarding the empirical AUC, AUCross and PlugInAUC outperform the other
methods in most cases. For unbalanced datasets such as for GiveMe, CSDS1 and
CSDS2, the empirical AUC drops for smaller coverages, contrarily to what is
expected. Such behaviour does not occur for those methods that estimate bounds
specifically for the AUC, i.e. AUCross and PlugInAUC. We show in the Appendix
that a trade-off exists in terms of selective accuracy, as both AUCross and
PlugInAUC are outperformed concerning the accuracy over the accepted region.
This is not surprising since they are optimizing the AUC metric, and it is
well-known that accuracy and AUC cannot be jointly optimized Cortes and Mohri,
(2003). The results in the Appendix also show that optimizing accuracy
compromises the positive rate, while our approach is fairer in this sense.

Finally, consider the running time performances (see the Appendix for
details). AUCross and SCross require $k$ executions of the base classifier
(loop at lines 3-6 of Alg. 2), while all other baselines train a single
(selective) classifier. On tabular data, using efficient base classifiers
results in a reasonably low total running time. Those two strategies are
computationally expensive for image data, where DNN base classifiers are
adopted. However, we observe that PlugInAUC exhibits performances comparable
to SelNet and SAT for CatsVsDogs, and it outperforms them in the case of
CIFAR-10-cat. In summary, we recommend using AUCross for tabular datasets and
PlugInAUC for image datasets.

##  7 Conclusions

Selective classification can help prevent poor or even harmful decisions by
abstaining from making an output, possibly demanding the decision to a human.
We have extended the selective classification framework to a widely used
classifier evaluation metric, the Area Under the ROC Curve. Through an
analytic characterization, we devised methods for computing an estimator of
the best score-bounded selection function. These methods are effective and
outperform existing approaches designed for optimizing accuracy.

Limitations that require future work include the following. First, the
approach should be extended to multi-class classification, for which the
notion of AUC has been considered, e.g., the Volume Under the Surface Ferri et
al., (2003) or the average of pairwise binary AUCs (the M metric) Hand and
Till, (2001). Second, since selective classification might amplify unfair
decisions Jones et al., (2021), we intend to study the behaviour of AUCross
with respect to fair base classifiers and to extend our bound estimators to
account for fairness requirements. Third, as suggested by our experimental
results, we could better determine the bounds for a target coverage $c$,
reconsidering the choice of centring the rejection area in the midpoint of the
bounds of Prop. 3. As shown in Table 2, this is especially relevant for large
$c$’s. Lastly, AUC-based selective classification can be extended to the
metric of weighted AUC Fawcett, 2006b ; Keilwagen et al., (2014), where
instances are weighted by importance.

Reproducibility. Data and anonymized source code can be downloaded from here.

## References

  * Barddal et al., (2020) Barddal, J. P., Loezer, L., Enembreck, F., and Lanzuolo, R. (2020).  Lessons learned from data stream classification applied to credit scoring.  Expert Syst. Appl., 162:113899.
  * Chernozhukov et al., (2018) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018).  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal, 21(1):C1–C68.
  * Chow, (1970) Chow, C. K. (1970).  On optimum recognition error and reject tradeoff.  IEEE Trans. Inf. Theory, 16(1):41–46.
  * Cleves, (2002) Cleves, M. A. (2002).  Comparative assessment of three common algorithms for estimating the variance of the area under the nonparametric receiver operating characteristic curve.  The Stata Journal, 2:280–289.
  * Cortes et al., (2016) Cortes, C., DeSalvo, G., and Mohri, M. (2016).  Boosting with abstention.  In NIPS, pages 1660–1668.
  * Cortes and Mohri, (2003) Cortes, C. and Mohri, M. (2003).  AUC optimization vs. error rate minimization.  In NIPS, pages 313–320. MIT Press.
  * Cortes and Mohri, (2004) Cortes, C. and Mohri, M. (2004).  Confidence intervals for the area under the ROC curve.  In NIPS, pages 305–312.
  * DeLong et al., (1988) DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988).  Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach.  Biometrics, 44(3):837–845.
  * Dua and Graff, (2017) Dua, D. and Graff, C. (2017).  UCI machine learning repository.
  * El-Yaniv and Wiener, (2010) El-Yaniv, R. and Wiener, Y. (2010).  On the foundations of noise-free selective classification.  J. Mach. Learn. Res., 11:1605–1641.
  * Engelmann, (2011) Engelmann, B. (2011).  Measures of a rating’s discriminative power: Applications and limitations.  In Engelmann, B. and Rauhmeier, R., editors, The Basel II Risk Parameters, pages 269–291. Springer, 2 edition.
  * (12) Fawcett, T. (2006a).  An introduction to ROC analysis.  Pattern Recognit. Lett., 27(8):861–874.
  * (13) Fawcett, T. (2006b).  ROC graphs with instance-varying costs.  Pattern Recognit. Lett., 27(8):882–891.
  * Ferri et al., (2003) Ferri, C., Hernández-Orallo, J., and Salido, M. A. (2003).  Volume under the ROC surface for multi-class problems.  In ECML, volume 2837 of Lecture Notes in Computer Science, pages 108–120. Springer.
  * Franc and Průša, (2019) Franc, V. and Průša, D. (2019).  On discriminative learning of prediction uncertainty.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 1963–1971. PMLR.
  * Fumera and Roli, (2002) Fumera, G. and Roli, F. (2002).  Support vector machines with embedded reject option.  In SVM, volume 2388 of Lecture Notes in Computer Science, pages 68–82. Springer.
  * Geifman and El-Yaniv, (2017) Geifman, Y. and El-Yaniv, R. (2017).  Selective classification for deep neural networks.  In NIPS, pages 4878–4887.
  * Geifman and El-Yaniv, (2019) Geifman, Y. and El-Yaniv, R. (2019).  Selectivenet: A deep neural network with an integrated reject option.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2151–2159. PMLR.
  * Goncalves et al., (2014) Goncalves, L., Subtil, A., Oliveira, M. R., and de Zea Bermudez, P. (2014).  ROC curve estimation: An overview.  Revstat - Statistical Journal, 12:1–20.
  * Gorishniy et al., (2021) Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. (2021).  Revisiting deep learning models for tabular data.  In NeurIPS, pages 18932–18943.
  * Hand and Till, (2001) Hand, D. J. and Till, R. J. (2001).  A simple generalisation of the area under the ROC curve for multiple class classification problems.  Mach. Learn., 45(2):171–186.
  * Hendrickx et al., (2021) Hendrickx, K., Perini, L., der Plas, D. V., Meert, W., and Davis, J. (2021).  Machine learning with a reject option: A survey.  CoRR, abs/2107.11277.
  * Herbei and Wegkamp, (2006) Herbei, R. and Wegkamp, M. H. (2006).  Classification with reject option.  Can. J. Stat., 34(4):709–721.
  * Huang and Ling, (2005) Huang, J. and Ling, C. X. (2005).  Using AUC and accuracy in evaluating learning algorithms.  IEEE Trans. Knowl. Data Eng., 17(3):299–310.
  * Huang et al., (2020) Huang, L., Zhang, C., and Zhang, H. (2020).  Self-adaptive training: beyond empirical risk minimization.  In NeurIPS.
  * Jones et al., (2021) Jones, E., Sagawa, S., Koh, P. W., Kumar, A., and Liang, P. (2021).  Selective classification can magnify disparities across groups.  In ICLR. OpenReview.net.
  * Ke et al., (2017) Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T. (2017).  Lightgbm: A highly efficient gradient boosting decision tree.  In NIPS, pages 3146–3154.
  * Keilwagen et al., (2014) Keilwagen, J., Grosse, I., and Grau, J. (2014).  Area under precision-recall curves for weighted and unweighted data.  PLoS ONE, 9(3):e92209.
  * Knight and Bassett, (2003) Knight, K. and Bassett, G. W. (2003).  Second order improvements of sample quantiles using subsamples.  Unpublished manuscript, http://www.utstat.toronto.edu/keith/papers/subsample.pdf.
  * Krizhevsky, (2009) Krizhevsky, A. (2009).  Learning multiple layers of features from tiny images.  Technical report.
  * Liu et al., (2019) Liu, Z., Wang, Z., Liang, P. P., Salakhutdinov, R., Morency, L., and Ueda, M. (2019).  Deep gamblers: Learning to abstain with portfolio theory.  In NeurIPS, pages 10622–10632.
  * Pietraszek, (2005) Pietraszek, T. (2005).  Optimizing abstaining classifiers using ROC analysis.  In ICML, volume 119 of ACM International Conference Proceeding Series, pages 665–672. ACM.
  * Rajkomar et al., (2018) Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018).  Scalable and accurate deep learning with electronic health records.  NPJ Digital Medicine, 1(1):1–10.
  * Shen et al., (2020) Shen, S., Yang, B., and Gao, W. (2020).  AUC optimization with a reject option.  In AAAI, pages 5684–5691. AAAI Press.
  * Simonyan and Zisserman, (2015) Simonyan, K. and Zisserman, A. (2015).  Very deep convolutional networks for large-scale image recognition.  In ICLR.
  * Tortorella, (2005) Tortorella, F. (2005).  A ROC-based reject rule for dichotomizers.  Pattern Recognit. Lett., 26(2):167–180.
  * Wu and Lee, (2014) Wu, Y.-C. and Lee, W.-C. (2014).  Alternative performance measures for prediction models.  PLoS ONE, 9(3):e91249.
  * Yang and Ying, (2023) Yang, T. and Ying, Y. (2023).  AUC maximization in the era of big data and AI: A survey.  ACM Computing Surveys, To appear.
  * Yeh and Lien, (2009) Yeh, I. and Lien, C. (2009).  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients.  Expert Syst. Appl., 36(2):2473–2480.

##  Appendix A Appendix

###  A.1 Proofs

Proposition 1.

###### Proof.

The missing part of the proof consists of the following equivalence:

$\displaystyle\bar{G}>G$ iff $\displaystyle\bar{A}(n-1)>An$ iff
$\displaystyle\\{\mbox{\ by}\ (8)\ \\}$ iff
$\displaystyle((A+\frac{1}{2}-\left[\frac{(r^{\prime}(\mathbf{x_{i}})-1)+(t(\mathbf{x_{i}})-1)+\nicefrac{{1}}{{2}}}{n^{+}n}\right])\frac{n^{+}n}{(n^{+}-1)(n-1}-\frac{1}{2})(n-1)>An$
iff
$\displaystyle(A+\frac{1}{2}-\frac{(r^{\prime}(\mathbf{x_{i}})+t(\mathbf{x_{i}})-1.5}{n^{+}n})\frac{n^{+}n}{(n^{+}-1}-\frac{(n-1}{2}>An$
iff $\displaystyle
A\frac{n^{+}n}{(n^{+}-1}+\frac{n^{+}n}{2(n^{+}-1}-\frac{(r^{\prime}(\mathbf{x_{i}})+t(\mathbf{x_{i}})-1.5}{(n^{+}-1}-\frac{(n-1}{2}>An$
iff $\displaystyle
A\frac{n}{(n^{+}-1}+\frac{n^{+}n}{2(n^{+}-1}-\frac{(r^{\prime}(\mathbf{x_{i}})+t(\mathbf{x_{i}})-1.5}{(n^{+}-1}-\frac{(n-1}{2}>0$
iff $\displaystyle
A\frac{n}{(n^{+}-1}+\frac{n^{+}n-(n-1)(n^{+}-1}{2(n^{+}-1}-\frac{(r^{\prime}(\mathbf{x_{i}})+t(\mathbf{x_{i}})-1.5}{(n^{+}-1}>0$
iff $\displaystyle
An+\frac{n+n^{+}-1}{2}-(r^{\prime}(\mathbf{x_{i}})+t(\mathbf{x_{i}})-1.5)>0$
iff $\displaystyle
An+\frac{n+n^{+}}{2}-t(\mathbf{x_{i}})>r^{\prime}(\mathbf{x_{i}})-1$ iff
$\displaystyle An+\frac{n+n^{+}}{2}-t(\mathbf{x_{i}})\geq
r^{\prime}(\mathbf{x_{i}}$ iff $\displaystyle
An+\frac{n-n^{+}}{2}+(n^{+}-t(\mathbf{x_{i}}))+\geq
r^{\prime}(\mathbf{x_{i}})$ iff
$\displaystyle\frac{r^{\prime}(\mathbf{x_{i}}}{n}\leq
A+\frac{p^{-}}{2}+\frac{(n^{+}-t(\mathbf{x_{i}})}{n}$

∎

Proposition 2.

###### Proof.

We can equivalently show the result for the Gini coefficient. Let $G=A/(A+B)$,
and let $\bar{G}=\bar{A}/(\bar{A}+\bar{B})$ be the Gini coefficient after
removing (a.k.a., abstaining on) one negative instance $\mathbf{x}_{i}$. We
have $\bar{A}+\bar{B}=(n^{-}-1)/(2(n-1))$. Then $\bar{G}>G$ iff

$\bar{A}(n-1)n^{-}>An(n^{-}-1)$ (11)

As in Proposition 1, we can link the area $\bar{A}$ after the removal of the
negative instance $\mathbf{x}_{i}$ to the original area $A$ as follows:

$\bar{A}+\frac{1}{2}=(A+\frac{1}{2}-\frac{t(\mathbf{x}_{i}}{n^{+}n})\frac{n}{n-1}$

As for the positive case, this can be intuitively understood by looking at
Figure 5, where the grey area highlights the loss from the original CAP plot
after we remove the negative instance. Then we rescale to account for the new
number of instances, as in Figure 6. Then:

$\displaystyle\bar{G}>G$ iff
$\displaystyle((A+\frac{1}{2}-\frac{t(\mathbf{x}_{i}}{n^{+}n})\frac{n}{n-1}-\frac{1}{2})(n-1)n^{-}>An(n^{-}-1)$
iff
$\displaystyle(A+\frac{1}{2}-\frac{t(\mathbf{x}_{i}}{n^{+}n})nn^{-}-\frac{(n-1)n^{-}}{2}>An(n^{-}-1$
iff $\displaystyle
Ann^{-}+\frac{nn^{-}}{2}-\frac{t(\mathbf{x}_{i}}{n^{+}}n^{-}-\frac{(n-1)n^{-}}{2}>Ann^{-}-An$
iff $\displaystyle
An+\frac{nn^{-}}{2}-\frac{(n-1)n^{-}}{2}>\frac{t(\mathbf{x}_{i}}{n^{+}}n^{-}$
iff $\displaystyle
A\frac{n}{n^{-}}+\frac{1}{2}>\frac{t(\mathbf{x}_{i}}{n^{+}}$ iff
$\displaystyle
A\frac{n}{n^{-}}+\frac{1}{2}-\frac{1}{n^{+}}\geq\frac{t(\mathbf{x}_{i}}{n^{+}}$

Since
$\widehat{AUC}(h,g^{\prime}|S_{n})=(G+1)/2=(2A/p^{-}+1)/2=A\nicefrac{{n}}{{n^{-}}}+\nicefrac{{1}}{{2}}$,
and

$\frac{t(\mathbf{x_{i}}}{n^{+}}\leq\widehat{AUC}(h,g|S_{n})-\frac{1}{n^{+}}$
(12)

is assumed to hold, we have the $\bar{G}>G$ (a.k.a., the conclusion of
Proposition 2) after removing one negative instance. Moreover, if (12) holds
for a second negative instance $\mathbf{x_{i}}$, then:

$\frac{t(\mathbf{x_{i}}}{n^{+}}\leq\widehat{AUC}(h,g|S_{n})-\frac{1}{n^{+}}\leq\widehat{AUC}(h,g^{\prime}|S_{n})-\frac{1}{n^{+}}$

where $g^{\prime}$ abstains on the first negative instance. In fact,
$\bar{G}>G$ implies that
$\widehat{AUC}(h,g|S_{n})<\widehat{AUC}(h,g^{\prime}|S_{n})$. Therefore, we
can iterate the conclusion that the Gini coefficient (or, equivalently,
empirical AUC) increases by abstaining on any number of negative instances
that satisfy the assumption of Proposition 2. ∎

Figure 5: CAP plot before removing a negative instance. Figure 6: CAP plot
after removing a negative instance.

###  A.2 Datasets description

Adult is an extract from the 1994 US Census, with class the binarization of
income into $\leq 50K$ and $>50K$. The final training set contains 30,162
instances and 55 features after one-hot encoding. The test set size is 15,060.

LendingClub regards repaying a loan obtained by an online platform. We used a
temporal split to build the final training set (1,364,697 instances) and the
test set (445,912 instances). The dataset has 65 features.

The GiveMe dataset aims at predicting the financial distress of a borrower
within two years. The training and the test set were obtained by stratified
random sampling, and they contain 12 features, and 112,500 and 37,500
instances respectively.

UCICredit regards credit card defaults in Taiwan. This dataset from Dua and
Graff, (2017) concerns whether or not a credit card holder will default in the
next six months Yeh and Lien, (2009). Training and test sets were obtained by
stratified random sampling. The training set includes 22,500 instances (7,500
for the test set) and 23 features.

CSDS1, CSDS2 and CSDS3 \- from Barddal et al., (2020) \- regard predicting
defaults in repaying a loan: within six months for CSDS1 (data span over 15
months), within 2 months for CSDS2 (data span over 25 months), and within
three months for CSDS3 (data span over 16 months). Training set and test set
were divided through a timestamp variable. The training set of CSDS1 consists
of 230,409 instances and 155 features (test set size is 76,939). For CSDS2,
the training set contains 37,100 instances and 35 features (test set size is
12,533). For CSDS3, the training set contains 71,177 instances and 144
features (test set size 23,288).

The CatsVsDogs dataset is a collection of cats and dogs images. The task here
is to distinguish between the two species. The training and test sets were
obtained as described in Liu et al., (2019). The training set contains 20,000
images, each one of 64x64 pixels. The test set consists of 5,000 images.

Finally we considered the image dataset CIFAR-10-cat from Krizhevsky, (2009).
For each of the 10 class labels, we have 5,000 32x32 images in the training
set and 1,000 in the test set. We transformed it into a binary classification
task by using the cat label as the positive class.

###  A.3 Models

####  AUCross and PlugInAUC.

In the main paper we used as base classifier a LightGBM classifier with
default parameters. In Tables 6-9 we provide also results for a Logistic
Regression and a Random Forest Classifier from sklearn package with default
parameters, a ResNet implementation with default parameters from rtdl
Gorishniy et al., (2021) and a XGBoost from xgboost package with default
parameters.

####  PlugIn and SCross.

As for AUCross, we considered a LightGBM classifier with default parameters.

####  SelNet.

Figure 7: Scheme for Selective Net architecture.

Selective Net is a selective model $(h,g)$ that optimizes at the same time
both $h(\mathbf{x})$ and $g(\mathbf{x})$. Its schema is summarized in Figure
7. The architecture is based on four distinct parts: the main body, the
predictive head, the selective head and the auxiliary head. The input is
initially processed by the main body: it consists of deep layers that are
shared by all the three heads. Any type of architecture can be used in this
part (e.g., convolutional layers, linear layers, recurrent layers ecc.). The
predictive head provides the final prediction $h(\mathbf{x})$; the selective
head outputs the selective function $g(\mathbf{x})$; the auxiliary head is
used to ensure that the main body part is exposed to all training instances,
i.e., it is used to avoid that SelNet overfits on the accepted instances. We
used pytorch to model SelNet. For tabular datasets, we built the main body
part using ResNet with default parameters provided by rtdl Gorishniy et al.,
(2021), as authors claim that ResNet is a valid baseline on tabular data. For
images, we used as the main body the VGG16 architecture Simonyan and
Zisserman, (2015) as done in the original paper Geifman and El-Yaniv, (2019).
We then added the classification head, the selection head and the auxiliary
head following SelNet paper. For CSDS1, Lending and GiveMe the prediction head
and the auxiliary head were made by a first linear layer with 512 nodes
followed by a batch normalization layer and ReLu activation; a second layer
with 256 nodes, batch normalization and ReLu activation; a final dense layer
with 128 nodes ending with two nodes and softmax activation. For the other
datasets the classification and auxiliary heads were made by a single linear
layer with 128 nodes and a final softmax activation. We built selective heads
using a linear layer with 128 nodes, batch normalization, relu activation and
another 64-node linear layer ending with a single node and Sigmoid activation.
All the models are available in the code here. We point out that the lack of a
clear design methodology of the SelectiveNet structure for a given dataset is
a major drawback of SelNet compared to the flexibility our model-agnostic
method. Models were trained, for an expected coverage $c$, using the same loss
function as in Geifman and El-Yaniv, (2019):

$\mathcal{L}=\alpha\frac{\frac{1}{n}\sum_{i=1}^{n}l(h(\mathbf{x}_{i}),y_{i})g(\mathbf{x}_{i})}{\hat{\phi}(g|S_{n})}+\lambda(\max(0,c-\hat{\phi}(g|S_{n})))^{2}+(1-\alpha)\frac{1}{n}\sum_{i=1}^{n}l(v(\mathbf{x}_{i}),y_{i}),$
(13)

where $h(\mathbf{x}_{i})$ is the classification head prediction,
$l(h(\mathbf{x}_{i}),y_{i})$ is the cross entropy loss, $g(\mathbf{x}_{i})$ is
the selection head output over instance $i$, $v(\mathbf{x}_{i})$ is the
auxiliary head prediction and
$\hat{\phi}(g|S_{n})=\nicefrac{{\sum_{i=1}^{n}g(\mathbf{x}_{i})}}{{n}}$ is the
empirical coverage. Both parameters $\lambda$ and $\alpha$ are set as in
Geifman and El-Yaniv, (2019) to $\lambda=32$ and $\alpha=.5$. The batch size
was $512$ for Lending; $128$ for GiveMe, CSDS1, CSDS2, CSDS3 and image data;
$32$ for Adult and UCICredit. The learning procedure was run for 300 epochs
with 13 as loss and it used as an optimizer Stochastic Gradient Descent
setting learning-rate=.1, momentum=.9,Nesterov=True and a decay of $.5$ every
25 epochs as in the original paper. The training was performed over 90% of
training set instances while we used the remaining 10% to calibrate the
selective head.

#### SAT.

Let us consider the standard binary classification problem where the
classifier $h$ can produce a score for instance $i$ belonging to class $0$ or
$1$333The problem is symmetrical in the binary setting., i.e. in our main
paper $s(\mathbf{x}_{i})=s_{1}(\mathbf{x}_{i})$. SAT Huang et al., (2020)
introduces an extra class $v$ (representing abstention) during training and
replace the confidence function with the score for the additional class $v$.
This allows for training a selective classifier in an end-to-end fashion.
Given a batch of data pairs $\\{(\mathbf{x}_{i},y_{i})\\}$ of size $M$, the
model score for class $a$ $s_{a}(\mathbf{x}_{i})$ and its exponential moving
average $t_{i}$ for each sample, we optimize the classifier $h$ by minimizing:

$\mathcal{L}(h_{\theta})=-\frac{1}{M}\sum_{i=1}^{M}[t_{i,y_{i}}\log(s_{y_{i}}(\mathbf{x}_{i}))+(1-t_{i,y_{i}})\log{s_{v}(\mathbf{x}_{i})}]$
(14)

where $s_{y_{i}}(\mathbf{x}_{i})$ denotes the score attributed by the
classifier to the real class of instance $i$. This loss is a composition of
two terms: the first one measures the standard cross-entropy loss between
prediction and original label $y_{i}$; the second term acts as the selection
function and identifies uncertain samples in the dataset. The value
$t_{i,y_{i}}$ trades-off these two terms: if $t_{i,y_{i}}$ is very small, the
sample is treated as uncertain and the second term enforces the selective
classifier to learn to abstain from this sample; if $t_{i,y_{i}}$ is close to
1, the loss recovers the standard cross entropy minimization and enforces the
selective classifier to make perfect predictions. The code was based on the
pytorch implementation available here. We employed the same batch sizes as for
SelNet and we set up training details as in the original paper of SAT: we used
the loss in 14 for 300 epochs and as an optimizer Stochastic Gradient Descent,
setting learning-rate=.1, momentum=.9,Nesterov=True and a decay of $.5$ every
25 epochs. The training was performed over 90% of training set instances while
we used the remaining 10% to calibrate the selection function.

###  A.4 Other metrics considered

We report results for selective accuracy and positive rate in Table 4. As
discussed in the main paper, AUCross and PlugInAUC do not guarantee
improvements in terms of accuracy whenever the target coverage decreases.
Interestingly, they are able to maintain the positive rate more stable than
the compared approaches. Finally, we report training times for all the methods
in Table 5. PlugInAUC and PlugIn are clear winners over tabular datases as
they can exploit fast classifiers. Both AUCross and SCross pay a factor
proportional to the number of folds $K$ used in the cross-fitting part of the
algorithm. This extra cost can be potentially mitigated on tabular datasets by
parallelizing the cross-fitting procedure. Finally, notice that SelNet is the
only approach which require a separate run for each target coverage $c$.

|  | Selective Accuracy | Positive Rate
---|---|---|---
| $c$ | AUCross | PlugIn | PlugInAUC | SCross | SelNet | SAT | AUCross | PlugIn | PlugInAUC | SCross | SelNet | SAT
Adult | .99 | .870 $\pm$ .003 | .872 $\pm$ .003 | .870 $\pm$ .003 | .871 $\pm$ .003 | .848 $\pm$ .003 | .845 $\pm$ .003 | .246 $\pm$ .004 | .245 $\pm$ .004 | .246 $\pm$ .004 | .245 $\pm$ .004 | .242 $\pm$ .004 | .246 $\pm$ .004
.95 | .875 $\pm$ .003 | .888 $\pm$ .003 | .874 $\pm$ .003 | .888 $\pm$ .003 | .858 $\pm$ .003 | .845 $\pm$ .003 | .246 $\pm$ .004 | .234 $\pm$ .004 | .247 $\pm$ .004 | .234 $\pm$ .004 | .235 $\pm$ .004 | .247 $\pm$ .004
.90 | .882 $\pm$ .003 | .903 $\pm$ .003 | .880 $\pm$ .003 | .902 $\pm$ .003 | .872 $\pm$ .003 | .845 $\pm$ .003 | .246 $\pm$ .004 | .221 $\pm$ .004 | .248 $\pm$ .004 | .220 $\pm$ .004 | .225 $\pm$ .004 | .249 $\pm$ .004
.85 | .888 $\pm$ .003 | .920 $\pm$ .003 | .887 $\pm$ .003 | .919 $\pm$ .003 | .886 $\pm$ .003 | .846 $\pm$ .003 | .247 $\pm$ .004 | .203 $\pm$ .004 | .249 $\pm$ .004 | .206 $\pm$ .004 | .115 $\pm$ .003 | .250 $\pm$ .004
.80 | .899 $\pm$ .003 | .936 $\pm$ .003 | .897 $\pm$ .003 | .934 $\pm$ .003 | .901 $\pm$ .003 | .846 $\pm$ .003 | .245 $\pm$ .004 | .185 $\pm$ .004 | .249 $\pm$ .004 | .190 $\pm$ .004 | .180 $\pm$ .004 | .251 $\pm$ .004
.75 | .907 $\pm$ .003 | .950 $\pm$ .003 | .905 $\pm$ .003 | .950 $\pm$ .003 | .908 $\pm$ .003 | .845 $\pm$ .004 | .245 $\pm$ .005 | .169 $\pm$ .004 | .250 $\pm$ .005 | .170 $\pm$ .004 | .149 $\pm$ .004 | .252 $\pm$ .004
Lending | .99 | .899 $\pm$ .001 | .899 $\pm$ .001 | .899 $\pm$ .001 | .899 $\pm$ .001 | .870 $\pm$ .001 | .754 $\pm$ .003 | .224 $\pm$ .001 | .222 $\pm$ .001 | .224 $\pm$ .001 | .222 $\pm$ .001 | .223 $\pm$ .001 | .249 $\pm$ .003
.95 | .908 $\pm$ .001 | .909 $\pm$ .001 | .909 $\pm$ .001 | .910 $\pm$ .001 | .873 $\pm$ .001 | .754 $\pm$ .003 | .217 $\pm$ .001 | .206 $\pm$ .001 | .217 $\pm$ .001 | .205 $\pm$ .001 | .215 $\pm$ .001 | .249 $\pm$ .003
.90 | .921 $\pm$ .001 | .923 $\pm$ .001 | .921 $\pm$ .001 | .924 $\pm$ .001 | .912 $\pm$ .001 | .754 $\pm$ .003 | .207 $\pm$ .001 | .184 $\pm$ .001 | .207 $\pm$ .001 | .183 $\pm$ .001 | .191 $\pm$ .001 | .249 $\pm$ .003
.85 | .934 $\pm$ .001 | .937 $\pm$ .001 | .934 $\pm$ .001 | .938 $\pm$ .001 | .885 $\pm$ .001 | .754 $\pm$ .003 | .197 $\pm$ .001 | .162 $\pm$ .001 | .197 $\pm$ .001 | .161 $\pm$ .001 | .134 $\pm$ .001 | .249 $\pm$ .003
.80 | .948 $\pm$ .001 | .951 $\pm$ .001 | .949 $\pm$ .001 | .951 $\pm$ .001 | .926 $\pm$ .001 | .754 $\pm$ .003 | .186 $\pm$ .001 | .140 $\pm$ .001 | .186 $\pm$ .001 | .139 $\pm$ .001 | .157 $\pm$ .001 | .249 $\pm$ .003
.75 | .964 $\pm$ .001 | .963 $\pm$ .001 | .965 $\pm$ .001 | .963 $\pm$ .001 | .924 $\pm$ .001 | .754 $\pm$ .003 | .174 $\pm$ .001 | .118 $\pm$ .001 | .174 $\pm$ .001 | .116 $\pm$ .001 | .143 $\pm$ .001 | .249 $\pm$ .003
GiveMe | .99 | .938 $\pm$ .002 | .942 $\pm$ .002 | .938 $\pm$ .002 | .942 $\pm$ .002 | .954 $\pm$ .002 | .916 $\pm$ .002 | .068 $\pm$ .002 | .063 $\pm$ .002 | .067 $\pm$ .002 | .063 $\pm$ .002 | .047 $\pm$ .002 | .085 $\pm$ .002
.95 | .937 $\pm$ .002 | .956 $\pm$ .002 | .937 $\pm$ .002 | .956 $\pm$ .002 | .954 $\pm$ .002 | .915 $\pm$ .002 | .068 $\pm$ .002 | .046 $\pm$ .002 | .068 $\pm$ .002 | .047 $\pm$ .002 | .047 $\pm$ .002 | .086 $\pm$ .002
.90 | .938 $\pm$ .002 | .967 $\pm$ .001 | .937 $\pm$ .002 | .967 $\pm$ .001 | .961 $\pm$ .001 | .913 $\pm$ .002 | .067 $\pm$ .002 | .034 $\pm$ .001 | .068 $\pm$ .002 | .034 $\pm$ .001 | .040 $\pm$ .001 | .088 $\pm$ .002
.85 | .939 $\pm$ .002 | .973 $\pm$ .001 | .938 $\pm$ .002 | .973 $\pm$ .001 | .967 $\pm$ .001 | .913 $\pm$ .002 | .067 $\pm$ .002 | .028 $\pm$ .001 | .068 $\pm$ .002 | .028 $\pm$ .001 | .034 $\pm$ .001 | .088 $\pm$ .002
.80 | .939 $\pm$ .002 | .977 $\pm$ .001 | .938 $\pm$ .002 | .978 $\pm$ .001 | .969 $\pm$ .001 | .912 $\pm$ .002 | .067 $\pm$ .002 | .024 $\pm$ .001 | .067 $\pm$ .002 | .023 $\pm$ .001 | .032 $\pm$ .001 | .089 $\pm$ .002
.75 | .939 $\pm$ .002 | .980 $\pm$ .001 | .939 $\pm$ .002 | .981 $\pm$ .001 | .971 $\pm$ .001 | .912 $\pm$ .002 | .067 $\pm$ .002 | .021 $\pm$ .001 | .068 $\pm$ .002 | .020 $\pm$ .001 | .030 $\pm$ .001 | .089 $\pm$ .002
UCICredit | .99 | .811 $\pm$ .005 | .814 $\pm$ .005 | .812 $\pm$ .005 | .814 $\pm$ .005 | .815 $\pm$ .005 | .812 $\pm$ .005 | .222 $\pm$ .005 | .220 $\pm$ .005 | .222 $\pm$ .005 | .220 $\pm$ .005 | .220 $\pm$ .005 | .222 $\pm$ .005
.95 | .810 $\pm$ .005 | .826 $\pm$ .005 | .811 $\pm$ .005 | .827 $\pm$ .005 | .824 $\pm$ .005 | .812 $\pm$ .005 | .225 $\pm$ .006 | .210 $\pm$ .005 | .224 $\pm$ .005 | .209 $\pm$ .005 | .180 $\pm$ .005 | .222 $\pm$ .005
.90 | .809 $\pm$ .005 | .838 $\pm$ .005 | .811 $\pm$ .005 | .839 $\pm$ .005 | .830 $\pm$ .005 | .813 $\pm$ .005 | .228 $\pm$ .006 | .194 $\pm$ .005 | .226 $\pm$ .006 | .195 $\pm$ .005 | .176 $\pm$ .005 | .221 $\pm$ .005
.85 | .806 $\pm$ .005 | .849 $\pm$ .005 | .811 $\pm$ .005 | .855 $\pm$ .005 | .845 $\pm$ .005 | .813 $\pm$ .005 | .233 $\pm$ .006 | .179 $\pm$ .005 | .229 $\pm$ .006 | .176 $\pm$ .005 | .156 $\pm$ .005 | .221 $\pm$ .005
.80 | .806 $\pm$ .006 | .863 $\pm$ .005 | .810 $\pm$ .006 | .867 $\pm$ .005 | .858 $\pm$ .005 | .813 $\pm$ .005 | .236 $\pm$ .006 | .161 $\pm$ .005 | .232 $\pm$ .006 | .158 $\pm$ .005 | .143 $\pm$ .005 | .221 $\pm$ .005
.75 | .806 $\pm$ .006 | .872 $\pm$ .005 | .808 $\pm$ .006 | .875 $\pm$ .005 | .871 $\pm$ .005 | .813 $\pm$ .005 | .239 $\pm$ .006 | .143 $\pm$ .005 | .237 $\pm$ .006 | .141 $\pm$ .005 | .130 $\pm$ .005 | .221 $\pm$ .005
CSDS1 | .99 | .857 $\pm$ .002 | .863 $\pm$ .002 | .857 $\pm$ .002 | .863 $\pm$ .002 | .861 $\pm$ .002 | .862 $\pm$ .002 | .145 $\pm$ .002 | .139 $\pm$ .002 | .144 $\pm$ .002 | .138 $\pm$ .002 | .140 $\pm$ .002 | .139 $\pm$ .002
.95 | .856 $\pm$ .002 | .875 $\pm$ .002 | .856 $\pm$ .002 | .875 $\pm$ .002 | .873 $\pm$ .002 | .871 $\pm$ .002 | .146 $\pm$ .002 | .126 $\pm$ .002 | .146 $\pm$ .002 | .126 $\pm$ .002 | .128 $\pm$ .002 | .130 $\pm$ .002
.90 | .854 $\pm$ .002 | .885 $\pm$ .002 | .855 $\pm$ .002 | .885 $\pm$ .002 | .882 $\pm$ .002 | .883 $\pm$ .002 | .147 $\pm$ .002 | .116 $\pm$ .002 | .147 $\pm$ .002 | .116 $\pm$ .002 | .119 $\pm$ .002 | .118 $\pm$ .002
.85 | .853 $\pm$ .002 | .892 $\pm$ .002 | .853 $\pm$ .002 | .892 $\pm$ .002 | .890 $\pm$ .002 | .890 $\pm$ .002 | .149 $\pm$ .002 | .109 $\pm$ .002 | .149 $\pm$ .002 | .109 $\pm$ .002 | .111 $\pm$ .002 | .111 $\pm$ .002
.80 | .852 $\pm$ .002 | .898 $\pm$ .002 | .852 $\pm$ .002 | .899 $\pm$ .002 | .897 $\pm$ .002 | .897 $\pm$ .002 | .150 $\pm$ .002 | .103 $\pm$ .002 | .150 $\pm$ .002 | .102 $\pm$ .002 | .104 $\pm$ .002 | .104 $\pm$ .002
.75 | .850 $\pm$ .002 | .904 $\pm$ .002 | .850 $\pm$ .002 | .904 $\pm$ .002 | .902 $\pm$ .002 | .902 $\pm$ .002 | .152 $\pm$ .002 | .097 $\pm$ .002 | .152 $\pm$ .002 | .097 $\pm$ .002 | .099 $\pm$ .002 | .099 $\pm$ .002
CSDS2 | .99 | .982 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .983 $\pm$ .002 | .982 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .018 $\pm$ .002 | .019 $\pm$ .002
.95 | .982 $\pm$ .002 | .983 $\pm$ .002 | .982 $\pm$ .002 | .985 $\pm$ .002 | .984 $\pm$ .002 | .982 $\pm$ .002 | .019 $\pm$ .002 | .018 $\pm$ .002 | .019 $\pm$ .002 | .016 $\pm$ .002 | .017 $\pm$ .002 | .019 $\pm$ .002
.90 | .982 $\pm$ .002 | .984 $\pm$ .002 | .982 $\pm$ .002 | .985 $\pm$ .002 | .986 $\pm$ .002 | .985 $\pm$ .002 | .019 $\pm$ .002 | .017 $\pm$ .002 | .019 $\pm$ .002 | .016 $\pm$ .002 | .015 $\pm$ .002 | .016 $\pm$ .002
.85 | .982 $\pm$ .002 | .985 $\pm$ .002 | .981 $\pm$ .002 | .986 $\pm$ .002 | .986 $\pm$ .002 | .985 $\pm$ .002 | .019 $\pm$ .002 | .016 $\pm$ .002 | .019 $\pm$ .002 | .015 $\pm$ .002 | .015 $\pm$ .002 | .016 $\pm$ .002
.80 | .982 $\pm$ .002 | .986 $\pm$ .002 | .981 $\pm$ .002 | .987 $\pm$ .002 | .987 $\pm$ .002 | .986 $\pm$ .002 | .019 $\pm$ .002 | .015 $\pm$ .002 | .019 $\pm$ .002 | .014 $\pm$ .002 | .014 $\pm$ .002 | .015 $\pm$ .002
.75 | .981 $\pm$ .002 | .986 $\pm$ .002 | .981 $\pm$ .002 | .987 $\pm$ .002 | .988 $\pm$ .002 | .987 $\pm$ .002 | .019 $\pm$ .002 | .015 $\pm$ .002 | .020 $\pm$ .002 | .014 $\pm$ .002 | .013 $\pm$ .002 | .014 $\pm$ .002
CSDS3 | .99 | .814 $\pm$ .003 | .816 $\pm$ .003 | .813 $\pm$ .003 | .816 $\pm$ .003 | .810 $\pm$ .003 | .809 $\pm$ .003 | .253 $\pm$ .003 | .252 $\pm$ .003 | .254 $\pm$ .003 | .251 $\pm$ .003 | .251 $\pm$ .003 | .252 $\pm$ .003
.95 | .817 $\pm$ .003 | .826 $\pm$ .003 | .816 $\pm$ .003 | .827 $\pm$ .003 | .822 $\pm$ .003 | .820 $\pm$ .003 | .254 $\pm$ .003 | .242 $\pm$ .003 | .254 $\pm$ .003 | .242 $\pm$ .003 | .244 $\pm$ .003 | .243 $\pm$ .003
.90 | .820 $\pm$ .003 | .841 $\pm$ .003 | .819 $\pm$ .003 | .841 $\pm$ .003 | .830 $\pm$ .003 | .835 $\pm$ .003 | .254 $\pm$ .003 | .228 $\pm$ .003 | .255 $\pm$ .003 | .229 $\pm$ .003 | .233 $\pm$ .003 | .232 $\pm$ .003
.85 | .825 $\pm$ .003 | .856 $\pm$ .003 | .823 $\pm$ .003 | .855 $\pm$ .003 | .844 $\pm$ .003 | .850 $\pm$ .003 | .254 $\pm$ .003 | .214 $\pm$ .003 | .255 $\pm$ .003 | .215 $\pm$ .003 | .219 $\pm$ .003 | .219 $\pm$ .003
.80 | .829 $\pm$ .003 | .871 $\pm$ .003 | .826 $\pm$ .003 | .869 $\pm$ .003 | .860 $\pm$ .003 | .861 $\pm$ .003 | .254 $\pm$ .004 | .199 $\pm$ .003 | .256 $\pm$ .004 | .200 $\pm$ .003 | .206 $\pm$ .003 | .207 $\pm$ .003
.75 | .835 $\pm$ .003 | .884 $\pm$ .003 | .830 $\pm$ .003 | .883 $\pm$ .003 | .871 $\pm$ .003 | .872 $\pm$ .003 | .254 $\pm$ .004 | .182 $\pm$ .003 | .257 $\pm$ .004 | .185 $\pm$ .003 | .178 $\pm$ .003 | .188 $\pm$ .003
CatsVsDogs | .99 | .954 $\pm$ .004 | .950 $\pm$ .004 | .944 $\pm$ .004 | .956 $\pm$ .003 | .941 $\pm$ .004 | .950 $\pm$ .004 | .500 $\pm$ .008 | .505 $\pm$ .008 | .503 $\pm$ .008 | .500 $\pm$ .008 | .500 $\pm$ .008 | .500 $\pm$ .008
.95 | .968 $\pm$ .003 | .949 $\pm$ .004 | .953 $\pm$ .004 | .969 $\pm$ .003 | .964 $\pm$ .003 | .964 $\pm$ .003 | .498 $\pm$ .008 | .514 $\pm$ .008 | .512 $\pm$ .008 | .495 $\pm$ .008 | .494 $\pm$ .008 | .497 $\pm$ .008
.90 | .977 $\pm$ .003 | .947 $\pm$ .004 | .971 $\pm$ .003 | .979 $\pm$ .003 | .978 $\pm$ .003 | .977 $\pm$ .003 | .494 $\pm$ .008 | .531 $\pm$ .008 | .518 $\pm$ .008 | .490 $\pm$ .008 | .482 $\pm$ .008 | .496 $\pm$ .008
.85 | .984 $\pm$ .002 | .944 $\pm$ .004 | .982 $\pm$ .003 | .985 $\pm$ .002 | .983 $\pm$ .002 | .984 $\pm$ .002 | .488 $\pm$ .008 | .546 $\pm$ .008 | .524 $\pm$ .008 | .488 $\pm$ .008 | .485 $\pm$ .008 | .494 $\pm$ .008
.80 | .987 $\pm$ .002 | .942 $\pm$ .004 | .986 $\pm$ .002 | .989 $\pm$ .002 | .990 $\pm$ .002 | .985 $\pm$ .002 | .473 $\pm$ .008 | .556 $\pm$ .008 | .528 $\pm$ .008 | .493 $\pm$ .008 | .476 $\pm$ .008 | .482 $\pm$ .008
.75 | .988 $\pm$ .002 | .938 $\pm$ .004 | .988 $\pm$ .002 | .990 $\pm$ .002 | .990 $\pm$ .002 | .987 $\pm$ .002 | .445 $\pm$ .008 | .561 $\pm$ .009 | .529 $\pm$ .009 | .500 $\pm$ .009 | .472 $\pm$ .008 | .464 $\pm$ .008
CIFAR10-CAT | .99 | .952 $\pm$ .003 | .952 $\pm$ .003 | .953 $\pm$ .003 | .956 $\pm$ .002 | .952 $\pm$ .003 | .958 $\pm$ .002 | .101 $\pm$ .004 | .092 $\pm$ .003 | .102 $\pm$ .004 | .097 $\pm$ .004 | .098 $\pm$ .004 | .096 $\pm$ .004
.95 | .952 $\pm$ .003 | .952 $\pm$ .003 | .951 $\pm$ .003 | .969 $\pm$ .002 | .967 $\pm$ .002 | .969 $\pm$ .002 | .100 $\pm$ .004 | .072 $\pm$ .003 | .106 $\pm$ .004 | .078 $\pm$ .003 | .084 $\pm$ .003 | .075 $\pm$ .003
.90 | .953 $\pm$ .003 | .950 $\pm$ .003 | .950 $\pm$ .003 | .977 $\pm$ .002 | .977 $\pm$ .002 | .973 $\pm$ .002 | .100 $\pm$ .004 | .074 $\pm$ .003 | .111 $\pm$ .004 | .049 $\pm$ .003 | .064 $\pm$ .003 | .044 $\pm$ .003
.85 | .954 $\pm$ .003 | .947 $\pm$ .003 | .948 $\pm$ .003 | .979 $\pm$ .002 | .977 $\pm$ .002 | .976 $\pm$ .002 | .099 $\pm$ .004 | .077 $\pm$ .003 | .115 $\pm$ .004 | .040 $\pm$ .003 | .024 $\pm$ .002 | .025 $\pm$ .002
.80 | .959 $\pm$ .003 | .945 $\pm$ .003 | .946 $\pm$ .003 | .980 $\pm$ .002 | .984 $\pm$ .002 | .975 $\pm$ .002 | .095 $\pm$ .004 | .080 $\pm$ .004 | .122 $\pm$ .004 | .037 $\pm$ .003 | .017 $\pm$ .002 | .026 $\pm$ .002
.75 | .971 $\pm$ .002 | .942 $\pm$ .003 | .943 $\pm$ .003 | .980 $\pm$ .002 | .986 $\pm$ .002 | .975 $\pm$ .002 | .084 $\pm$ .003 | .084 $\pm$ .004 | .129 $\pm$ .004 | .036 $\pm$ .003 | .015 $\pm$ .002 | .026 $\pm$ .002
| # | 4/54 | 21/54 | 2/54 | 29/54 | 12/54 | 3/54 | 39/54 | 4/54 | 20/54 | 5/54 | 1/54 | 14/54

Table 4: Performance metrics (1,000 bootstrap runs over the test set, results
as mean $\pm$ stdev).

|  | Training Time (seconds)
---|---|---
| $c$ | AUCross | Plug-In | Plug-In-AUC | SCross | SelNet | SAT
Adult | .99 | 1.43 | 0.22 | 0.22 | 1.15 | 1787.93 | 2319.66
.95 | 1.43 | 0.22 | 0.22 | 1.15 | 1779.74 | 2319.66
.90 | 1.43 | 0.22 | 0.22 | 1.15 | 1786.99 | 2319.66
.85 | 1.43 | 0.22 | 0.22 | 1.15 | 1781.64 | 2319.66
.80 | 1.43 | 0.22 | 0.22 | 1.15 | 1789.44 | 2319.66
.75 | 1.43 | 0.22 | 0.22 | 1.15 | 1788.53 | 2319.66
Lending | .99 | 13.16 | 2.55 | 2.55 | 12.10 | 9168.31 | 14182.26
.95 | 13.16 | 2.55 | 2.55 | 12.10 | 9198.68 | 14182.26
.90 | 13.16 | 2.55 | 2.55 | 12.10 | 9187.33 | 14182.26
.85 | 13.16 | 2.55 | 2.55 | 12.10 | 9204.72 | 14182.26
.80 | 13.16 | 2.55 | 2.55 | 12.10 | 9165.67 | 14182.26
.75 | 13.16 | 2.55 | 2.55 | 12.10 | 9199.28 | 14182.26
GiveMe | .99 | 1.60 | 0.27 | 0.27 | 1.39 | 2341.00 | 3104.83
.95 | 1.60 | 0.27 | 0.27 | 1.39 | 2323.07 | 3104.83
.90 | 1.60 | 0.27 | 0.27 | 1.39 | 2332.69 | 3104.83
.85 | 1.60 | 0.27 | 0.27 | 1.39 | 2332.97 | 3104.83
.80 | 1.60 | 0.27 | 0.27 | 1.39 | 2326.58 | 3104.83
.75 | 1.60 | 0.27 | 0.27 | 1.39 | 2333.06 | 3104.83
UCICredit | .99 | 1.26 | 0.20 | 0.20 | 1.11 | 1547.44 | 2070.54
.95 | 1.26 | 0.20 | 0.20 | 1.11 | 1549.75 | 2070.54
.90 | 1.26 | 0.20 | 0.20 | 1.11 | 1527.84 | 2070.54
.85 | 1.26 | 0.20 | 0.20 | 1.11 | 1547.15 | 2070.54
.80 | 1.26 | 0.20 | 0.20 | 1.11 | 1548.82 | 2070.54
.75 | 1.26 | 0.20 | 0.20 | 1.11 | 1541.74 | 2070.54
CSDS1 | .99 | 5.60 | 1.02 | 1.02 | 5.31 | 4815.42 | 5524.00
.95 | 5.60 | 1.02 | 1.02 | 5.31 | 4812.96 | 5524.00
.90 | 5.60 | 1.02 | 1.02 | 5.31 | 4813.66 | 5524.00
.85 | 5.60 | 1.02 | 1.02 | 5.31 | 4817.26 | 5524.00
.80 | 5.60 | 1.02 | 1.02 | 5.31 | 4818.25 | 5524.00
.75 | 5.60 | 1.02 | 1.02 | 5.31 | 4812.89 | 5524.00
CSDS2 | .99 | 1.23 | 0.21 | 0.21 | 1.41 | 609.99 | 357.92
.95 | 1.23 | 0.21 | 0.21 | 1.41 | 609.32 | 357.92
.90 | 1.23 | 0.21 | 0.21 | 1.41 | 606.45 | 357.92
.85 | 1.23 | 0.21 | 0.21 | 1.41 | 606.47 | 357.92
.80 | 1.23 | 0.21 | 0.21 | 1.41 | 610.34 | 357.92
.75 | 1.23 | 0.21 | 0.21 | 1.41 | 605.20 | 357.92
CSDS3 | .99 | 5.07 | 0.80 | 0.80 | 4.60 | 1188.31 | 748.88
.95 | 5.07 | 0.80 | 0.80 | 4.60 | 1193.35 | 748.88
.90 | 5.07 | 0.80 | 0.80 | 4.60 | 1192.50 | 748.88
.85 | 5.07 | 0.80 | 0.80 | 4.60 | 1190.79 | 748.88
.80 | 5.07 | 0.80 | 0.80 | 4.60 | 1191.03 | 748.88
.75 | 5.07 | 0.80 | 0.80 | 4.60 | 1191.56 | 748.88
CatsVsDogs | .99 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2700.46 | 2735.28
.95 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2699.13 | 2735.28
.90 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2733.35 | 2735.28
.85 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2769.44 | 2735.28
.80 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2766.80 | 2735.28
.75 | 14668.55 | 2624.68 | 2624.68 | 14475.64 | 2762.34 | 2735.28
CIFAR10-CAT | .99 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3317.14 | 3239.50
.95 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3296.70 | 3239.50
.90 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3337.28 | 3239.50
.85 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3358.18 | 3239.50
.80 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3262.57 | 3239.50
.75 | 17401.61 | 3026.98 | 3026.98 | 16924.30 | 3294.08 | 3239.50
| # | 0/54 | 54/54 | 54/54 | 0/54 | 0/54 | 0/54

Table 5: Time required for running different methods (in seconds).

###  A.5 Results for different classifiers

We report in Tables 6-9 the results for AUCross and PlugInAUC using different
classifiers over tabular datasets. Regarding coverage, we see the harshest
violations for both AUCross-RandForest and PlugInAUC-RandForest over CSDS2 and
for AUCross-ResNet over GiveMe. Coverage violations occur also for Lending
dataset, independently of the classifier. Regarding the AUC, both AUCross and
PlugInAUC succeed in increasing it while target coverage drops, regardless of
the considered base classifier. At the same time, we notice that for all the
base classifiers lowering coverage does not guarantee selective accuracy to
increase, highlighting once more the trade-off between these two metrics.
Finally, we see similar results across all the classifiers for positive rate.

###  A.6 Results for different $K$

We report empirical coverage and selective AUC for different choices of the
parameter $K$ in Table 10. The default value $K=5$ shows a slightly better
trade-off between empirical coverage and selectice AUC.

|  | Empirical Coverage
---|---|---
|  | AUCross | PlugInAUC
| $c$ | Logistic | RandForest | ResNet | XGBoost | Logistic | RandForest | ResNet | XGBoost
Adult | .99 | .991 $\pm$ (.001) | .988 $\pm$ (.001) | .994 $\pm$ (.001) | .990 $\pm$ (.001) | .988 $\pm$ (.001) | .989 $\pm$ (.001) | .988 $\pm$ (.001) | .992 $\pm$ (.001)
.95 | .950 $\pm$ (.002) | .945 $\pm$ (.002) | .961 $\pm$ (.002) | .951 $\pm$ (.002) | .947 $\pm$ (.002) | .949 $\pm$ (.002) | .946 $\pm$ (.002) | .951 $\pm$ (.002)
.90 | .901 $\pm$ (.003) | .893 $\pm$ (.003) | .912 $\pm$ (.003) | .891 $\pm$ (.003) | .896 $\pm$ (.003) | .894 $\pm$ (.003) | .903 $\pm$ (.003) | .899 $\pm$ (.003)
.85 | .850 $\pm$ (.003) | .853 $\pm$ (.003) | .840 $\pm$ (.004) | .846 $\pm$ (.003) | .848 $\pm$ (.003) | .836 $\pm$ (.004) | .849 $\pm$ (.003) | .850 $\pm$ (.003)
.80 | .800 $\pm$ (.004) | .804 $\pm$ (.004) | .792 $\pm$ (.004) | .793 $\pm$ (.004) | .799 $\pm$ (.004) | .783 $\pm$ (.004) | .801 $\pm$ (.004) | .800 $\pm$ (.004)
.75 | .747 $\pm$ (.004) | .752 $\pm$ (.004) | .735 $\pm$ (.004) | .744 $\pm$ (.004) | .744 $\pm$ (.004) | .743 $\pm$ (.004) | .749 $\pm$ (.004) | .748 $\pm$ (.004)
Lending | .99 | .996 $\pm$ (.001) | .992 $\pm$ (.001) | .995 $\pm$ (.001) | .996 $\pm$ (.001) | .996 $\pm$ (.001) | .992 $\pm$ (.001) | .995 $\pm$ (.001) | .996 $\pm$ (.001)
.95 | .976 $\pm$ (.001) | .971 $\pm$ (.001) | .976 $\pm$ (.001) | .980 $\pm$ (.001) | .976 $\pm$ (.001) | .971 $\pm$ (.001) | .975 $\pm$ (.001) | .980 $\pm$ (.001)
.90 | .951 $\pm$ (.001) | .945 $\pm$ (.001) | .951 $\pm$ (.001) | .958 $\pm$ (.001) | .951 $\pm$ (.001) | .945 $\pm$ (.001) | .949 $\pm$ (.001) | .958 $\pm$ (.001)
.85 | .926 $\pm$ (.001) | .918 $\pm$ (.001) | .925 $\pm$ (.001) | .936 $\pm$ (.001) | .926 $\pm$ (.001) | .917 $\pm$ (.001) | .922 $\pm$ (.001) | .936 $\pm$ (.001)
.80 | .900 $\pm$ (.001) | .890 $\pm$ (.001) | .898 $\pm$ (.001) | .912 $\pm$ (.001) | .899 $\pm$ (.001) | .884 $\pm$ (.001) | .894 $\pm$ (.001) | .912 $\pm$ (.001)
.75 | .871 $\pm$ (.001) | .856 $\pm$ (.001) | .868 $\pm$ (.001) | .886 $\pm$ (.001) | .871 $\pm$ (.001) | .855 $\pm$ (.001) | .865 $\pm$ (.001) | .886 $\pm$ (.001)
GiveMe | .99 | .990 $\pm$ (.001) | .942 $\pm$ (.002) | .944 $\pm$ (.002) | .990 $\pm$ (.001) | .991 $\pm$ (.001) | .944 $\pm$ (.002) | .977 $\pm$ (.001) | .991 $\pm$ (.001)
.95 | .953 $\pm$ (.002) | .942 $\pm$ (.002) | .755 $\pm$ (.003) | .951 $\pm$ (.002) | .950 $\pm$ (.002) | .944 $\pm$ (.002) | .930 $\pm$ (.002) | .948 $\pm$ (.002)
.90 | .904 $\pm$ (.002) | .858 $\pm$ (.002) | .640 $\pm$ (.003) | .898 $\pm$ (.002) | .898 $\pm$ (.002) | .863 $\pm$ (.002) | .884 $\pm$ (.002) | .898 $\pm$ (.002)
.85 | .855 $\pm$ (.002) | .788 $\pm$ (.003) | .570 $\pm$ (.003) | .849 $\pm$ (.002) | .844 $\pm$ (.002) | .794 $\pm$ (.003) | .841 $\pm$ (.002) | .849 $\pm$ (.002)
.80 | .805 $\pm$ (.003) | .762 $\pm$ (.003) | .499 $\pm$ (.003) | .799 $\pm$ (.003) | .796 $\pm$ (.003) | .780 $\pm$ (.003) | .797 $\pm$ (.003) | .797 $\pm$ (.003)
.75 | .758 $\pm$ (.003) | .741 $\pm$ (.003) | .431 $\pm$ (.003) | .750 $\pm$ (.003) | .746 $\pm$ (.003) | .667 $\pm$ (.003) | .748 $\pm$ (.003) | .750 $\pm$ (.003)
UCICredit | .99 | .991 $\pm$ (.002) | .949 $\pm$ (.003) | .987 $\pm$ (.002) | .990 $\pm$ (.002) | .990 $\pm$ (.002) | .954 $\pm$ (.003) | .991 $\pm$ (.002) | .990 $\pm$ (.002)
.95 | .953 $\pm$ (.003) | .949 $\pm$ (.003) | .939 $\pm$ (.003) | .947 $\pm$ (.003) | .962 $\pm$ (.003) | .904 $\pm$ (.004) | .946 $\pm$ (.003) | .948 $\pm$ (.003)
.90 | .903 $\pm$ (.004) | .878 $\pm$ (.004) | .873 $\pm$ (.004) | .894 $\pm$ (.004) | .913 $\pm$ (.004) | .857 $\pm$ (.005) | .901 $\pm$ (.004) | .894 $\pm$ (.004)
.85 | .853 $\pm$ (.005) | .827 $\pm$ (.005) | .814 $\pm$ (.005) | .840 $\pm$ (.005) | .864 $\pm$ (.004) | .838 $\pm$ (.005) | .849 $\pm$ (.005) | .848 $\pm$ (.005)
.80 | .808 $\pm$ (.005) | .777 $\pm$ (.005) | .759 $\pm$ (.005) | .784 $\pm$ (.005) | .816 $\pm$ (.005) | .775 $\pm$ (.005) | .796 $\pm$ (.005) | .806 $\pm$ (.005)
.75 | .762 $\pm$ (.005) | .712 $\pm$ (.006) | .706 $\pm$ (.006) | .729 $\pm$ (.006) | .770 $\pm$ (.005) | .733 $\pm$ (.006) | .744 $\pm$ (.006) | .752 $\pm$ (.005)
CSDS1 | .99 | .991 $\pm$ (.001) | .913 $\pm$ (.002) | .991 $\pm$ (.001) | .991 $\pm$ (.001) | .991 $\pm$ (.001) | .961 $\pm$ (.001) | .990 $\pm$ (.001) | .991 $\pm$ (.001)
.95 | .951 $\pm$ (.001) | .913 $\pm$ (.002) | .953 $\pm$ (.001) | .951 $\pm$ (.001) | .950 $\pm$ (.001) | .914 $\pm$ (.001) | .949 $\pm$ (.001) | .950 $\pm$ (.001)
.90 | .902 $\pm$ (.002) | .870 $\pm$ (.002) | .905 $\pm$ (.002) | .901 $\pm$ (.002) | .900 $\pm$ (.002) | .872 $\pm$ (.002) | .899 $\pm$ (.002) | .899 $\pm$ (.002)
.85 | .852 $\pm$ (.002) | .822 $\pm$ (.002) | .854 $\pm$ (.002) | .851 $\pm$ (.002) | .851 $\pm$ (.002) | .820 $\pm$ (.002) | .851 $\pm$ (.002) | .850 $\pm$ (.002)
.80 | .803 $\pm$ (.002) | .784 $\pm$ (.002) | .797 $\pm$ (.002) | .799 $\pm$ (.002) | .800 $\pm$ (.002) | .786 $\pm$ (.002) | .798 $\pm$ (.002) | .800 $\pm$ (.002)
.75 | .752 $\pm$ (.002) | .729 $\pm$ (.002) | .744 $\pm$ (.002) | .749 $\pm$ (.002) | .749 $\pm$ (.002) | .698 $\pm$ (.002) | .746 $\pm$ (.002) | .752 $\pm$ (.002)
CSDS2 | .99 | .993 $\pm$ (.001) | .653 $\pm$ (.005) | .990 $\pm$ (.001) | .990 $\pm$ (.001) | .990 $\pm$ (.001) | .829 $\pm$ (.004) | .989 $\pm$ (.001) | .991 $\pm$ (.001)
.95 | .953 $\pm$ (.002) | .653 $\pm$ (.005) | .942 $\pm$ (.003) | .949 $\pm$ (.002) | .950 $\pm$ (.003) | .829 $\pm$ (.004) | .947 $\pm$ (.003) | .954 $\pm$ (.002)
.90 | .903 $\pm$ (.003) | .653 $\pm$ (.005) | .881 $\pm$ (.003) | .890 $\pm$ (.003) | .894 $\pm$ (.003) | .829 $\pm$ (.004) | .902 $\pm$ (.003) | .907 $\pm$ (.003)
.85 | .854 $\pm$ (.004) | .653 $\pm$ (.005) | .824 $\pm$ (.004) | .834 $\pm$ (.004) | .841 $\pm$ (.004) | .650 $\pm$ (.005) | .856 $\pm$ (.004) | .851 $\pm$ (.004)
.80 | .803 $\pm$ (.004) | .653 $\pm$ (.005) | .761 $\pm$ (.004) | .781 $\pm$ (.004) | .798 $\pm$ (.004) | .650 $\pm$ (.005) | .806 $\pm$ (.004) | .806 $\pm$ (.004)
.75 | .756 $\pm$ (.004) | .653 $\pm$ (.005) | .703 $\pm$ (.005) | .732 $\pm$ (.004) | .749 $\pm$ (.004) | .516 $\pm$ (.005) | .756 $\pm$ (.004) | .754 $\pm$ (.004)
CSDS3 | .99 | .990 $\pm$ (.001) | .980 $\pm$ (.001) | .992 $\pm$ (.001) | .989 $\pm$ (.001) | .992 $\pm$ (.001) | .989 $\pm$ (.001) | .991 $\pm$ (.001) | .991 $\pm$ (.001)
.95 | .948 $\pm$ (.002) | .933 $\pm$ (.002) | .954 $\pm$ (.002) | .949 $\pm$ (.002) | .955 $\pm$ (.002) | .941 $\pm$ (.002) | .952 $\pm$ (.002) | .955 $\pm$ (.002)
.90 | .899 $\pm$ (.002) | .887 $\pm$ (.003) | .907 $\pm$ (.002) | .903 $\pm$ (.002) | .905 $\pm$ (.002) | .896 $\pm$ (.003) | .901 $\pm$ (.003) | .906 $\pm$ (.002)
.85 | .851 $\pm$ (.003) | .831 $\pm$ (.003) | .860 $\pm$ (.003) | .854 $\pm$ (.003) | .857 $\pm$ (.003) | .838 $\pm$ (.003) | .850 $\pm$ (.003) | .860 $\pm$ (.003)
.80 | .799 $\pm$ (.003) | .795 $\pm$ (.003) | .789 $\pm$ (.003) | .814 $\pm$ (.003) | .799 $\pm$ (.003) | .803 $\pm$ (.003) | .803 $\pm$ (.003) | .814 $\pm$ (.003)
.75 | .744 $\pm$ (.003) | .738 $\pm$ (.003) | .744 $\pm$ (.003) | .760 $\pm$ (.003) | .746 $\pm$ (.003) | .748 $\pm$ (.003) | .753 $\pm$ (.003) | .762 $\pm$ (.003)
| # | 22/42 | 7/42 | 3/42 | 16/42 | 12/42 | 9/42 | 13/42 | 18/42
| $V$ | .015 $\pm$ .034 | .064 $\pm$ .087 | .064 $\pm$ .108 | .019 $\pm$ .038 | .017 $\pm$ .034 | .056 $\pm$ .069 | .016 $\pm$ .032 | .017 $\pm$ .038

Table 6: Empirical coverage for AUCross and PlugInAUC using different
classifiers (1,000 bootstrap runs over the test set, results as mean $\pm$
stdev).

|  | Selective AUC
---|---|---
|  | AUCross | PlugInAUC
| $c$ | Logistic | RandForest | ResNet | XGBoost | Logistic | RandForest | ResNet | XGBoost
Adult | .99 | .903 $\pm$ .003 | .888 $\pm$ .003 | .901 $\pm$ .003 | .928 $\pm$ .003 | .904 $\pm$ .003 | .886 $\pm$ .003 | .906 $\pm$ .003 | .927 $\pm$ .003
.95 | .910 $\pm$ .003 | .895 $\pm$ .003 | .907 $\pm$ .003 | .934 $\pm$ .003 | .911 $\pm$ .003 | .892 $\pm$ .003 | .912 $\pm$ .003 | .934 $\pm$ .003
.90 | .918 $\pm$ .003 | .903 $\pm$ .003 | .914 $\pm$ .003 | .944 $\pm$ .002 | .918 $\pm$ .003 | .900 $\pm$ .003 | .919 $\pm$ .003 | .942 $\pm$ .003
.85 | .926 $\pm$ .003 | .910 $\pm$ .003 | .925 $\pm$ .003 | .950 $\pm$ .002 | .925 $\pm$ .003 | .908 $\pm$ .003 | .927 $\pm$ .003 | .949 $\pm$ .002
.80 | .934 $\pm$ .003 | .918 $\pm$ .003 | .933 $\pm$ .003 | .959 $\pm$ .002 | .932 $\pm$ .003 | .916 $\pm$ .003 | .934 $\pm$ .003 | .956 $\pm$ .002
.75 | .941 $\pm$ .003 | .925 $\pm$ .003 | .942 $\pm$ .003 | .965 $\pm$ .002 | .939 $\pm$ .003 | .922 $\pm$ .003 | .942 $\pm$ .003 | .962 $\pm$ .002
Lending | .99 | .956 $\pm$ .001 | .977 $\pm$ .001 | .981 $\pm$ .001 | .987 $\pm$ .001 | .956 $\pm$ .001 | .976 $\pm$ .001 | .975 $\pm$ .001 | .987 $\pm$ .001
.95 | .958 $\pm$ .001 | .979 $\pm$ .001 | .983 $\pm$ .001 | .988 $\pm$ .001 | .958 $\pm$ .001 | .979 $\pm$ .001 | .978 $\pm$ .001 | .989 $\pm$ .001
.90 | .962 $\pm$ .001 | .982 $\pm$ .001 | .986 $\pm$ .001 | .990 $\pm$ .001 | .962 $\pm$ .001 | .982 $\pm$ .001 | .981 $\pm$ .001 | .990 $\pm$ .001
.85 | .965 $\pm$ .001 | .984 $\pm$ .001 | .988 $\pm$ .001 | .992 $\pm$ .001 | .965 $\pm$ .001 | .984 $\pm$ .001 | .985 $\pm$ .001 | .992 $\pm$ .001
.80 | .968 $\pm$ .001 | .987 $\pm$ .001 | .991 $\pm$ .001 | .993 $\pm$ .001 | .968 $\pm$ .001 | .987 $\pm$ .001 | .988 $\pm$ .001 | .993 $\pm$ .001
.75 | .971 $\pm$ .001 | .989 $\pm$ .001 | .993 $\pm$ .001 | .994 $\pm$ .001 | .971 $\pm$ .001 | .989 $\pm$ .001 | .991 $\pm$ .001 | .994 $\pm$ .001
GiveMe | .99 | .699 $\pm$ .006 | .836 $\pm$ .005 | .670 $\pm$ .006 | .865 $\pm$ .004 | .698 $\pm$ .006 | .840 $\pm$ .005 | .834 $\pm$ .005 | .862 $\pm$ .004
.95 | .703 $\pm$ .006 | .836 $\pm$ .005 | .688 $\pm$ .007 | .871 $\pm$ .004 | .702 $\pm$ .006 | .840 $\pm$ .005 | .840 $\pm$ .005 | .869 $\pm$ .004
.90 | .708 $\pm$ .006 | .846 $\pm$ .005 | .705 $\pm$ .008 | .878 $\pm$ .004 | .708 $\pm$ .006 | .850 $\pm$ .005 | .847 $\pm$ .005 | .876 $\pm$ .004
.85 | .713 $\pm$ .006 | .854 $\pm$ .005 | .720 $\pm$ .008 | .885 $\pm$ .004 | .714 $\pm$ .006 | .857 $\pm$ .005 | .853 $\pm$ .005 | .883 $\pm$ .004
.80 | .720 $\pm$ .006 | .858 $\pm$ .005 | .732 $\pm$ .008 | .892 $\pm$ .004 | .720 $\pm$ .006 | .859 $\pm$ .005 | .859 $\pm$ .005 | .891 $\pm$ .004
.75 | .726 $\pm$ .006 | .861 $\pm$ .005 | .749 $\pm$ .009 | .899 $\pm$ .004 | .727 $\pm$ .007 | .870 $\pm$ .005 | .866 $\pm$ .005 | .897 $\pm$ .004
UCICredit | .99 | .701 $\pm$ .009 | .760 $\pm$ .008 | .768 $\pm$ .007 | .767 $\pm$ .008 | .702 $\pm$ .009 | .757 $\pm$ .008 | .768 $\pm$ .007 | .760 $\pm$ .008
.95 | .704 $\pm$ .009 | .760 $\pm$ .008 | .774 $\pm$ .007 | .774 $\pm$ .008 | .704 $\pm$ .009 | .764 $\pm$ .008 | .774 $\pm$ .007 | .765 $\pm$ .008
.90 | .709 $\pm$ .009 | .769 $\pm$ .008 | .783 $\pm$ .007 | .780 $\pm$ .008 | .708 $\pm$ .009 | .770 $\pm$ .008 | .779 $\pm$ .007 | .772 $\pm$ .008
.85 | .714 $\pm$ .009 | .776 $\pm$ .008 | .792 $\pm$ .007 | .787 $\pm$ .008 | .713 $\pm$ .009 | .772 $\pm$ .008 | .785 $\pm$ .007 | .778 $\pm$ .008
.80 | .719 $\pm$ .009 | .784 $\pm$ .008 | .799 $\pm$ .007 | .794 $\pm$ .008 | .718 $\pm$ .009 | .780 $\pm$ .008 | .793 $\pm$ .008 | .783 $\pm$ .008
.75 | .724 $\pm$ .009 | .794 $\pm$ .008 | .807 $\pm$ .007 | .801 $\pm$ .008 | .723 $\pm$ .009 | .786 $\pm$ .008 | .802 $\pm$ .008 | .790 $\pm$ .008
CSDS1 | .99 | .676 $\pm$ .003 | .636 $\pm$ .004 | .679 $\pm$ .003 | .680 $\pm$ .003 | .676 $\pm$ .003 | .634 $\pm$ .003 | .679 $\pm$ .003 | .680 $\pm$ .003
.95 | .679 $\pm$ .003 | .636 $\pm$ .004 | .682 $\pm$ .003 | .683 $\pm$ .003 | .679 $\pm$ .003 | .636 $\pm$ .004 | .683 $\pm$ .003 | .683 $\pm$ .003
.90 | .682 $\pm$ .003 | .639 $\pm$ .004 | .685 $\pm$ .003 | .686 $\pm$ .003 | .682 $\pm$ .003 | .639 $\pm$ .004 | .686 $\pm$ .003 | .687 $\pm$ .003
.85 | .685 $\pm$ .003 | .640 $\pm$ .004 | .690 $\pm$ .004 | .690 $\pm$ .003 | .685 $\pm$ .004 | .641 $\pm$ .004 | .690 $\pm$ .004 | .691 $\pm$ .003
.80 | .689 $\pm$ .004 | .643 $\pm$ .004 | .694 $\pm$ .004 | .694 $\pm$ .004 | .689 $\pm$ .004 | .644 $\pm$ .004 | .693 $\pm$ .004 | .695 $\pm$ .003
.75 | .693 $\pm$ .004 | .644 $\pm$ .004 | .698 $\pm$ .004 | .697 $\pm$ .004 | .693 $\pm$ .004 | .647 $\pm$ .004 | .697 $\pm$ .004 | .698 $\pm$ .004
CSDS2 | .99 | .615 $\pm$ .019 | .604 $\pm$ .023 | .615 $\pm$ .019 | .575 $\pm$ .021 | .616 $\pm$ .019 | .587 $\pm$ .021 | .624 $\pm$ .020 | .577 $\pm$ .019
.95 | .615 $\pm$ .019 | .604 $\pm$ .023 | .622 $\pm$ .020 | .574 $\pm$ .021 | .621 $\pm$ .020 | .587 $\pm$ .021 | .626 $\pm$ .020 | .578 $\pm$ .019
.90 | .619 $\pm$ .020 | .604 $\pm$ .023 | .623 $\pm$ .020 | .574 $\pm$ .021 | .627 $\pm$ .020 | .587 $\pm$ .021 | .628 $\pm$ .020 | .580 $\pm$ .019
.85 | .622 $\pm$ .020 | .604 $\pm$ .023 | .628 $\pm$ .020 | .576 $\pm$ .022 | .631 $\pm$ .021 | .580 $\pm$ .023 | .633 $\pm$ .021 | .583 $\pm$ .020
.80 | .626 $\pm$ .021 | .604 $\pm$ .023 | .619 $\pm$ .020 | .571 $\pm$ .022 | .636 $\pm$ .021 | .580 $\pm$ .023 | .639 $\pm$ .022 | .585 $\pm$ .020
.75 | .633 $\pm$ .022 | .604 $\pm$ .023 | .616 $\pm$ .020 | .568 $\pm$ .022 | .644 $\pm$ .022 | .591 $\pm$ .025 | .639 $\pm$ .022 | .588 $\pm$ .021
CSDS3 | .99 | .840 $\pm$ .003 | .838 $\pm$ .003 | .840 $\pm$ .003 | .845 $\pm$ .003 | .840 $\pm$ .003 | .835 $\pm$ .003 | .845 $\pm$ .003 | .843 $\pm$ .003
.95 | .846 $\pm$ .003 | .845 $\pm$ .003 | .845 $\pm$ .003 | .851 $\pm$ .003 | .844 $\pm$ .003 | .841 $\pm$ .003 | .850 $\pm$ .003 | .848 $\pm$ .003
.90 | .852 $\pm$ .003 | .851 $\pm$ .003 | .851 $\pm$ .003 | .858 $\pm$ .003 | .851 $\pm$ .003 | .847 $\pm$ .003 | .857 $\pm$ .003 | .856 $\pm$ .003
.85 | .859 $\pm$ .003 | .859 $\pm$ .003 | .858 $\pm$ .003 | .865 $\pm$ .003 | .856 $\pm$ .003 | .855 $\pm$ .004 | .864 $\pm$ .003 | .863 $\pm$ .003
.80 | .866 $\pm$ .003 | .865 $\pm$ .004 | .869 $\pm$ .003 | .871 $\pm$ .003 | .865 $\pm$ .003 | .861 $\pm$ .004 | .870 $\pm$ .003 | .869 $\pm$ .003
.75 | .873 $\pm$ .003 | .873 $\pm$ .004 | .875 $\pm$ .003 | .879 $\pm$ .003 | .872 $\pm$ .003 | .868 $\pm$ .004 | .878 $\pm$ .003 | .877 $\pm$ .003
| # | 3/42 | 0/42 | 12/42 | 29/42 | 1/42 | 0/42 | 18/42 | 24/42

Table 7: Selective AUC for AUCross and PlugInAUC using different classifiers
(1,000 bootstrap runs over the test set, results as mean $\pm$ stdev).

|  | Selective Accuracy
---|---|---
|  | AUCross | PlugInAUC
| $c$ | Logistic | RandForest | ResNet | XGBoost | Logistic | RandForest | ResNet | XGBoost
Adult | .99 | .846 $\pm$ .003 | .838 $\pm$ .004 | .844 $\pm$ .003 | .870 $\pm$ .003 | .847 $\pm$ .003 | .839 $\pm$ .004 | .844 $\pm$ .003 | .869 $\pm$ .003
.95 | .851 $\pm$ .003 | .842 $\pm$ .004 | .847 $\pm$ .004 | .874 $\pm$ .003 | .851 $\pm$ .003 | .842 $\pm$ .004 | .847 $\pm$ .003 | .874 $\pm$ .003
.90 | .856 $\pm$ .003 | .847 $\pm$ .004 | .853 $\pm$ .004 | .883 $\pm$ .003 | .855 $\pm$ .003 | .847 $\pm$ .004 | .853 $\pm$ .003 | .880 $\pm$ .003
.85 | .863 $\pm$ .004 | .853 $\pm$ .004 | .861 $\pm$ .004 | .890 $\pm$ .003 | .861 $\pm$ .004 | .851 $\pm$ .004 | .860 $\pm$ .004 | .888 $\pm$ .003
.80 | .870 $\pm$ .004 | .860 $\pm$ .004 | .870 $\pm$ .004 | .901 $\pm$ .003 | .868 $\pm$ .004 | .858 $\pm$ .004 | .866 $\pm$ .004 | .896 $\pm$ .003
.75 | .879 $\pm$ .004 | .868 $\pm$ .004 | .883 $\pm$ .003 | .911 $\pm$ .003 | .875 $\pm$ .004 | .864 $\pm$ .004 | .874 $\pm$ .004 | .905 $\pm$ .003
Lending | .99 | .848 $\pm$ .001 | .878 $\pm$ .001 | .883 $\pm$ .001 | .903 $\pm$ .001 | .848 $\pm$ .001 | .877 $\pm$ .001 | .874 $\pm$ .001 | .904 $\pm$ .001
.95 | .857 $\pm$ .001 | .890 $\pm$ .001 | .894 $\pm$ .001 | .913 $\pm$ .001 | .857 $\pm$ .001 | .888 $\pm$ .001 | .884 $\pm$ .001 | .914 $\pm$ .001
.90 | .869 $\pm$ .001 | .903 $\pm$ .001 | .909 $\pm$ .001 | .927 $\pm$ .001 | .869 $\pm$ .001 | .902 $\pm$ .001 | .896 $\pm$ .001 | .928 $\pm$ .001
.85 | .882 $\pm$ .001 | .918 $\pm$ .001 | .924 $\pm$ .001 | .941 $\pm$ .001 | .882 $\pm$ .001 | .917 $\pm$ .001 | .910 $\pm$ .001 | .942 $\pm$ .001
.80 | .896 $\pm$ .001 | .933 $\pm$ .001 | .940 $\pm$ .001 | .957 $\pm$ .001 | .895 $\pm$ .001 | .936 $\pm$ .001 | .925 $\pm$ .001 | .957 $\pm$ .001
.75 | .911 $\pm$ .001 | .952 $\pm$ .001 | .957 $\pm$ .001 | .975 $\pm$ .001 | .911 $\pm$ .001 | .951 $\pm$ .001 | .941 $\pm$ .001 | .974 $\pm$ .001
GiveMe | .99 | .934 $\pm$ .002 | .935 $\pm$ .002 | .933 $\pm$ .002 | .937 $\pm$ .002 | .934 $\pm$ .002 | .935 $\pm$ .002 | .933 $\pm$ .002 | .936 $\pm$ .002
.95 | .934 $\pm$ .002 | .935 $\pm$ .002 | .929 $\pm$ .002 | .937 $\pm$ .002 | .934 $\pm$ .002 | .935 $\pm$ .002 | .932 $\pm$ .002 | .936 $\pm$ .002
.90 | .934 $\pm$ .002 | .934 $\pm$ .002 | .928 $\pm$ .002 | .937 $\pm$ .002 | .934 $\pm$ .002 | .934 $\pm$ .002 | .931 $\pm$ .002 | .936 $\pm$ .002
.85 | .933 $\pm$ .002 | .932 $\pm$ .002 | .928 $\pm$ .002 | .937 $\pm$ .002 | .934 $\pm$ .002 | .932 $\pm$ .002 | .930 $\pm$ .002 | .936 $\pm$ .002
.80 | .934 $\pm$ .002 | .933 $\pm$ .002 | .926 $\pm$ .002 | .938 $\pm$ .002 | .935 $\pm$ .002 | .932 $\pm$ .002 | .929 $\pm$ .002 | .937 $\pm$ .002
.75 | .934 $\pm$ .002 | .933 $\pm$ .002 | .923 $\pm$ .003 | .939 $\pm$ .002 | .935 $\pm$ .002 | .929 $\pm$ .002 | .929 $\pm$ .002 | .937 $\pm$ .002
UCICredit | .99 | .784 $\pm$ .005 | .806 $\pm$ .005 | .814 $\pm$ .005 | .805 $\pm$ .005 | .785 $\pm$ .005 | .805 $\pm$ .005 | .814 $\pm$ .005 | .805 $\pm$ .005
.95 | .780 $\pm$ .005 | .806 $\pm$ .005 | .812 $\pm$ .005 | .805 $\pm$ .005 | .783 $\pm$ .005 | .804 $\pm$ .005 | .812 $\pm$ .005 | .804 $\pm$ .005
.90 | .777 $\pm$ .006 | .803 $\pm$ .005 | .809 $\pm$ .005 | .803 $\pm$ .005 | .777 $\pm$ .006 | .801 $\pm$ .006 | .810 $\pm$ .005 | .803 $\pm$ .005
.85 | .773 $\pm$ .006 | .801 $\pm$ .006 | .808 $\pm$ .006 | .800 $\pm$ .006 | .774 $\pm$ .006 | .799 $\pm$ .006 | .806 $\pm$ .006 | .802 $\pm$ .005
.80 | .769 $\pm$ .006 | .800 $\pm$ .006 | .806 $\pm$ .006 | .796 $\pm$ .006 | .770 $\pm$ .006 | .795 $\pm$ .006 | .804 $\pm$ .006 | .800 $\pm$ .006
.75 | .765 $\pm$ .006 | .799 $\pm$ .006 | .802 $\pm$ .006 | .794 $\pm$ .006 | .765 $\pm$ .006 | .793 $\pm$ .006 | .806 $\pm$ .006 | .798 $\pm$ .006
CSDS1 | .99 | .857 $\pm$ .002 | .846 $\pm$ .002 | .857 $\pm$ .002 | .856 $\pm$ .002 | .857 $\pm$ .002 | .847 $\pm$ .002 | .857 $\pm$ .002 | .856 $\pm$ .002
.95 | .856 $\pm$ .002 | .846 $\pm$ .002 | .856 $\pm$ .002 | .855 $\pm$ .002 | .856 $\pm$ .002 | .845 $\pm$ .002 | .856 $\pm$ .002 | .854 $\pm$ .002
.90 | .854 $\pm$ .002 | .844 $\pm$ .002 | .854 $\pm$ .002 | .854 $\pm$ .002 | .855 $\pm$ .002 | .844 $\pm$ .002 | .854 $\pm$ .002 | .853 $\pm$ .002
.85 | .852 $\pm$ .002 | .842 $\pm$ .002 | .853 $\pm$ .002 | .852 $\pm$ .002 | .852 $\pm$ .002 | .841 $\pm$ .002 | .853 $\pm$ .002 | .851 $\pm$ .002
.80 | .851 $\pm$ .002 | .840 $\pm$ .002 | .851 $\pm$ .002 | .850 $\pm$ .002 | .851 $\pm$ .002 | .840 $\pm$ .002 | .851 $\pm$ .002 | .850 $\pm$ .002
.75 | .850 $\pm$ .002 | .836 $\pm$ .002 | .849 $\pm$ .002 | .848 $\pm$ .002 | .850 $\pm$ .002 | .834 $\pm$ .002 | .849 $\pm$ .002 | .848 $\pm$ .002
CSDS2 | .99 | .982 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .981 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
.95 | .982 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002 | .981 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
.90 | .982 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .981 $\pm$ .002 | .982 $\pm$ .002 | .981 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
.85 | .982 $\pm$ .002 | .980 $\pm$ .002 | .981 $\pm$ .002 | .981 $\pm$ .002 | .982 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
.80 | .982 $\pm$ .002 | .980 $\pm$ .002 | .980 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
.75 | .982 $\pm$ .002 | .980 $\pm$ .002 | .980 $\pm$ .002 | .980 $\pm$ .002 | .982 $\pm$ .002 | .979 $\pm$ .002 | .982 $\pm$ .002 | .982 $\pm$ .002
CSDS3 | .99 | .808 $\pm$ .003 | .808 $\pm$ .003 | .806 $\pm$ .003 | .811 $\pm$ .003 | .809 $\pm$ .003 | .804 $\pm$ .003 | .808 $\pm$ .003 | .808 $\pm$ .003
.95 | .810 $\pm$ .003 | .810 $\pm$ .003 | .807 $\pm$ .003 | .813 $\pm$ .003 | .809 $\pm$ .003 | .805 $\pm$ .003 | .809 $\pm$ .003 | .810 $\pm$ .003
.90 | .811 $\pm$ .003 | .813 $\pm$ .003 | .808 $\pm$ .003 | .815 $\pm$ .003 | .809 $\pm$ .003 | .807 $\pm$ .003 | .811 $\pm$ .003 | .814 $\pm$ .003
.85 | .812 $\pm$ .003 | .817 $\pm$ .003 | .810 $\pm$ .003 | .818 $\pm$ .003 | .810 $\pm$ .003 | .811 $\pm$ .003 | .813 $\pm$ .003 | .817 $\pm$ .003
.80 | .814 $\pm$ .003 | .820 $\pm$ .003 | .817 $\pm$ .003 | .822 $\pm$ .003 | .812 $\pm$ .003 | .815 $\pm$ .003 | .816 $\pm$ .003 | .821 $\pm$ .003
.75 | .817 $\pm$ .003 | .826 $\pm$ .003 | .820 $\pm$ .003 | .828 $\pm$ .003 | .815 $\pm$ .003 | .819 $\pm$ .003 | .821 $\pm$ .003 | .825 $\pm$ .003
| # | 12/42 | 0/42 | 14/42 | 26/42 | 13/42 | 0/42 | 17/42 | 29/42

Table 8: Selective accuracy for AUCross and PlugInAUC using different
classifiers (1,000 bootstrap runs over the test set, results as mean $\pm$
stdev).

|  | Positive Rate
---|---|---
|  | AUCross | PlugInAUC
| $c$ | Logistic | RandForest | ResNet | XGBoost | Logistic | RandForest | ResNet | XGBoost
Adult | .99 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004
.95 | .245 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .247 $\pm$ .004 | .247 $\pm$ .004 | .246 $\pm$ .004
.90 | .245 $\pm$ .004 | .246 $\pm$ .004 | .245 $\pm$ .004 | .246 $\pm$ .004 | .247 $\pm$ .004 | .248 $\pm$ .004 | .246 $\pm$ .004 | .248 $\pm$ .004
.85 | .245 $\pm$ .004 | .245 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004 | .248 $\pm$ .004 | .250 $\pm$ .004 | .246 $\pm$ .004 | .246 $\pm$ .004
.80 | .244 $\pm$ .004 | .244 $\pm$ .004 | .242 $\pm$ .004 | .244 $\pm$ .004 | .247 $\pm$ .004 | .250 $\pm$ .004 | .246 $\pm$ .004 | .247 $\pm$ .004
.75 | .243 $\pm$ .004 | .243 $\pm$ .005 | .238 $\pm$ .005 | .243 $\pm$ .005 | .248 $\pm$ .004 | .250 $\pm$ .005 | .245 $\pm$ .005 | .248 $\pm$ .005
Lending | .99 | .224 $\pm$ .001 | .222 $\pm$ .001 | .223 $\pm$ .001 | .224 $\pm$ .001 | .224 $\pm$ .001 | .222 $\pm$ .001 | .224 $\pm$ .001 | .224 $\pm$ .001
.95 | .216 $\pm$ .001 | .213 $\pm$ .001 | .214 $\pm$ .001 | .216 $\pm$ .001 | .217 $\pm$ .001 | .213 $\pm$ .001 | .216 $\pm$ .001 | .216 $\pm$ .001
.90 | .206 $\pm$ .001 | .202 $\pm$ .001 | .202 $\pm$ .001 | .205 $\pm$ .001 | .207 $\pm$ .001 | .202 $\pm$ .001 | .207 $\pm$ .001 | .205 $\pm$ .001
.85 | .196 $\pm$ .001 | .190 $\pm$ .001 | .190 $\pm$ .001 | .193 $\pm$ .001 | .196 $\pm$ .001 | .190 $\pm$ .001 | .196 $\pm$ .001 | .195 $\pm$ .001
.80 | .184 $\pm$ .001 | .179 $\pm$ .001 | .177 $\pm$ .001 | .181 $\pm$ .001 | .185 $\pm$ .001 | .175 $\pm$ .001 | .184 $\pm$ .001 | .182 $\pm$ .001
.75 | .171 $\pm$ .001 | .165 $\pm$ .001 | .164 $\pm$ .001 | .168 $\pm$ .001 | .172 $\pm$ .001 | .163 $\pm$ .001 | .171 $\pm$ .001 | .170 $\pm$ .001
GiveMe | .99 | .067 $\pm$ .002 | .068 $\pm$ .002 | .068 $\pm$ .002 | .067 $\pm$ .002 | .068 $\pm$ .002 | .068 $\pm$ .002 | .068 $\pm$ .002 | .068 $\pm$ .002
.95 | .067 $\pm$ .002 | .068 $\pm$ .002 | .072 $\pm$ .002 | .068 $\pm$ .002 | .067 $\pm$ .002 | .068 $\pm$ .002 | .069 $\pm$ .002 | .068 $\pm$ .002
.90 | .068 $\pm$ .002 | .070 $\pm$ .002 | .072 $\pm$ .002 | .068 $\pm$ .002 | .067 $\pm$ .002 | .069 $\pm$ .002 | .070 $\pm$ .002 | .068 $\pm$ .002
.85 | .068 $\pm$ .002 | .072 $\pm$ .002 | .073 $\pm$ .002 | .068 $\pm$ .002 | .067 $\pm$ .002 | .071 $\pm$ .002 | .071 $\pm$ .002 | .068 $\pm$ .002
.80 | .067 $\pm$ .002 | .071 $\pm$ .002 | .075 $\pm$ .002 | .068 $\pm$ .002 | .067 $\pm$ .002 | .071 $\pm$ .002 | .072 $\pm$ .002 | .068 $\pm$ .002
.75 | .067 $\pm$ .002 | .070 $\pm$ .002 | .077 $\pm$ .003 | .067 $\pm$ .002 | .066 $\pm$ .002 | .075 $\pm$ .002 | .073 $\pm$ .002 | .067 $\pm$ .002
UCICredit | .99 | .222 $\pm$ .005 | .226 $\pm$ .006 | .223 $\pm$ .005 | .222 $\pm$ .005 | .222 $\pm$ .005 | .224 $\pm$ .005 | .222 $\pm$ .005 | .222 $\pm$ .005
.95 | .227 $\pm$ .006 | .226 $\pm$ .006 | .226 $\pm$ .006 | .223 $\pm$ .005 | .225 $\pm$ .005 | .227 $\pm$ .006 | .226 $\pm$ .006 | .224 $\pm$ .005
.90 | .231 $\pm$ .006 | .232 $\pm$ .006 | .232 $\pm$ .006 | .227 $\pm$ .006 | .230 $\pm$ .006 | .232 $\pm$ .006 | .229 $\pm$ .006 | .227 $\pm$ .006
.85 | .235 $\pm$ .006 | .235 $\pm$ .006 | .237 $\pm$ .006 | .232 $\pm$ .006 | .234 $\pm$ .006 | .234 $\pm$ .006 | .236 $\pm$ .006 | .230 $\pm$ .006
.80 | .239 $\pm$ .006 | .239 $\pm$ .006 | .242 $\pm$ .006 | .238 $\pm$ .006 | .238 $\pm$ .006 | .241 $\pm$ .006 | .240 $\pm$ .006 | .233 $\pm$ .006
.75 | .243 $\pm$ .006 | .243 $\pm$ .007 | .249 $\pm$ .007 | .242 $\pm$ .006 | .244 $\pm$ .006 | .245 $\pm$ .006 | .241 $\pm$ .006 | .238 $\pm$ .006
CSDS1 | .99 | .144 $\pm$ .002 | .147 $\pm$ .002 | .145 $\pm$ .002 | .145 $\pm$ .002 | .144 $\pm$ .002 | .146 $\pm$ .002 | .144 $\pm$ .002 | .144 $\pm$ .002
.95 | .145 $\pm$ .002 | .147 $\pm$ .002 | .146 $\pm$ .002 | .146 $\pm$ .002 | .145 $\pm$ .002 | .147 $\pm$ .002 | .145 $\pm$ .002 | .146 $\pm$ .002
.90 | .147 $\pm$ .002 | .148 $\pm$ .002 | .147 $\pm$ .002 | .147 $\pm$ .002 | .147 $\pm$ .002 | .148 $\pm$ .002 | .147 $\pm$ .002 | .147 $\pm$ .002
.85 | .149 $\pm$ .002 | .150 $\pm$ .002 | .148 $\pm$ .002 | .149 $\pm$ .002 | .149 $\pm$ .002 | .150 $\pm$ .002 | .148 $\pm$ .002 | .149 $\pm$ .002
.80 | .150 $\pm$ .002 | .151 $\pm$ .002 | .150 $\pm$ .002 | .151 $\pm$ .002 | .150 $\pm$ .002 | .151 $\pm$ .002 | .150 $\pm$ .002 | .150 $\pm$ .002
.75 | .152 $\pm$ .002 | .154 $\pm$ .002 | .152 $\pm$ .002 | .153 $\pm$ .002 | .151 $\pm$ .002 | .155 $\pm$ .002 | .152 $\pm$ .002 | .153 $\pm$ .002
CSDS2 | .99 | .019 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .020 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002
.95 | .019 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .020 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002
.90 | .019 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .020 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002
.85 | .019 $\pm$ .002 | .021 $\pm$ .002 | .020 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .018 $\pm$ .002
.80 | .019 $\pm$ .002 | .021 $\pm$ .002 | .021 $\pm$ .002 | .020 $\pm$ .002 | .019 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002
.75 | .019 $\pm$ .002 | .021 $\pm$ .002 | .021 $\pm$ .002 | .021 $\pm$ .002 | .019 $\pm$ .002 | .022 $\pm$ .002 | .019 $\pm$ .002 | .019 $\pm$ .002
CSDS3 | .99 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003 | .254 $\pm$ .003
.95 | .255 $\pm$ .003 | .255 $\pm$ .003 | .255 $\pm$ .003 | .254 $\pm$ .003 | .256 $\pm$ .003 | .255 $\pm$ .003 | .256 $\pm$ .003 | .255 $\pm$ .003
.90 | .258 $\pm$ .003 | .256 $\pm$ .003 | .257 $\pm$ .003 | .255 $\pm$ .003 | .259 $\pm$ .003 | .256 $\pm$ .003 | .258 $\pm$ .003 | .254 $\pm$ .003
.85 | .261 $\pm$ .003 | .257 $\pm$ .004 | .259 $\pm$ .003 | .256 $\pm$ .003 | .263 $\pm$ .003 | .257 $\pm$ .004 | .260 $\pm$ .003 | .255 $\pm$ .004
.80 | .263 $\pm$ .004 | .257 $\pm$ .004 | .258 $\pm$ .004 | .256 $\pm$ .004 | .265 $\pm$ .004 | .256 $\pm$ .004 | .261 $\pm$ .004 | .255 $\pm$ .004
.75 | .266 $\pm$ .004 | .257 $\pm$ .004 | .259 $\pm$ .004 | .255 $\pm$ .004 | .268 $\pm$ .004 | .257 $\pm$ .004 | .261 $\pm$ .004 | .256 $\pm$ .004
| # | 16/42 | 22/42 | 16/42 | 11/42 | 14/42 | 27/42 | 13/42 | 4/42

Table 9: Positive rate for AUCross and PlugInAUC using different classifiers
(1,000 bootstrap runs over the test set, results as mean $\pm$ stdev).

|  | Empirical Coverage | Selective AUC
---|---|---|---
| $c$ | $K=2$ | $K=3$ | $K=5$ | $K=7$ | $K=10$ | $K=2$ | $K=3$ | $K=5$ | $K=7$ | $K=10$
Adult | .99 | .990 $\pm$ .001 | .991 $\pm$ .001 | .989 $\pm$ .001 | .991 $\pm$ .001 | .991 $\pm$ .001 | .929 $\pm$ .003 | .928 $\pm$ .003 | .929 $\pm$ .003 | .928 $\pm$ .003 | .928 $\pm$ .003
.95 | .952 $\pm$ .002 | .951 $\pm$ .002 | .950 $\pm$ .002 | .949 $\pm$ .002 | .947 $\pm$ .002 | .935 $\pm$ .003 | .935 $\pm$ .003 | .935 $\pm$ .003 | .935 $\pm$ .003 | .936 $\pm$ .003
.90 | .896 $\pm$ .003 | .897 $\pm$ .003 | .896 $\pm$ .003 | .896 $\pm$ .003 | .900 $\pm$ .003 | .943 $\pm$ .002 | .943 $\pm$ .002 | .943 $\pm$ .002 | .943 $\pm$ .002 | .943 $\pm$ .002
.85 | .848 $\pm$ .003 | .850 $\pm$ .003 | .849 $\pm$ .003 | .848 $\pm$ .003 | .849 $\pm$ .003 | .950 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002
.80 | .796 $\pm$ .004 | .797 $\pm$ .004 | .797 $\pm$ .004 | .799 $\pm$ .004 | .800 $\pm$ .004 | .958 $\pm$ .002 | .958 $\pm$ .002 | .958 $\pm$ .002 | .957 $\pm$ .002 | .957 $\pm$ .002
.75 | .750 $\pm$ .004 | .752 $\pm$ .004 | .750 $\pm$ .004 | .753 $\pm$ .004 | .751 $\pm$ .004 | .963 $\pm$ .002 | .963 $\pm$ .002 | .963 $\pm$ .002 | .964 $\pm$ .002 | .964 $\pm$ .002
Lending | .99 | .996 $\pm$ .001 | .996 $\pm$ .001 | .996 $\pm$ .001 | .996 $\pm$ .001 | .996 $\pm$ .001 | .983 $\pm$ .001 | .983 $\pm$ .001 | .983 $\pm$ .001 | .983 $\pm$ .001 | .983 $\pm$ .001
.95 | .980 $\pm$ .001 | .980 $\pm$ .001 | .980 $\pm$ .001 | .980 $\pm$ .001 | .980 $\pm$ .001 | .985 $\pm$ .001 | .985 $\pm$ .001 | .985 $\pm$ .001 | .985 $\pm$ .001 | .985 $\pm$ .001
.90 | .958 $\pm$ .001 | .959 $\pm$ .001 | .959 $\pm$ .001 | .959 $\pm$ .001 | .959 $\pm$ .001 | .987 $\pm$ .001 | .987 $\pm$ .001 | .987 $\pm$ .001 | .987 $\pm$ .001 | .987 $\pm$ .001
.85 | .935 $\pm$ .001 | .936 $\pm$ .001 | .936 $\pm$ .001 | .936 $\pm$ .001 | .936 $\pm$ .001 | .989 $\pm$ .001 | .989 $\pm$ .001 | .989 $\pm$ .001 | .989 $\pm$ .001 | .989 $\pm$ .001
.80 | .912 $\pm$ .001 | .912 $\pm$ .001 | .912 $\pm$ .001 | .912 $\pm$ .001 | .912 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001
.75 | .886 $\pm$ .001 | .886 $\pm$ .001 | .886 $\pm$ .001 | .886 $\pm$ .001 | .886 $\pm$ .001 | .992 $\pm$ .001 | .992 $\pm$ .001 | .992 $\pm$ .001 | .992 $\pm$ .001 | .992 $\pm$ .001
GiveMe | .99 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .991 $\pm$ .001 | .868 $\pm$ .004 | .868 $\pm$ .004 | .868 $\pm$ .004 | .868 $\pm$ .004 | .867 $\pm$ .004
.95 | .951 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002 | .950 $\pm$ .002 | .874 $\pm$ .004 | .874 $\pm$ .004 | .874 $\pm$ .004 | .874 $\pm$ .004 | .874 $\pm$ .004
.90 | .900 $\pm$ .002 | .899 $\pm$ .002 | .898 $\pm$ .002 | .899 $\pm$ .002 | .898 $\pm$ .002 | .882 $\pm$ .004 | .882 $\pm$ .004 | .883 $\pm$ .004 | .883 $\pm$ .004 | .883 $\pm$ .004
.85 | .850 $\pm$ .002 | .849 $\pm$ .002 | .848 $\pm$ .002 | .849 $\pm$ .002 | .849 $\pm$ .002 | .890 $\pm$ .004 | .890 $\pm$ .004 | .890 $\pm$ .004 | .890 $\pm$ .004 | .890 $\pm$ .004
.80 | .794 $\pm$ .003 | .795 $\pm$ .003 | .795 $\pm$ .003 | .796 $\pm$ .003 | .796 $\pm$ .003 | .898 $\pm$ .004 | .897 $\pm$ .004 | .898 $\pm$ .004 | .898 $\pm$ .004 | .897 $\pm$ .004
.75 | .747 $\pm$ .003 | .749 $\pm$ .003 | .748 $\pm$ .003 | .749 $\pm$ .003 | .750 $\pm$ .003 | .904 $\pm$ .004 | .903 $\pm$ .004 | .904 $\pm$ .004 | .904 $\pm$ .004 | .903 $\pm$ .004
UCICredit | .99 | .989 $\pm$ .002 | .989 $\pm$ .002 | .988 $\pm$ .002 | .988 $\pm$ .002 | .990 $\pm$ .002 | .772 $\pm$ .008 | .772 $\pm$ .007 | .772 $\pm$ .008 | .772 $\pm$ .008 | .772 $\pm$ .008
.95 | .944 $\pm$ .003 | .946 $\pm$ .003 | .944 $\pm$ .003 | .945 $\pm$ .003 | .946 $\pm$ .003 | .778 $\pm$ .008 | .778 $\pm$ .008 | .778 $\pm$ .008 | .778 $\pm$ .008 | .778 $\pm$ .008
.90 | .886 $\pm$ .004 | .894 $\pm$ .004 | .896 $\pm$ .004 | .895 $\pm$ .004 | .900 $\pm$ .004 | .786 $\pm$ .008 | .785 $\pm$ .008 | .785 $\pm$ .008 | .784 $\pm$ .008 | .784 $\pm$ .008
.85 | .830 $\pm$ .005 | .838 $\pm$ .005 | .840 $\pm$ .005 | .840 $\pm$ .005 | .843 $\pm$ .005 | .794 $\pm$ .008 | .793 $\pm$ .007 | .792 $\pm$ .008 | .792 $\pm$ .008 | .792 $\pm$ .008
.80 | .775 $\pm$ .005 | .789 $\pm$ .005 | .789 $\pm$ .005 | .788 $\pm$ .005 | .795 $\pm$ .005 | .802 $\pm$ .008 | .800 $\pm$ .008 | .800 $\pm$ .008 | .800 $\pm$ .008 | .800 $\pm$ .008
.75 | .720 $\pm$ .006 | .737 $\pm$ .006 | .737 $\pm$ .006 | .735 $\pm$ .006 | .745 $\pm$ .006 | .811 $\pm$ .008 | .808 $\pm$ .008 | .809 $\pm$ .008 | .809 $\pm$ .008 | .807 $\pm$ .008
CSDS1 | .99 | .991 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .686 $\pm$ .003 | .686 $\pm$ .003 | .686 $\pm$ .003 | .686 $\pm$ .003 | .686 $\pm$ .003
.95 | .950 $\pm$ .001 | .950 $\pm$ .001 | .950 $\pm$ .001 | .949 $\pm$ .001 | .950 $\pm$ .001 | .689 $\pm$ .003 | .689 $\pm$ .003 | .689 $\pm$ .003 | .689 $\pm$ .003 | .689 $\pm$ .003
.90 | .898 $\pm$ .002 | .899 $\pm$ .002 | .898 $\pm$ .002 | .898 $\pm$ .002 | .898 $\pm$ .002 | .693 $\pm$ .003 | .693 $\pm$ .003 | .693 $\pm$ .003 | .693 $\pm$ .003 | .693 $\pm$ .003
.85 | .848 $\pm$ .002 | .849 $\pm$ .002 | .849 $\pm$ .002 | .848 $\pm$ .002 | .848 $\pm$ .002 | .697 $\pm$ .003 | .697 $\pm$ .003 | .697 $\pm$ .003 | .697 $\pm$ .003 | .697 $\pm$ .003
.80 | .796 $\pm$ .002 | .799 $\pm$ .002 | .799 $\pm$ .002 | .797 $\pm$ .002 | .797 $\pm$ .002 | .702 $\pm$ .004 | .702 $\pm$ .004 | .702 $\pm$ .004 | .702 $\pm$ .004 | .702 $\pm$ .004
.75 | .746 $\pm$ .002 | .748 $\pm$ .002 | .748 $\pm$ .002 | .747 $\pm$ .002 | .747 $\pm$ .002 | .706 $\pm$ .004 | .706 $\pm$ .004 | .706 $\pm$ .004 | .706 $\pm$ .004 | .706 $\pm$ .004
CSDS2 | .99 | .990 $\pm$ .001 | .992 $\pm$ .001 | .990 $\pm$ .001 | .990 $\pm$ .001 | .992 $\pm$ .001 | .615 $\pm$ .020 | .618 $\pm$ .020 | .616 $\pm$ .020 | .618 $\pm$ .020 | .617 $\pm$ .020
.95 | .952 $\pm$ .002 | .951 $\pm$ .002 | .952 $\pm$ .002 | .956 $\pm$ .002 | .956 $\pm$ .002 | .622 $\pm$ .020 | .619 $\pm$ .020 | .619 $\pm$ .020 | .619 $\pm$ .020 | .621 $\pm$ .020
.90 | .904 $\pm$ .003 | .904 $\pm$ .003 | .904 $\pm$ .003 | .909 $\pm$ .003 | .911 $\pm$ .003 | .620 $\pm$ .020 | .626 $\pm$ .020 | .620 $\pm$ .020 | .625 $\pm$ .020 | .624 $\pm$ .020
.85 | .854 $\pm$ .004 | .854 $\pm$ .004 | .860 $\pm$ .004 | .861 $\pm$ .003 | .866 $\pm$ .003 | .622 $\pm$ .020 | .629 $\pm$ .021 | .631 $\pm$ .021 | .626 $\pm$ .021 | .626 $\pm$ .021
.80 | .808 $\pm$ .004 | .804 $\pm$ .004 | .811 $\pm$ .004 | .814 $\pm$ .004 | .819 $\pm$ .004 | .628 $\pm$ .021 | .636 $\pm$ .021 | .631 $\pm$ .021 | .632 $\pm$ .021 | .630 $\pm$ .021
.75 | .760 $\pm$ .004 | .754 $\pm$ .004 | .760 $\pm$ .004 | .765 $\pm$ .004 | .770 $\pm$ .004 | .628 $\pm$ .021 | .637 $\pm$ .021 | .635 $\pm$ .021 | .634 $\pm$ .021 | .630 $\pm$ .021
CSDS3 | .99 | .991 $\pm$ .001 | .992 $\pm$ .001 | .991 $\pm$ .001 | .991 $\pm$ .001 | .991 $\pm$ .001 | .851 $\pm$ .003 | .851 $\pm$ .003 | .851 $\pm$ .003 | .851 $\pm$ .003 | .851 $\pm$ .003
.95 | .948 $\pm$ .002 | .950 $\pm$ .002 | .948 $\pm$ .002 | .950 $\pm$ .002 | .951 $\pm$ .002 | .858 $\pm$ .003 | .857 $\pm$ .003 | .858 $\pm$ .003 | .857 $\pm$ .003 | .857 $\pm$ .003
.90 | .902 $\pm$ .002 | .901 $\pm$ .002 | .901 $\pm$ .002 | .902 $\pm$ .002 | .901 $\pm$ .002 | .865 $\pm$ .003 | .865 $\pm$ .003 | .865 $\pm$ .003 | .865 $\pm$ .003 | .865 $\pm$ .003
.85 | .850 $\pm$ .003 | .853 $\pm$ .003 | .849 $\pm$ .003 | .850 $\pm$ .003 | .850 $\pm$ .003 | .873 $\pm$ .003 | .873 $\pm$ .003 | .873 $\pm$ .003 | .873 $\pm$ .003 | .873 $\pm$ .003
.80 | .809 $\pm$ .003 | .810 $\pm$ .003 | .805 $\pm$ .003 | .805 $\pm$ .003 | .805 $\pm$ .003 | .879 $\pm$ .003 | .879 $\pm$ .003 | .880 $\pm$ .003 | .880 $\pm$ .003 | .880 $\pm$ .003
.75 | .760 $\pm$ .003 | .759 $\pm$ .003 | .759 $\pm$ .003 | .758 $\pm$ .003 | .758 $\pm$ .003 | .886 $\pm$ .003 | .887 $\pm$ .003 | .887 $\pm$ .003 | .887 $\pm$ .003 | .887 $\pm$ .003
| # | 15/42 | 23/42 | 20/42 | 16/42 | 24/42 | 29/42 | 28/42 | 31/42 | 30/42 | 27/42
| $V$ | .019 $\pm$ .038 | .017 $\pm$ .038 | .018 $\pm$ .038 | .018 $\pm$ .038 | .018 $\pm$ .038 |  |  |  |  |

Table 10: Performance metrics for AUCross using different number of folds $K$
over tabular datasets (1,000 bootstrap runs over the test set, results as mean
$\pm$ stdev).