# Which models are innately best at uncertainty estimation?

Ido Galil
Technion
<EMAIL_ADDRESS>&Mohammed Dabbah
Amazon
<EMAIL_ADDRESS>&Ran El-Yaniv
Technion, Deci.AI
<EMAIL_ADDRESS>
###### Abstract

Due to the comprehensive nature of this paper, it has been updated and split
into two separate papers: Galil et al. (2023a), focusing on the framework to
benchmark class-out-of-distribution data and the following performance
analysis, and Galil et al. (2023b) which covers the analysis of 523 ImageNet
classifiers for their uncertainty estimation performance (ranking, calibration
and selective performance) in an in-distribution setting. We recommend reading
them instead.

Deep neural networks must be equipped with an uncertainty estimation mechanism
when deployed for risk-sensitive tasks. This paper studies the relationship
between deep architectures and their training regimes with their corresponding
selective prediction and uncertainty estimation performance. We consider both
in-distribution uncertainties and class-out-of-distribution ones. Moreover, we
consider some of the most popular estimation performance metrics previously
proposed including AUROC, ECE, AURC, and coverage for selective accuracy
constraint. We present a novel and comprehensive study of selective prediction
and the uncertainty estimation performance of 484 existing pretrained deep
ImageNet classifiers that are available at popular repositories. We identify
numerous and previously unknown factors that affect uncertainty estimation and
examine the relationships between the different metrics. We find that
distillation-based training regimes consistently yield better uncertainty
estimations than other training schemes such as vanilla training, pretraining
on a larger dataset and adversarial training. We also provide strong empirical
evidence showing that ViT is by far the most superior architecture in terms of
uncertainty estimation performance, judging by any aspect, in both in-
distribution and class-out-of-distribution scenarios.

##  1 Introduction

Deep neural networks (DNNs) show great performance in a wide variety of
application domains including computer vision, natural language understanding
and audio processing. Successful deployment of these models, however, is
critically dependent on providing an effective _uncertainty estimation_ of
their predictions in the form of some kind of _selective prediction_ or
providing a probabilistic confidence score for their predictions.

Figure 1: A comparison of 484 models by their AUROC ($\times 100$, higher is
better) and -log(ECE) (higher is better) on ImageNet. Each marker’s size is
determined by the model’s number of parameters. A full version graph is given
in Figure 8. Distilled models are better than non-distilled ones. ViT models
are naturally better at all aspects of uncertainty estimation, while
EfficientNet-V2 and GENet models are worse.

But how should we evaluate the performance of uncertainty estimation? Let us
consider two classification models for the stock market that predict whether a
stock’s value is about to increase, decrease or remain neutral (three-class
classification). Suppose that model A has a 95% true accuracy, and generates a
confidence score of 0.95 on every prediction (even on misclassified
instances); model B has a 40% true accuracy, but always gives a confidence
score of 0.6 on correct predictions, and 0.4 on incorrect ones. Model B can be
utilized easily to generate perfect investment decisions. Using _selective
prediction_ DBLP:journals/corr/GeifmanE17, Model B will reject all investments
on stocks whenever the confidence score is 0.4. While model A offers many more
investment opportunities, each of its predictions carries a 5% risk of
failure.

Among the various metrics proposed for evaluating the performance of
uncertainty estimation are: _Area Under the Receiver Operating Characteristic_
(AUROC or AUC), _Area Under the Risk-Coverage curve_ (AURC) Geifman et al.
(2018), selective risk or coverage for a _selective accuracy constraint_
(SAC), _Negative Log-likelihood_ (NLL), _Expected Calibration Error_ (ECE),
which is often used for evaluating a model’s _calibration_ (see Section 2) and
_Brier score_ Brier (1950). All these metrics are well known and are often
used for comparing the uncertainty estimation performance of models
DBLP:journals/corr/abs-2007-01458; Nado et al. (2021);
DBLP:journals/corr/abs-1902-02476; lakshminarayanan2017simple. Somewhat
surprisingly, NLL, Brier, AURC, and ECE all fail to reveal the uncertainty
superiority of Model B in our investment example (see Appendix A for the
calculations). Both AUROC and SAC, on the other hand, reveal the advantage of
Model B perfectly (see Appendix A for details). It is not hard to construct
counter examples where these two metrics fails and others (e.g., ECE) succeed.
The risk-coverage (RC) curve El-Yaniv and Wiener (2010) is perhaps one of the
most informative and practical representations of the overall uncertainty
profile of a given model.

In general, though, two RC curves are not necessarily comparable if one does
not fully dominate the other (see Figure 2). The advantage of scalar metrics
such as the above is that they summarize the model’s overall uncertainty
estimation behavior by reducing it to a single scalar. When not carefully
chosen, however, these reductions could result in a loss of vital information
about the problem (for example, reducing an RC curve to an AURC does not show
that Model B has an optimal 0 risk if the coverage is smaller than 0.4). Thus,
the choice of the “correct” single scalar performance metric unfortunately
must be task-specific. When comparing the uncertainty estimation performance
of deep architectures that exhibit different accuracies, we find that AUROC
and SAC can effectively “normalize” accuracy differences that plague the
usefulness of other metrics (see Section 2). This normalization is essential
to our study where we compare uncertainty performance of hundreds of models
that can greatly differ in their accuracies.

In applications where risk (or coverage) constraints are dictated
DBLP:journals/corr/GeifmanE17, the most straightforward and natural metric is
the SAC (or selective risk), which directly measures the coverage (resp.,
risk) given at the required level of risk (resp., coverage) constraint. We
demonstrate this in Appendix J, evaluating which models give the most coverage
for a SAC of 99%. Sometimes, however, such constraints are unknown in advance,
or even irrelevant, e.g., the constructed model should serve a variety of risk
constraint use cases, or the model may not be allowed to abstain from
predicting at all.

In this paper we conduct a comprehensive study of DNNs’ ability to estimate
uncertainty by evaluating 484 models pretrained on ImageNet Deng et al.
(2009), taken from the PyTorch and timm respositories Paszke et al. (2019);
Wightman (2019). We identify the main factors contributing to or harming the
confidence ranking of predictions (“ranking” for short), calibration and
selective prediction. Furthermore, we also consider the source of uncertainty
as either _internal_ (originating from known classes seen during training) or
_external_ (originating from unseen or unknown class-out-of-distribution
(C-OOD) data) and evaluate these models in multiple ways. After first
evaluating models solely on in-distribution (ID) data, we then define and test
a new model-dependent scheme for evaluating C-OOD detection performance that
divides the C-OOD data into groups according to how difficult it is for the
model to identify classes as external (C-OOD detection).

Our paper contains numerous empirical contributions that are important to
selective prediction and uncertainty estimation. In addition, we propose a
novel scheme for evaluating C-OOD detection performance. Among the most
interesting conclusions our study reveals are: (1) Training regimes
incorporating any kind of _knowledge distillation_ (KD) Hinton et al. (2015)
leads to DNNs with improved uncertainty estimation performance evaluated by
any metric, in both internal and external settings (i.e., leading also to
better C-OOD detection), more than by using any other training tricks (such as
pretraining on a larger dataset, adversarial training, etc.). (2) Some
architectures are naturally superb at all aspects of uncertainty estimation
and in all settings, e.g., vision transformers (ViTs) Dosovitskiy et al.
(2020); Steiner et al. (2021), while other architectures tend to perform
worse, e.g., EfficientNet-V2 and GENet DBLP:journals/corr/abs-2104-00298; Lin
et al. (2020). These results are visualized in Figure 1. (3) The superiority
of ViTs remains even when the comparison considers the models’ sizes—meaning
that for any size, ViTs outperform the competition in uncertainty estimation
performance, as visualized in Appendix B in Figures 9 and 10. (4) The simple
post-training calibration method of _temperature scaling_
DBLP:journals/corr/GuoPSW17, which is known to improve ECE, for the most part
also improves ranking (AUROC) and selective prediction—meaning not only does
it calibrate the probabilistic estimation for each individual instance, but it
also improves the partial order of all instances induced by those improved
estimations, pushing instances more likely to be correct to have higher
confidence than instances less likely to be correct (see Section 3). (5)
Contrary to previous work by DBLP:journals/corr/GuoPSW17, we observe that
while there is a strong correlation between accuracy/number of parameters and
ECE or AUROC within each specific family of models of the same architecture,
the correlation flips between a strong negative and a strong positive
correlation depending on the type of architecture being observed. For example,
as ViT architectures increase in size and accuracy, their ECE deteriorates
while their AUROC improves. The exact opposite, however, could be observed in
XCiTs elnouby2021xcit as their ECE improves with size while their AUROC
deteriorates. (see Appendix G). (6) The best model in terms of AUROC or SAC is
not always the best in terms of calibration, as illustrated in Figure 1, and
the trade-off should be considered when choosing a model based on its
application. Due to lack of space, a number of additional interesting
observations are briefly mentioned in the paper without supporting empirical
evidence (which is provided in the appendix).

##  2 How to evaluate deep uncertainty estimation performance

Let $\mathcal{X}$ be the input space and $\mathcal{Y}$ be the label space. Let
$P(\mathcal{X},\mathcal{Y})$ be an unknown distribution over
$\mathcal{X}\times\mathcal{Y}$. A model $f$ is a prediction function
$f:\mathcal{X}\rightarrow\mathcal{Y}$, and its predicted label for an image
$x$ is denoted by $\hat{y}_{f}(x)$. The model’s _true_ risk w.r.t. $P$ is
$R(f|P)=E_{P(\mathcal{X},\mathcal{Y})}[\ell(f(x),y)]$, where
$\ell:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}$ is a given loss
function, for example, 0/1 loss for classification. Given a labeled set
$S_{m}=\\{(x_{i},y_{i})\\}_{i=1}^{m}\subseteq(\mathcal{X}\times\mathcal{Y})$,
sampled i.i.d. from $P(\mathcal{X},\mathcal{Y})$, the _empirical risk_ of
model $f$ is
$\hat{r}(f|S_{m})\triangleq\frac{1}{m}\sum_{i=1}^{m}\ell(f(x_{i}),y_{i}).$
Following Geifman et al. (2018), for a given model $f$ we define a _confidence
score_ function $\kappa(x,\hat{y}|f)$, where $x\in{\mathcal{X}}$, and
$\hat{y}\in{\mathcal{Y}}$ is the model’s prediction for $x$, as follows. The
function $\kappa$ should quantify confidence in the prediction of $\hat{y}$
for the input $x$, based on signals from model $f$. This function should
induce a partial order over instances in $\mathcal{X}$, and is not required to
distinguish between points with the same score.

The most common and well-known $\kappa$ function for a classification model
$f$ (with softmax at its last layer) is its softmax response values:
$\kappa(x,\hat{y}|f)\triangleq f(x)_{\hat{y}}$ Cordella et al. (1995); De
Stefano et al. (2000). While this is the main $\kappa$ we evaluate, we also
test the popular uncertainty estimation technique of _Monte-Carlo dropout_
(MC-Dropout) gal2016dropout, which is motivated by Bayesian reasoning.
Although these methods use the direct output from $f$, $\kappa$ could be a
different model unrelated to $f$ and unable to affect $f$’s predictions. Note
that to enable a probabilistic interpretation, $\kappa$ can only be calibrated
if its values reside in $[0,1]$ whereas for ranking and selective prediction
any value in $\mathbb{R}$ can be used.

Figure 2: A comparison of RC-curves made by the best (ViT-L/16-384) and worst
(EfficientNet-V2-XL) models we evaluated in terms of AUROC. Comparing
ViT-B/32-SAM to EfficientNet-V2 exemplifies the fact that neither accuracy nor
AURC reflect selective performance well enough.

A _selective model_ $f$ El-Yaniv and Wiener (2010); Chow (1957) uses a
_selection function_ $g:\mathcal{X}\rightarrow\\{0,1\\}$ to serve as a binary
selector for $f$, enabling it to abstain from giving predictions for certain
inputs. $g$ can be defined by a threshold $\theta$ on the values of a $\kappa$
function such that
$g_{\theta}(x|\kappa,f)=\mathds{1}[\kappa(x,\hat{y}_{f}(x)|f)>\theta]$. The
performance of a selective model is measured using coverage and risk, where
_coverage_ , defined as $\phi(f,g)=E_{P}[g(x)]$, is the probability mass of
the non-rejected instances in $\mathcal{X}$. The _selective risk_ of the
selective model $(f,g)$ is defined as
$R(f,g)\triangleq\frac{E_{P}[\ell(f(x),y)g(x)]}{\phi(f,g)}$. These quantities
can be evaluated empirically over a finite labeled set $S_{m}$, with the
_empirical coverage_ defined as
$\hat{\phi}(f,g|S_{m})=\frac{1}{m}\sum_{i=1}^{m}g(x_{i})$, and the _empirical
selective risk_ defined as
$\hat{r}(f,g|S_{m})\triangleq\frac{\frac{1}{m}\sum_{i=1}^{m}\ell(f(x_{i}),y_{i})g(x_{i})}{\hat{\phi}(f,g|S_{m})}$.
Similarly, SAC is defined as the largest coverage available for a specific
accuracy constraint. A way to visually inspect the behavior of a $\kappa$
function for selective prediction can be done using an RC curve—a curve
showing the selective risk as a function of coverage, measured on some chosen
test set; see Figure 2 for an example.

The AURC and E-AURC metrics were defined by Geifman et al. (2018) for
quantifying the selective quality of $\kappa$ functions via a single number,
with AURC being defined as the area under the RC curve. AURC, however, is very
sensitive to the model’s accuracy, and in an attempt to mitigate this, E-AURC
was suggested. The latter also suffers from sensitivity to accuracy, as we
demonstrate in Appendix C.

Let us consider the two models in Figure 2 for risk-sensitive deployment;
EfficientNet-V2-XL DBLP:journals/corr/abs-2104-00298 and ViT-B/32-SAM
DBLP:journals/corr/abs-2106-01548. While the former model has better overall
accuracy and AURC (metrics that could lead us to believe the model is best for
our needs), it cannot guarantee a Top-1 ImageNet selective accuracy above 95%
for any coverage. ViT-B/32-SAM, on the other hand, can provide accuracies
above 95% for all coverages below 50%.

When there are requirements for specific coverages, the most direct metric to
utilize would be the matching selective risks, by which we can select the
model offering the best performance for our task. If instead a specific range
of coverages is specified, we could measure the area under the RC curve for
those coverages:
$\text{AURC}_{\mathcal{C}}(\kappa,f|S_{m})=\frac{1}{|\mathcal{C}|}\underset{c\in{\mathcal{C}}}{\sum}\hat{r}(f,g_{c}|S_{m})$,
with $\mathcal{C}$ being those required coverages. Lastly, if a certain
accuracy constraint is specified, the chosen model should be the one providing
the largest coverage for that constraint (the largest coverage for a certain
SAC).

Often, these requirements are not known or can change as a result of changing
circumstances or individual needs. Also, using metrics sensitive to accuracy
such as AURC makes designing architectures and methods to improve $\kappa$
very hard, since an improvement in these metrics could be attributed to either
an increase in overall accuracy (if such occurred) or to a real improvement in
the model’s ranking performance. Lastly, some tasks might not allow the model
to abstain from making predictions at all, but instead require interpretable
and well-calibrated probabilities of correctness, which could be measured
using ECE.

###  2.1 Measuring ranking and calibration

A $\kappa$ function is not necessarily able to change the model’s predictions.
Thus, its means for improving the selective risk is by ranking correct and
incorrect predictions better, inducing a more accurate partial order over
instances in $\mathcal{X}$. Thus, for every two random samples
$(x_{1},y_{1}),(x_{2},y_{2})\sim P(\mathcal{X},\mathcal{Y})$ and given that
$\ell(f(x_{1}),y_{1})>\ell(f(x_{2}),y_{2})$, the _ranking_ performance of
$\kappa$ is defined as the probability that $\kappa$ ranks $x_{2}$ higher than
$x_{1}$:

$\mathbf{Pr}[\kappa(x_{1},\hat{y}|f)<\kappa(x_{2},\hat{y}|f)|\ell(f(x_{1}),y_{1})>\ell(f(x_{2}),y_{2})]$
(1)

We discuss this definition in greater detail in Appendix D. The AUROC metric
is often used in the field of machine learning. When the 0/1 loss is in play,
it is known that AUROC in fact equals the probability in Equation (1) Fawcett
(2006) and thus is a proper metric to measure ranking in classification (AKA
discrimination). AUROC is furthermore equivalent to the Goodman and Kruskal’s
_$\gamma$ -correlation_ Goodman and Kruskal (1954), which for decades has been
extensively used to measure ranking (known as “resolution”) in the field of
metacognition Nelson (1984). The precise relationship between
$\gamma$-correlation and AUROC is $\gamma=2\cdot\text{AUROC}-1$ Higham and
Higham (2018). We note also that both the $\gamma$-correlation and AUROC are
nearly identical or closely related to various other correlations and metrics;
$\gamma$-correlation (AUROC) becomes identical to Kendall’s $\tau$ (up to a
linear transformation) in the absence of tied values. Both metrics are also
closely related to _rank-biserial correlation_ , the _Gini coefficient_ (not
to be confused with the measure from economics) and the _Mann–Whitney U test_
, hinting at their importance and usefulness in a variety of fields and
settings. In Appendix E, we briefly compare the ranking performance of neural
networks and humans based on metacognitive research and address a criticism of
using AUROC to measure ranking in Appendix F

The most widely used metric for calibration is ECE Naeini et al. (2015). For a
finite test set of size $N$, ECE is calculated by grouping all instances into
$m$ interval bins (such that $m\ll N$), each of size $\frac{1}{m}$ (the
confidence interval of bin $B_{j}$ is $(\frac{j-1}{m},\frac{j}{m}]$). With
acc($B_{j}$) being the mean accuracy in bin $B_{j}$ and conf($B_{j}$) being
its mean confidence, ECE is defined as

$\begin{split}ECE=\sum_{j=1}^{m}\frac{|B_{j}|}{N}\sum_{i\in
B_{j}}\bigg{|}\frac{\mathds{1}[\hat{y}_{f}(x_{i})=y_{i}]}{|B_{j}|}-\frac{\kappa(x,\hat{y}_{f}(x_{i})|f)}{|B_{j}|}\bigg{|}\\\
=\sum_{j=1}^{m}\frac{|B_{j}|}{N}|\text{acc}(B_{j})-\text{conf}(B_{j})|\end{split}$

Since ECE is widely accepted we use it here to evaluate calibration, and
follow DBLP:journals/corr/GuoPSW17 in setting the number of bins to $m=15$.
Many alternatives to ECE exist, to allow an adaptive binning scheme or to
evaluate the calibration on the non-chosen labels as well
DBLP:journals/corr/abs-1904-01685; DBLP:journals/corr/abs-1902-06977. Relevant
to our objective is that by using binning, this metric is not affected by the
overall accuracy as is the Brier score, for example.

##  3 In-Distribution Analysis

Figure 3: A comparison of different methods and their AUROC improvement
relative to the same model’s performance without employing the method. Markers
above the x axis represent models that benefited from the evaluated method,
and vice versa. Figure 4: A comparison of different methods and their ECE
improvement relative to the same model’s performance without employing the
method. Markers above the x axis represent models that benefited from the
evaluated method, and vice versa. Temperature scaling can sometimes harm ECE,
even though its purpose is to improve it.

While AUROC and ECE are (negatively) correlated (they have a Spearman
correlation of -0.5, meaning that generally as AUROC improves so does ECE),
their agreement on the best performing model depends greatly on the
architectural family in question. For example, the Spearman correlation
between the two metrics evaluated on 28 undistilled XCiTs is 0.76 (meaning ECE
deteriorates as AUROC improves), while for the 33 ResNets he2015deep
evaluated, the correlation is -0.74. Another general observation is that,
contrary to previous work by DBLP:journals/corr/GuoPSW17 concerning ECE, the
correlations between AUROC or ECE and the accuracy or the number of model
parameters are nearly zero, although each family tends to have a strong
correlation, either negative or positive. We include a family-based comparison
in Appendix G for correlations between AUROC/ECE and accuracy, number of
parameters and input size. These results suggest that while some architectures
might utilize extra resources to achieve improved uncertainty estimation
capabilities, other architectures do not and are even harmed in this respect.

We evaluated several training regimes: (1) Training that involves knowledge
distillation in any form, including transformer-specific distillation
DBLP:journals/corr/abs-2012-12877, knapsack pruning with distillation (in
which the teacher is the original unpruned model) Aflalo et al. (2020) and a
pretraining technique which employs distillation
DBLP:journals/corr/abs-2104-10972; (2) adversarial training
DBLP:journals/corr/abs-1911-09665; Tramèr et al. (2018); (3) pretraining on
ImageNet21k (“pure”, with no additions) DBLP:journals/corr/abs-2104-00298;
Touvron et al. (2021a); and (4) various forms of weakly or semi-supervised
learning DBLP:journals/corr/abs-1805-00932; Yalniz et al. (2019);
DBLP:journals/corr/abs-1911-04252. Of these methods, training methods
incorporating distillation improve AUROC and ECE the most (see Figures 3 and
4). Moreover, distillation seems to greatly improve both metrics even when the
teacher itself is much worse at both metrics. We discuss these effects in
greater detail in Appendix H. Note, that although our small sample comparing
pretraining on ImageNet21k to vanilla training suggests pretraining may be
harmful to AUROC, the size of the sample prevents us from making any
definitive conclusions. Based on these early results, we hope that further
research can be conducted into the effects of pretraining on model performance
with more data points.

Evaluations of the simple post-training calibration method of temperature
scaling (TS) DBLP:journals/corr/GuoPSW17, which is widely known to improve ECE
without changing the model’s accuracy, also revealed several interesting
facts: (1) TS consistently and greatly improves AUROC and selective
performance (see Figure 3)—meaning not only does TS calibrate the
probabilistic estimation for each individual instance, but it also improves
the partial order of all instances induced by those improved estimations.
While TS is well known and used for calibration, to the best of our knowledge,
its benefits for selective prediction were previously _unknown_. (2) While TS
is usually beneficial, it could harm some models (see Figures 3 and 4). While
it is surprising that TS–a calibration method–would harm ECE, this phenomenon
is explained by the fact that TS optimizes NLL and not ECE (to avoid trivial
solutions), and the two may sometimes misalign. (3) Models that benefit from
TS in terms of AUROC tend to have been assigned a temperature smaller than 1
by the calibration process. This, however, does not hold true for ECE (see
Figures 15 and 16 in Appendix I). (4) While all models usually improve with
TS, the overall ranking of uncertainty performance between families tends to
stay similar, with the worse (in terms of ECE and AUROC) models closing most
of the gap between them and the mediocre ones.

Figure 5: Comparison of models by their overall accuracy and the coverage they
are able to provide a selective accuracy constraint of Top-1 99% on ImageNet.
A higher coverage is better. Only ViT models are able to provide coverage
beyond 30% for this constraint. They provide more coverage than any other
model compared to their accuracy or size.

The ViT architecture far surpasses any other family of models in terms of
AUROC and ECE (see Figure 1; Figure 14 in Appendix I shows this is true even
after using TS) as well as for the SAC of 99% we explored (see Figure 5 and
Appendix J). Moreover, for any size, ViT models outperform their competition
in all of these metrics (see Figures 9 and 10 in Appendix B and Figure 17 in
Appendix J). While most ViT models we evaluated were pretrained on
ImageNet-21k and enjoyed augmentations tailored to them Steiner et al. (2021),
comparing them with the non-pretrained and not strongly augmented ViT SAMs
DBLP:journals/corr/abs-2106-01548, which are consequently much worse in terms
of accuracy than regular ViTs but yet are good at ranking, confirms that the
high performance in AUROC and SAC is not due to pretraining or augmentation.
It does suggest that it is the architecture itself that is naturally superb at
ranking. This conclusion does not hold for its ECE, which is average compared
to other models, indicating pretraining or augmentations might be responsible
for ViTs’ exceptional calibration.

We also evaluate the AUROC performance of MC-Dropout using predictive entropy
as its confidence score and 30 dropout-enabled forward passes. We do not
measure its affects on ECE since entropy scores do not reside in $[0,1]$.
Using MC-Dropout causes a consistent drop in both AUROC and selective
performance compared with using the same models with softmax as the $\kappa$
(see Appendix K and Figure 3). MC-Dropout’s underperformance was also
previously observed in DBLP:journals/corr/GeifmanE17.

##  4 Uncertainty due to Class-Out-Of-Distribution

Figure 6: OOD performance across 11 severity levels. Note how the detection
performance decreases for all models as we increase the difficulty until it
reaches near chance detection performance at the last severity ($s_{10})$. The
top curve belongs to ViT-L/16-384, which beats all models at every severity
level. We also observe how the previous C-OOD benchmark, ImageNet-O does not
reflect the true OOD performance of the models, since it was designed to
specifically fool ResNet-50, and so it is more difficult for models similar to
ResNet-50 than other models.

When the underlying distribution $P(x,y)$ used to train a model changes, we
may no longer expect that the model will perform correctly. Changes in $P$ can
be the result of many natural or adversarial processes such as natural
deviation in the input space ${\cal X}$, noisy sensor reading of inputs,
abrupt changes due to random events, newly arrived or refined input classes,
etc. We distinguish between input distributional changes in $P_{X|Y}$ and
changes in the label distribution. We focus on the latter case and consider
the _class-out-of-distribution_ (C-OOD) scenario where the label support set
${\cal Y}$ changes to a different set, ${\cal Y}_{OOD}$, which contains new
classes that were not observed in training. We note that both aspects of
distributional deviations have been considered in the literature Hendrycks and
Dietterich (2019); Liang et al. (2017b); Hendrycks et al. (2021), and a number
of methods have been introduced to deal with these cases Liang et al. (2017b);
Lee et al. (2018); Golan and El-Yaniv (2018).

We consider the following _detection_ task, in which our model is required to
distinguish between samples belonging to classes it has seen in training,
where $x\sim P(x|y\in{\cal Y})$, and novel classes, i.e., $x\sim P(x|y\in{\cal
Y}_{OOD})$. We examine the detection performance of DNN classification models
that use their confidence rate function $\kappa$ to detect OOD labels where
the basic premise is that instances whose labels are in ${\cal Y}_{OOD}$
correspond to low $\kappa$ values.

A crucial question in any study of distributional deviations is what we choose
as our experimental data to proxy meaningful deviations. For our study of
C-OOD we introduce a novel method for generating C-OOD data with a
controllable degree of _severity_. Let ${\cal Y}_{OOD}$ be a large set of OOD
classes (e.g., labels from ImageNet-21k), and let $s(y|f,\kappa)$ be a
_severity score_ that reflects the difficulty of model $f$, which uses
$\kappa$ to detect instances from class $y\in{\cal Y}_{OOD}$. Having defined a
function $s(y|f,\kappa)$ (see details below) we can build multiple C-OOD
datasets with progressively increasing severity levels. Importantly, the
resulting C-OOD data is specifically tailored to model $f$ itself (and its
$\kappa$).

Given a model $f$ (and its $\kappa$), we define $s(y|f,\kappa)$ to be the
average confidence given by $\kappa$ to samples from class $y\in{\cal
Y}_{OOD}$. When considering ID instances we expect $\kappa$ to give high
values for highly confident predictions. Therefore, the larger $s(y|f,\kappa)$
is, the harder it is for $\kappa$ to detect the OOD class $y$ among ID
classes. We estimate $s(y|f,\kappa)$ for each class in ImageNet-21K (not in
ImageNet-1K) using a sample from the class and take as C-OOD data a different
sample from that class. Using $s$ we sub-sample 11 groups of classes (severity
levels) from ${\cal Y}_{OOD}$, with increasing severities, such that severity
level $i$ is the $i^{th}$ percentile of all severities. We further expand on
how we chose the severity levels, their statistical meaning and the
construction of the multiple C-OOD datasets for our experiments in Appendix L.

Figure 6 presents the C-OOD detection performance of 484 models across 11
C-OOD severity levels (recall that severity levels are constructed
individually for each model). The box plots clearly show a monotone AUROC
performance degradation. In addition, we see that ViT-L/16-384 is consistently
the best model for each level (recall that this model is also the best for
ID).

Most works on OOD detection use small scale datasets that generally do not
resemble the training distribution and, therefore are, easy to detect. The use
of such sets often causes C-OOD detectors to appear better than they truly are
in harder tasks. Motivated by this deficiency, Hendrycks et al. (2021)
introduced the ImageNet-O dataset as a solution. ImageNet-O, however, has two
limitations. First, it lacks severity levels. Second, the original intent in
the creation of ImageNet-O was to include only hard C-OOD instances.The
definition of “OOD hardness”, however, was carried out with respect to
ResNet-50’s difficulty in detecting OOD classes. This property makes
ImageNet-O strongly biased. Indeed, the right-most box in Figure 6 corresponds
to the performance of the 484 models over ImageNet-O. The torange dot in that
box corresponds to ResNet-50, whose OOD detection performance is severely
harmed by these data. Nevertheless, it is evident that numerous models perform
quite well. In this respect, it can be argued that the proposed C-OOD dataset
generator (see Appendix L) has better properties and is clearly, not biased
toward a single architecture.

Figure 7: OOD detection performance in severity 5 vs. In-Distribution
uncertainty estimation performance. We notice that the best performing network
in one task is not the best in the other, but both belong to the same family,
ViTs

The next question we ask is does ID uncertainty estimation ranking performance
indicate better C-OOD detection performance (and vice versa)? Figure 7 shows a
scatter plot of ID vs. C-OOD AUROC performance of all the tested models. The
overall Spearman correlation is 0.43. The legend indicates correlations
obtained by specific families. For instance, ViTs are among the most
correlated largest sample size family (0.74). This means that a ViT model is
likely to perform well in both ID and C-OOD. Note that the worst performing
models in C-OOD detection are small, optimized models. We further discuss
correlations with other factors such as accuracy, model size, input size and
embedding size in Appendix M.7.

Note that we chose to focus on softmax as the baseline $\kappa$ for our
experiments based on many OOD detection papers Hendrycks and Gimpel (2018);
Berger et al. (2021); Hendrycks et al. (2019); Liang et al. (2017a); Shalev et
al. (2019), since it is a baseline estimator that is widely accepted in the
OOD detection literature. We did experiment with additional $\kappa$
functions, such as entropy (Appendix M.4) and MC-Dropout (Appendix M.5).
Moreover, our novel C-OOD benchmark dataset will hopefully encourage future
research into finding the optimal OOD detection estimator for deep neural
networks.

Due to lack of space, a number of additional interesting observations and
results are presented in Appendix M. We mention the most interesting ones
here: (1) In accordance with the ID results (see Section 3), among all
training regimes, distillation improves performance the most across all
severities; see Figure 20 in Appendix M.3. (2) At the outset it could be
anticipated that ImageNet21k pretraining will hinder C-OOD detection
performance (due to its exposure to the OOD classes in training).
Surprisingly, we observe that pretraining on ImageNet21k somewhat helps
performance at severity levels up to level 6; see Figure 20 in Appendix M.3.
(3) ViTs appear to achieve the best C-OOD detection performance per-model size
(# parameters); see M.1. (4) Using entropy, as $\kappa$, improves C-OOD
detection performance in most cases; see Appendix M.4. (5) Out of all $\kappa$
functions we have evaluated for C-OOD detection, MC-Dropout appears the most
potent; see Appendix M.5. This is especially surprising due to the fact that
MC-Dropout underperformed in ID settings (see Section 3), and that ID ranking
performance and C-OOD detection performance are correlated.

##  5 Concluding Remarks

We presented a comprehensive study of the effectiveness of numerous DNN
architectures (families) in providing reliable uncertainty estimation,
including the impact of various techniques on improving such capabilities.
Moreover, we considered both in-distribution and novel (graded) class-out-of-
distribution settings. Our study led to many discoveries and perhaps the most
important ones are: (1) architectures trained with distillation almost always
improve their uncertainty estimation performance, (2) temperature scaling is
very useful not only for calibration but also for ranking and selective
prediction, and (3) no other DNN (evaluated in this study) had ever
demonstrated an uncertainty estimation performance comparable—in any metric
tested or setting, in-distribution or class-out-of-distribution—to the ViT
architecture.

Our work leaves open many interesting avenues for future research and we would
like to mention a few. Perhaps the most interesting question is why
distillation is so beneficial in boosting uncertainty estimation. Next, what
is the architectural secret in vision transformers (ViT) that enables their
uncertainty estimation supremacy. This question is even more puzzling given
the fact that ViT supremacy is not shared with many other supposedly similar
transformer-based models that we tested such as Touvron et al. (2021b);
DBLP:journals/corr/abs-2012-12877; Liu et al. (2021); Han et al. (2021);
Graham et al. (2021); d’Ascoli et al. (2021); Heo et al. (2021); Xu et al.
(2021); elnouby2021xcit; Zhang et al. (2021); Chu et al. (2021); Chen et al.
(2021). Finally, can we create specialized training regimes (e.g., Geifman and
El-Yaniv (2019)), specialized augmentations, or even specialized neural
architecture search (NAS) strategies that can promote superior uncertainty
estimation performance?

## References

  * Ackerman et al. [2016] Rakefet Ackerman, Avi Parush, Fareda Nassar, and Avraham Shtub.  Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing.  _Computers in Human Behavior_ , 54:101–113, January 2016\.  doi: 10.1016/j.chb.2015.07.041.  URL https://doi.org/10.1016/j.chb.2015.07.041.
  * Ackerman et al. [2019] Rakefet Ackerman, Avigdor Gal, Tomer Sagi, and Roee Shraga.  A cognitive model of human bias in matching.  In _PRICAI 2019: Trends in Artificial Intelligence_ , pages 632–646. Springer International Publishing, 2019.  doi: 10.1007/978-3-030-29908-8_50.  URL https://doi.org/10.1007/978-3-030-29908-8_50.
  * Aflalo et al. [2020] Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik-Manor.  Knapsack pruning with inner distillation.  _CoRR_ , abs/2002.08258, 2020.  URL https://arxiv.org/abs/2002.08258.
  * Basile et al. [2018] Alexandra Basile, Maggie E. Toplak, and Brendan F. Andrade.  Using metacognitive methods to examine emotion recognition in children with ADHD.  _Journal of Attention Disorders_ , 25(2):245–257, November 2018.  doi: 10.1177/1087054718808602.  URL https://doi.org/10.1177/1087054718808602.
  * Berger et al. [2021] Christoph Berger, Magdalini Paschali, Ben Glocker, and Konstantinos Kamnitsas.  Confidence-based out-of-distribution detection: A comparative study and analysis.  _CoRR_ , abs/2107.02568, 2021.  URL https://arxiv.org/abs/2107.02568.
  * Brier [1950] Glenn W. Brier.  Verification of Forecasts Expressed in Terms of Probability.  _Monthly Weather Review_ , 78(1):1, January 1950\.  doi: 10.1175/1520-0493(1950)0780001:VOFEIT2.0.CO;2.
  * Chen et al. [2021] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian.  Visformer: The vision-friendly transformer.  _CoRR_ , abs/2104.12533, 2021.  URL https://arxiv.org/abs/2104.12533.
  * Chow [1957] C. K. Chow.  An optimum character recognition system using decision functions.  _IRE Transactions on Electronic Computers_ , EC-6(4):247–254, 1957.  doi: 10.1109/TEC.1957.5222035.
  * Chu et al. [2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers, 2021.
  * Cordella et al. [1995] L. P. Cordella, C. De Stefano, F. Tortorella, and M. Vento.  A method for improving classification reliability of multilayer perceptrons.  _IEEE Transactions on Neural Networks_ , 6(5):1140–1147, 1995.  doi: 10.1109/72.410358.
  * d’Ascoli et al. [2021] Stéphane d’Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  _CoRR_ , abs/2103.10697, 2021.  URL https://arxiv.org/abs/2103.10697.
  * De Stefano et al. [2000] C. De Stefano, C. Sansone, and M. Vento.  To reject or not to reject: that is the question-an answer in case of neural classifiers.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 30(1):84–94, 2000.  doi: 10.1109/5326.827457.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Ding et al. [2019] Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi.  Evaluation of neural network uncertainty estimation with application to resource-constrained platforms.  _CoRR_ , abs/1903.02050, 2019.  URL http://arxiv.org/abs/1903.02050.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929, 2020.  URL https://arxiv.org/abs/2010.11929.
  * El-Yaniv and Wiener [2010] Ran El-Yaniv and Yair Wiener.  On the foundations of noise-free selective classification.  _Journal of Machine Learning Research_ , 11(5), 2010.
  * Fawcett [2006] Tom Fawcett.  An introduction to roc analysis.  _Pattern Recognition Letters_ , 27(8):861–874, 2006.  ISSN 0167-8655.  doi: https://doi.org/10.1016/j.patrec.2005.10.010.  URL https://www.sciencedirect.com/science/article/pii/S016786550500303X.  ROC Analysis in Pattern Recognition.
  * Fiedler et al. [2019] K. Fiedler, Rakefet Ackerman, and Chiara Scarampi.  ! metacognition : Monitoring and controlling one ’ s own knowledge , reasoning and decisions.  2019\.
  * Frankle and Carbin [2018] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Training pruned neural networks.  _CoRR_ , abs/1803.03635, 2018.  URL http://arxiv.org/abs/1803.03635.
  * Galil et al. [2023a] Ido Galil, Mohammed Dabbah, and Ran El-Yaniv.  A framework for benchmarking class-out-of-distribution detection and its application to imagenet.  In _International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=Iuubb9W6Jtk.
  * Galil et al. [2023b] Ido Galil, Mohammed Dabbah, and Ran El-Yaniv.  What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers?  In _International Conference on Learning Representations_ , 2023b.  URL https://openreview.net/forum?id=p66AzKi6Xim.
  * Gao et al. [2021] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr.  Res2net: A new multi-scale backbone architecture.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(2):652–662, Feb 2021.  ISSN 1939-3539.  doi: 10.1109/tpami.2019.2938758.  URL http://dx.doi.org/10.1109/TPAMI.2019.2938758.
  * Geifman and El-Yaniv [2019] Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  _CoRR_ , abs/1901.09192, 2019.  URL http://arxiv.org/abs/1901.09192.
  * Geifman et al. [2018] Yonatan Geifman, Guy Uziel, and Ran El-Yaniv.  Bias-reduced uncertainty estimation for deep neural classifiers.  In _International Conference on Learning Representations_ , 2018.
  * Golan and El-Yaniv [2018] Izhak Golan and Ran El-Yaniv.  Deep anomaly detection using geometric transformations, 2018.
  * Goodman and Kruskal [1954] Leo A. Goodman and William H. Kruskal.  Measures of association for cross classifications.  _Journal of the American Statistical Association_ , 49(268):732–764, December 1954.  doi: 10.1080/01621459.1954.10501231.  URL https://doi.org/10.1080/01621459.1954.10501231.
  * Graham et al. [2021] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  _CoRR_ , abs/2104.01136, 2021.  URL https://arxiv.org/abs/2104.01136.
  * Griffin et al. [2019] Thomas D. Griffin, Jennifer Wiley, and Keith W. Thiede.  The effects of comprehension-test expectancies on metacomprehension accuracy.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , 45(6):1066–1092, June 2019.  doi: 10.1037/xlm0000634.  URL https://doi.org/10.1037/xlm0000634.
  * Han et al. [2021] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  _CoRR_ , abs/2103.00112, 2021.  URL https://arxiv.org/abs/2103.00112.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations, 2019.
  * Hendrycks and Gimpel [2018] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018.
  * Hendrycks et al. [2019] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _CoRR_ , abs/1907.07174, 2019.  URL http://arxiv.org/abs/1907.07174.
  * Hendrycks et al. [2021] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271, 2021.
  * Heo et al. [2021] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  _CoRR_ , abs/2103.16302, 2021.  URL https://arxiv.org/abs/2103.16302.
  * Higham and Higham [2018] Philip A. Higham and D. Paul Higham.  New improved gamma: Enhancing the accuracy of goodman–kruskal’s gamma using ROC curves.  _Behavior Research Methods_ , 51(1):108–125, September 2018.  doi: 10.3758/s13428-018-1125-5.  URL https://doi.org/10.3758/s13428-018-1125-5.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network, 2015.
  * Howard et al. [2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3.  _CoRR_ , abs/1905.02244, 2019.  URL http://arxiv.org/abs/1905.02244.
  * Lee et al. [2018] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks, 2018.
  * Liang et al. [2017a] Shiyu Liang, Yixuan Li, and R. Srikant.  Principled detection of out-of-distribution examples in neural networks.  _CoRR_ , abs/1706.02690, 2017a.  URL http://arxiv.org/abs/1706.02690.
  * Liang et al. [2017b] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv preprint arXiv:1706.02690_ , 2017b.
  * Lin et al. [2020] Ming Lin, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin.  Neural architecture design for gpu-efficient networks, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  _CoRR_ , abs/2103.14030, 2021.  URL https://arxiv.org/abs/2103.14030.
  * Mahajan et al. [2018] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining, 2018.
  * Nado et al. [2021] Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Farquhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran.  Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning.  _CoRR_ , abs/2106.04015, 2021.  URL https://arxiv.org/abs/2106.04015.
  * Naeini et al. [2015] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_ , AAAI’15, page 2901–2907. AAAI Press, 2015.  ISBN 0262511290.
  * Nelson [1984] Thomas O. Nelson.  A comparison of current measures of the accuracy of feeling-of-knowing predictions.  _Psychological Bulletin_ , 95(1):109–133, 1984\.  doi: 10.1037/0033-2909.95.1.109.  URL https://doi.org/10.1037/0033-2909.95.1.109.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Shalev et al. [2019] Gabi Shalev, Yossi Adi, and Joseph Keshet.  Out-of-distribution detection using multiple semantic label representations, 2019.
  * Steiner et al. [2021] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  _CoRR_ , abs/2106.10270, 2021.  URL https://arxiv.org/abs/2106.10270.
  * Touvron et al. [2021a] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou.  Resmlp: Feedforward networks for image classification with data-efficient training.  _CoRR_ , abs/2105.03404, 2021a.  URL https://arxiv.org/abs/2105.03404.
  * Touvron et al. [2021b] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  _CoRR_ , abs/2103.17239, 2021b.  URL https://arxiv.org/abs/2103.17239.
  * Tramèr et al. [2018] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkZvSe-RZ.
  * Undorf and Bröder [2019] Monika Undorf and Arndt Bröder.  Cue integration in metamemory judgements is strategic.  _Quarterly Journal of Experimental Psychology_ , 73(4):629–642, October 2019.  doi: 10.1177/1747021819882308.  URL https://doi.org/10.1177/1747021819882308.
  * Wightman [2019] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le.  Self-training with noisy student improves imagenet classification, 2020\.
  * Xu et al. [2021] Weijian Xu, Yifan Xu, Tyler A. Chang, and Zhuowen Tu.  Co-scale conv-attentional image transformers.  _CoRR_ , abs/2104.06399, 2021.  URL https://arxiv.org/abs/2104.06399.
  * Yalniz et al. [2019] I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan.  Billion-scale semi-supervised learning for image classification, 2019\.
  * Zhang et al. [2020] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, and Alexander Smola.  Resnest: Split-attention networks, 2020.
  * Zhang et al. [2021] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister.  Aggregating nested transformers.  _CoRR_ , abs/2105.12723, 2021.  URL https://arxiv.org/abs/2105.12723.

##  Appendix A The investment example

Let us consider two classification models for the stock market that predict
whether a stock’s value is about to increase, decrease or remains neutral
(three-class classification). Suppose that Model A has a 95% true accuracy,
and generates a confidence score of 0.95 on any prediction (even on
missclassified instances); Model B has a 40% true accuracy, but always gives a
confidence score of 0.6 on correct predictions, and 0.4 on incorrect ones. We
now try and evaluate these two models with the uncertainty metrics mentioned
in Section 1 to see which can reveal Model B’s superior uncertainty estimation
performance. AURC will fail due to its sensitivity to accuracy (the AURC of
Model B is 0.12, more than twice as bad as the AURC for Model A, which is
0.05). NLL will rank Model A four times higher (Model A’s NLL is 0.23 and
Model B’s is 0.93). The Brier score would also much prefer Model A (giving it
a score of 0.096 while giving Model B a score of 0.54). Evaluating the models’
calibration with ECE will also not reveal Model B’s advantages, since it is
less calibrated than Model A, which has perfect calibration (Model A has an
ECE of 0, and Model B has a worse ECE of 0.4).

AUROC, on the other hand, would give Model B a perfect score of 1 and a
terrible score of 0.5 to Model A. The selective risk for Model B would be
better for any _coverage_ of stock predictions below 40%, and for any SAC
above 95% the coverage for Model A would be 0, but 0.4 for Model B.

Those two metrics are not perfect for any example. If instead we were to
compare two different models for the task of predicting the weather, such that
we cannot abstain from making predictions but are required to provide an
accurate probabilistic uncertainty estimation of the model’s predictions,
AUROC and selective risk would be meaningless (due to the model’s inability to
abstain in this task), but ECE or the Brier Score would better evaluate the
performance the new task requires.

##  Appendix B Ranking and Calibration Visual Comparison

A comparison of 484 models by their AUROC ($\times 100$, higher is better) and
-log(ECE) (higher is better) on ImageNet is visualized in Figure 8. An
interactive version of this Figure is provided as supplementary material. To
compare models fairly by their size, we plot two graphs with the logarithm of
the number of parameters as the X axis, so that models sharing the same x
value can be compared solely based on their y value. In Figure 9 we set the X
axis to be AUROC (higher is better), and see ViTs outperform any other
architecture with a comparable amount of parameters by a large margin. We can
also observe using distillation creates a consistent improvement in AUROC. In
Figure 10 we set the X axis to be the negative logarithm of ECE (higher is
better) and observe a very similar trend, with ViT outperforming its
competition for any model size.

Figure 8: A comparison of 484 models by their AUROC ($\times 100$, higher is
better) and log(ECE) (lower is better) on ImageNet. Each marker’s size is
determined by the model’s number of parameters. Each dotted marker represents
a distilled version of the original. An interactive version of this Figure is
provided as supplementary material. Figure 9: A comparison of 484 models by
their AUROC ($\times 100$, higher is better) and log(number of model’s
parameters) on ImageNet. Each dotted marker represents a distilled version of
the original. Figure 10: A comparison of 484 models by their -log(ECE) (higher
is better) and log(number of model’s parameters) on ImageNet. Each dotted
marker represents a distilled version of the original.

##  Appendix C Demonstration of E-AURC’s dependence on the model’s accuracy

_Excess-AURC_ (E-AURC) was suggested by Geifman et al. [2018] as an
alternative to AURC (explained in Section 2). To calculate E-AURC, two AURC
scores need to be calculated: (1) $AURC(model)$, the AURC value of the actual
model and (2) $AURC(model^{*})$, the AURC value of a hypothetical model with
identical predicted labels as the first model, but that outputs confidence
values that induce a perfect partial order on the instances in terms of their
correctness. The latter means that all incorrectly predicted instances are
assigned confidence values lower than the correctly predicted instances.

E-AURC is then defined as $AURC(model)-AURC(model^{*})$. In essence, this
metrics acknowledges that given a model’s accuracy, the area of
$AURC(model^{*})$ is always unavoidable no matter how good the partial order
is, but anything above that could have been minimized if the $\kappa$ function
was better at ranking, assigning correct instances higher values than
incorrect ones and inducing a better partial order over the instances.

This metric indeed helps to reduce some of the sensitivity to accuracy
suffered by AURC, and for the example presented in Section 1, E-AURC would
have given a perfect score of 0 to the model inducing a perfect partial order
by its confidence values (Model B). It is easy, however, to craft examples
showing that E-AURC prefers models with higher accuracy, even if they have
lower or equal capacity to rank.

To demonstrate this in a simple way, let us consider two models with a
complete lack of capacity to rank correct and incorrect predictions correctly,
always outputting the same confidence score. Model A has an accuracy of 20%
(thus an error rate of 80%), and Model B has an accuracy of 80% (and an error
rate of 20%). A good ranking metric should evaluate them equally (the same way
E-AURC gives the same score for two models that rank perfectly regardless of
their accuracy). In Figure 11 we plot their RC curves with dashed lines, which
are both straight lines due to their lack of ranking ability.

Figure 11: The RC curves for Models A and B are plotted with dashed lines. The
RC curves for the hypothetically optimal versions of Models A and B are
plotted with solid lines.

We can calculate both of these models AURCs,
$AURC(modelA)=0.8,AURC(modelB)=0.2$.

The next thing to calculate is the best AURC values those models could have
achieved given the same accuracy if they had a perfect partial order. We plot
these hypothetical models’ RC curves in Figure 11 as solid lines. Their
selective risk remains 0 for every coverage below their total accuracy, since
these hypothetical models assigned the highest confidence to all of their
correct instances first. As the coverage increases and they have no more
correct instances to select, they begin to give instances that are incorrect,
and thus their selective risk deteriorates for higher coverages. Calculating
both of these hypothetical models’ AURCs gives us the following:
$AURC(modelA^{*})=0.482,AURC(modelB^{*})=0.022$. Subtracting our results we
get:
$\text{E-AURC}(modelA)=0.8-0.482=0.318,\text{E-AURC}(modelB)=0.2-0.022=0.178$
Hence, E-AURC prefers Model B over Model A, even though both do not
discriminate at all between incorrect and correct instances.

##  Appendix D More on the Definition of Ranking

Let us consider a finite set $S_{m}=\\{(x_{i},y_{i})\\}_{i=1}^{m}\sim
P_{X,Y}$. We assume that there are no two identical values given by $\kappa$
on $S_{m}$. Such an assumption is reasonable when choosing a continuous
confidence signal.

We further denote $c$ as the number of concordant pairs (i.e., pairs in
$S_{m}$ that satisfy the condition
$[\kappa(x_{i},\hat{y}|f)<\kappa(x_{j},\hat{y}|f)\cap\ell(f(x_{i}),y_{i})>\ell(f(x_{j}),y_{j})]$)
and $d$ as the number of discordant pairs (i.e., pairs in $S_{m}$ that satisfy
the condition
$[\kappa(x_{i},\hat{y}|f)>\kappa(x_{j},\hat{y}|f)\cap\ell(f(x_{i}),y_{i})>\ell(f(x_{j}),y_{j})]$

We assume, for now, that there are no two identical values given by $\ell$ on
$S_{m}$. Accordingly, we can further develop Equation (1) from Section 2.1
using the definition of conditional probability,

$\begin{split}\mathbf{Pr}[\kappa(x_{i},\hat{y}|f)<\kappa(x_{j},\hat{y}|f)|\ell(f(x_{i}),y_{i})>\ell(f(x_{j}),y_{j})]=\\\
\frac{\mathbf{Pr}[\kappa(x_{i},\hat{y}|f)<\kappa(x_{j},\hat{y}|f)\cap\ell(f(x_{i}),y_{i})>\ell(f(x_{j}),y_{j})]}{\mathbf{Pr}[\ell(f(x_{i}),y_{i})>\ell(f(x_{j}),y_{j})]},\end{split}$

which can be approximated empirically, using the most likelihood estimator, as

$\begin{split}\frac{\text{c}}{{m\choose 2}}.\end{split}$ (2)

We notice that the last equation is identical to Kendall’s $\tau$ up to a
linear transformation, which equals

$\begin{split}\frac{c-d}{{m\choose 2}}=\frac{c-d+c-c}{{m\choose 2}}\\\
=\frac{2c-(c+d)}{{m\choose 2}}=\frac{2c}{{m\choose 2}}-\frac{c+d}{{m\choose
2}}=\\\ 2\cdot\frac{c}{{m\choose 2}}-1=2\cdot[\text{Equation
\ref{eq:discrimination_development_MLE}}]-1.\end{split}$

Otherwise, if the loss assigns two identical values to a pair of points in
$S_{m}$, but $\kappa$ does not, then we get:

$\begin{split}\frac{c}{c+d}.\end{split}$ (3)

which is identical to Goodman & Kruskal’s $\gamma$-correlation up to a linear
transformation

$\begin{split}\frac{c-d}{c+d}=\frac{c-d+c-c}{c+d}=\frac{2c-(c+d)}{c+d}=\\\
\frac{2c}{c+d}-\frac{c+d}{c+d}=2\cdot[\text{Equation
\ref{eq:discrimination_development}}]-1.\end{split}$

###  D.1 Inequalities of the definition

One might wonder why Equation (1) should have strict inequalities rather than
non-strict ones to define ranking. As we discuss below, this would damage the
definition:

(1) If the losses had a non-strict inequality:

$\begin{split}\mathbf{Pr}[\kappa(x_{1},\hat{y}|f)<\kappa(x_{2},\hat{y}|f)|\ell(f(x_{1}),y_{1})\geq\ell(f(x_{2}),y_{2})]\end{split}$

Consequently, in the case of classification, for example, this probability
would increase for any pairs consisting of correct instances with different
confidences, which yields no benefit in ranking between incorrect and correct
instances and motivates giving different confidence values for instances with
the same loss—a fact that would not truly add any value.

(2) If the $\kappa$ values had a non-strict inequality:

$\begin{split}\mathbf{Pr}[\kappa(x_{1},\hat{y}|f)\leq\kappa(x_{2},\hat{y}|f)|\ell(f(x_{1}),y_{1})>\ell(f(x_{2}),y_{2})].\end{split}$

This probability would increase for any pair $(x_{1},x_{2})$ such that
$\kappa(x_{1},\hat{y}|f)=\kappa(x_{2},\hat{y}|f)$ and
$\ell(f(x_{1}))>\ell(f(x_{2}))$, although $\kappa$ should have ranked $x_{1}$
with a lower value. Furthermore, if a $\kappa$ function were to assign the
same confidence score to all $x\in\mathcal{X}$, then when there are no two
identical values of losses, the definition’s probability would be 1;
otherwise, the more different values for losses there are, the larger it would
grow. In classification with a 0/1 loss, for example, assigning the same
confidence score to all instances would result in the probability being
$Accuracy(f)\cdot(1-Accuracy(f))$, which is largest when $Accuracy(f)=0.5$.

##  Appendix E Ranking capacity comparison between humans and Neural Networks

In the field of metacognition, interestingly, the predictive value of
confidence is evaluated by two different aspects: by its ability to
_discriminate_ between correct and incorrect predictions (also known as
_resolution_ in metacognition or ranking in our context) and by its ability to
give well calibrated confidence estimations, not being over- or underconfident
Fiedler et al. [2019]. These two aspects correspond perfectly with much of the
research done in the deep learning field, with the nearly matching metric to
AUROC of $\gamma$-correlation (see Section 2).

This allows us to compare how well humans rank predictions in various tasks
and how models rank their own in others. Human AUROC measurements in various
tasks (translated from $\gamma$-correlation) tends to range from 0.6 to 0.75
Undorf and Bröder [2019], Basile et al. [2018], Ackerman et al. [2016], but
could vary, usually towards much lower values Griffin et al. [2019]. In our
comprehensive evaluation on ImageNet, AUROC ranged from 0.77 to 0.88 (with the
median value being 0.85), and in CIFAR-10 these measurements jump to the range
of 0.92 to 0.94.

While such comparisons between neural networks and humans are somewhat unfair
due to the great sensitivity required for the task, research that directly
compares humans and machine learning algorithms on the same task exist. For
example, in Ackerman et al. [2019], algorithms far surpass even the group of
highest performing individuals in terms of ranking.

##  Appendix F Criticisms of AUROC as a ranking metric

In this section we show why AUROC does not simply reward models for having
lower accuracy, addressing such criticism. The paper by Ding et al. [2019]
presented a semi-artificial experiment to demonstrate that AUROC might get
larger the worse the model’s accuracy becomes. They consider a model $f$ and
its $\kappa$ function evaluated on a classification test set $\mathcal{X}$,
giving each a prediction $\hat{y}_{f}(x)$ and a confidence score
$\kappa(x,\hat{y}_{f}(x)|f)$, which in this case is the model’s softmax
response. Let
$\mathcal{X}^{c}=\\{x^{c}\in\mathcal{X}|\hat{y}_{f}(x^{c})=y(x)\\}$ be the set
of all instances correctly predicted by the model $f$, and define
$x_{(i)}^{c}\in\mathcal{X}^{c}$ to be the correct instance that received the
i-lowest confidence score from $\kappa$. Their example continues to consider
an artificial model $f^{m}$ to be an exact clone of $f$ with the following
modification: for every $i\leq m$, the model $f^{m}$ now predicts a different,
incorrect label for $x_{(i)}^{c}$; however, its given confidence score remains
identical:
$\kappa(x_{(i)}^{c},\hat{y}_{f}(x_{(i)}^{c})|f)=\kappa(x_{(i)}^{c},\hat{y}_{f^{m}}(x_{(i)}^{c})|f^{m})$.
$f^{0}$ is exactly identical to $f$, by this definition, not changing any
predictions. The paper shows how an artificially created model $f^{m}$ obtains
a higher AUROC score the bigger its $m$. This happens even though “nothing”
changed but a hit to the model’s accuracy performance (by making mistakes on
more instances).

First, to understand why this happens, let us consider $f^{1}$: AUROC for
$\kappa$ increases the more pairs of
[$\kappa(x_{1})<\kappa(x_{2})|\hat{y}_{f}(x_{1})\neq
y(x_{1}),\hat{y}_{f}(x_{2})=y(x_{2})$] there are. The model $f^{1}$ is now
giving an incorrect classification to $x_{(1)}^{c}$, but this instance’s
position in the partial order induced by $\kappa$ has remained the same (since
$\kappa(x_{(1)}^{c})$ is unchanged); therefore, $|\mathcal{X}^{c}|-1$
correctly ranked pairs were added:
[$\kappa(x_{(1)}^{c})<\kappa(x_{(i)}^{c})|\hat{y}_{f}(x_{(1)}^{c})\neq
y(x_{(1)}^{c}),\hat{y}_{f}(x_{(i)}^{c})=y(x_{(i)}^{c})$] for every
$1<i\leq|\mathcal{X}^{c}|$. Nevertheless, this does not guarantee an increase
to AUROC by itself: if, previously, all pairs of (correct,incorrect) instances
were ranked correctly by $\kappa$, AUROC would already be 1.0 for $f^{0}$ and
would not change for $f^{1}$. If AUROC for $f^{1}$ is higher than it was for
$f^{0}$, this means there exists at least one instance $x^{w}$ that was
incorrectly predicted by the original model $f^{0}$ such that
$\kappa(x_{(1)}^{c})<\kappa(x^{w})$. Every such _originally_ wrongly ranked
pair (by $f^{0}$) of
[$\kappa(x_{(1)}^{c})<\kappa(x^{w})|\hat{y}_{f}(x^{w})\neq
y(x^{w}),\hat{y}_{f}(x_{(1)}^{c})=y(x_{(1)}^{c})$] has been eliminated by
$f^{1}$ wrongly predicting $x_{(1)}^{c}$. This, therefore, causes AUROC to
increase at the expense of the model’s accuracy.

Such an analysis neglects many factors, which is probably why such an effect
is only likely to be observed in artificial models (and not among the actual
models we have empirically tested):

  1. 1.

It is unreasonable to assume that the confidence score given by $\kappa$ will
remain exactly the same for an instance $x_{(i)}^{c}$ given it now has a
different prediction. In the case of $\kappa$ being softmax, it assumes the
model’s logits have changed in a very precise and nontrivial manner.
Additionally, by our broad definition of $\kappa$, which allows $\kappa$ to
even be produced from an entirely different model than $f$, $\kappa$ receives
the prediction and model as a given input (and cannot change or affect
neither), and it is unlikely to assume changing its inputs will not change its
output.

  2. 2.

Suppose we find the setting reasonable and assume we can actually create a
model $f^{m}$ as described. Let us observe a model $f^{p}$ such that
$p=\min_{m}(\text{AUROC of $f^{m}$=1})$, meaning that $f^{p}$ ranks its
predictions perfectly, unlike the original $f^{0}$. Is it really true that
$f^{p}$ has no better uncertainty estimation than $f^{0}$? Model $f^{p}$
behaves very much like the investment “Model B” from our example in Section 1,
possessing perfect knowledge of when it is wrong and when it is correct,
allowing its users risk-free classification. So, given a model $f$, we can use
the above process to produce an improved model $f^{p}$, and then we can even
calibrate its $\kappa$ to output 0% for all instances below its threshold and
100% for all those above to produce a perfect model, which might have a small
coverage but is correct every time, knows it and notifies its user when it
truly knows the prediction. The increase in AUROC reflects such an
improvement.

Not only do we disagree with such an analysis and its conclusions, but we also
have vast empirical evidence to show that AUROC does not prefer lower accuracy
models unless there is a good reason for it to do so, as we demonstrate in
Figure 2 (comparing EfficientNet-V2-XL to ViT-B/32-SAM). In fact, out of the
484 models we tested, the model with the highest AUROC has also the $4^{th}$
highest accuracy of all models, and the overall Spearman correlation between
AUROC and accuracy of all the models we tested is 0.03. Furthermore, Figure 2
also exemplifies why AURC, which was suggested by the mentioned paper as the
alternative to AUROC, is a bad choice as a single number metric, and might
lead us to deploy a model that has a worse selective risk for most coverages
only due to its higher overall accuracy.

##  Appendix G Effects of the model’s accuracy, number of parameters and input
size on in-distribution uncertainty estimation performance

Table 1 shows the relationship between uncertainty estimation performance and
model’s attributes and resources (accuracy, number of parameters and input
size), measured by Spearman correlation. We measure uncertainty estimation
performance by AUROC (higher is better) and -ECE (higher is better). Positive
correlations indiciate good utilization of resources for uncertainty
estimation (for example, a positive correlation between -ECE and the number of
parameters indicates that as the number of parameters increases, the
calibration improves). An interesting observation is that distillation can
drastically change the correlation between a resource and the uncertainty
estimation performance metrics. For example, undistilled XCiTs have a Spearman
correlation of -0.79 between their number of parameters and AUROC, indicating
that more parameters are correlated with lower ranking performance, while
distilled XCiTs have a correlation of 0.35 between the two.

Table 1: The relationship between uncertainty estimation performance and the
model’s attributes and resources (accuracy, number of parameters and input
size), measured by Spearman correlation. Positive correlations indicate good
utilization of resources for uncertainty estimation.

Architecture | AUROC & Accuracy | -ECE & Accuracy | AUROC & #Parameters | -ECE & #Parameters | AUROC & Input Size | -ECE & Input Size | # Models Evaluated
---|---|---|---|---|---|---|---
EfficientNet | -0.16 | -0.29 | -0.22 | -0.29 | -0.26 | -0.38 | 50
ResNet | -0.28 | -0.22 | 0.16 | 0.03 | -0.40 | -0.44 | 33
XCiT distilled | 0.60 | 0.09 | 0.35 | 0.02 | 0.51 | 0.12 | 28
XCiT | -0.68 | 0.89 | -0.79 | 0.94 | - | - | 28
ViT | 0.95 | -0.62 | 0.71 | -0.78 | 0.22 | -0.27 | 20
SE_ResNet | -0.46 | -0.02 | -0.53 | 0.20 | -0.02 | -0.35 | 18
EfficientNetV2 | -0.70 | -0.45 | -0.63 | -0.47 | -0.59 | -0.40 | 15
NFNet | 0.56 | 0.78 | 0.63 | 0.81 | 0.48 | 0.60 | 13
Inception | -0.29 | 0.09 | -0.43 | 0.30 | -0.08 | 0.23 | 13
RegNetY | -0.03 | -0.98 | 0.27 | -0.86 | - | - | 12
RegNetX | 0.20 | -0.96 | 0.20 | -0.96 | - | - | 12
CaiT distilled | 0.44 | -0.87 | 0.35 | -0.87 | 0.58 | -0.50 | 10
DLA | 0.64 | -0.90 | 0.77 | -0.90 | - | - | 10
MobileNetV3 | 0.37 | 0.59 | 0.42 | 0.60 | - | - | 10
Res2Net | -0.70 | 0.27 | -0.68 | 0.60 | - | - | 9
VGG | 0.81 | -0.98 | 0.71 | -0.90 | - | - | 8
RepVGG | -0.71 | 0.50 | -0.57 | 0.21 | - | - | 8
BiT | -0.33 | -0.81 | -0.20 | -0.85 | -0.46 | -0.25 | 8
ResNeXt | -0.96 | 0.39 | -0.22 | -0.30 | - | - | 7
ResNet RS | 0.00 | 0.79 | -0.18 | 0.82 | -0.30 | 0.82 | 7
MixConv | -0.11 | 0.89 | -0.24 | 0.86 | - | - | 7
DenseNet | 0.43 | -0.14 | 0.72 | 0.12 | - | - | 6
HardCoReNAS | -0.60 | 0.26 | -0.49 | 0.37 | - | - | 6
Swin | 0.71 | 0.14 | 0.77 | 0.26 | 0.41 | 0.00 | 6
ECANet | -0.20 | 0.60 | -0.43 | 0.37 | 0.83 | 0.37 | 6
Twins | -0.26 | 0.94 | -0.14 | 0.89 | - | - | 6
SWSL ResNet | 0.94 | -0.89 | 0.77 | -0.83 | - | - | 6
GENet | 0.50 | -1.00 | 0.50 | -1.00 | 0.87 | -0.87 | 6
SSL ResNet | 0.14 | -1.00 | 0.26 | -0.94 | - | - | 6
TResNet | 0.10 | -0.30 | 0.53 | 0.53 | -0.58 | -0.87 | 5
CoaT | -0.10 | 0.90 | -0.10 | 0.50 | - | - | 5
LeViT distilled | 0.60 | -0.90 | 0.60 | -0.90 | - | - | 5
ResMLP | 0.20 | 1.00 | 0.15 | 0.97 | - | - | 5
MobileNetV2 | -0.30 | 0.00 | -0.21 | 0.10 | - | - | 5
PiT distilled | 1.00 | -1.00 | 1.00 | -1.00 | - | - | 4
PiT | -0.40 | 1.00 | -0.40 | 1.00 | - | - | 4
WSP ResNeXt | 1.00 | 0.80 | 1.00 | 0.80 | - | - | 4
ResMLP distilled | 0.80 | 0.20 | 0.80 | 0.20 | - | - | 4
MnasNet | 0.40 | 0.20 | 0.63 | 0.95 | - | - | 4
DeiT distilled | 0.80 | -1.00 | 0.80 | -1.00 | 0.77 | -0.77 | 4
DeiT | 0.40 | 0.80 | 0.40 | 0.80 | 0.26 | 0.26 | 4

##  Appendix H Knowledge distillation effects on in-distribution uncertainty
estimation

Figure 12: Comparing vanilla models to those incorporating KD into their
training (represented by markers with thick borders and a dot). In a pruning
scenario that included distillation, yellow markers indicate that the original
model was also the teacher. The performance of each model is measured in AUROC
(higher is better) and -log(ECE) (higher is better).

Figure 12 compares vanilla models to those incorporating KD into their
training (represented by markers with thick borders and a dot). In a pruning
scenario that included distillation, yellow markers indicate that the original
model was also the teacher Aflalo et al. [2020]. While distillation using a
different model tends to improve uncertainty estimation in both aspects,
distillation by the model itself seems to improve only one—suggesting it is
generally more beneficial to use a different model as a teacher. The fact that
KD improves the model over its original form, however, is surprising, and
suggests the distillation process itself helps uncertainty estimation. Note
that although this specific method involves pruning, evaluations of models
pruned without incorporating distillation Frankle and Carbin [2018] revealed
no improvement.

Moreover, it seems that the teacher does not have to be good in uncertainty
estimation itself; Figure 13 shows this by comparing the teacher architecture
and the students in each case.

Figure 13: Comparing teacher models (yellow markers) to their KD students
(represented by markers with thick borders and a dot). The performance of each
model is measured in AUROC (higher is better) and -log(ECE) (higher is
better).

While the training method by DBLP:journals/corr/abs-2104-10972 included
pretraining on ImageNet-21k and demonstrated impressive improvements,
comparison of models that were pretrained on ImageNet21k
DBLP:journals/corr/abs-2104-00298, Touvron et al. [2021a] with identical
models that were not pretrained showed no clear improvement in ECE, and, in
fact, exhibit a degradation of AUROC (see Figures 3 and 4 in Section 3). This
suggests that pretraining alone does not improve uncertainty estimation.

##  Appendix I More information about Temperature Scaling

Figure 14: A comparison of 484 models after being calibrated with TS,
evaluated by their AUROC ($\times 100$, higher is better) and -log(ECE)
(higher is better) on ImageNet. Each marker’s size is determined by the
model’s number of parameters. ViT models are still among the best performing
architectures for all aspects of uncertainty estimation.

In Figure 14 we see how temperature scaling (TS) affects the overall ranking
of models in terms of AUROC and ECE. While the ranking between the different
architecture remains similar, the poorly performing models are much improved
and minimize the gap between them and the best models. One particularly
notable exception is HardCoRe-NAS DBLP:journals/corr/abs-2102-11646, with its
lowest latency versions becoming the top performers in terms of ECE.

Figure 15: Out of 484 models evaluated, models that were assigned a
temperature higher than 1 by the calibration process tended to degrade in
AUROC performance rather than improve. Markers above the x axis represent
models that benefited from TS, and vice versa. Figure 16: The relationship
between temperature and the success of TS, unlike the case for AUROC, seems
unrelated.

In addition, models that benefit from TS in terms of AUROC tend to have been
assigned a temperature lower than 1 by the calibration process (see Figure
15). The same, however, does not hold true for ECE (see Figure 16). This
example also emphasizes the fact that models benefitting from TS in terms of
AUROC do not necessarily benefit in terms of ECE, and vice versa. Therefore,
determining whether to calibrate the deployed model with TS is, unfortunately,
a task-specific decision.

We conduct TS as was suggested in DBLP:journals/corr/GuoPSW17. For each model
we take a random stratified sampling of 5,000 instances from the ImageNet
validation set to calibrate on, and reserve the remainder 45,000 instances for
testing. Using the box-constrained L-BFGS (Limited-Memory Broyden-Fletcher-
Goldfarb-Shanno) algorithm, we optimize for 5,000 iterations (though fewer
iterations usually converge into the same temperature parameter) using a
learning rate of 0.01.

##  Appendix J Architecture choice for practical deployment based on Selective
Performance

As discussed in Section 2, when we know the coverage or risk we require for
deployment, the most direct metric to check is which model obtains the best
risk for the coverage required (selective risk), or which model gets the
largest coverage for the accuracy constraint (SAC). While each deployment
scenario specifies its own constraints, for demonstration purposes we consider
a scenario in which misclassifications are by far more costly than abstaining
from giving correct predictions. An example for this could be classifying a
huge unlabeled dataset (or for cleaning bad labels from a labeled dataset).
While it is desirable to assign labels to a larger portion of the dataset (or
to correct more of the wrong labels), it is crucial that these labels are as
accurate as possible (or that correctly labeled instances are not replaced
with a bad label).

Figure 17: A comparison of 484 models by their log(number of model’s
parameters) and the coverage they are able to provide for a SAC of 99% (higher
is better) on ImageNet.

To explore such a scenario, we evaluate all models on ImageNet to see which
ones give us the largest coverage for a required accuracy of 99%. In Figure 5,
Section 3 (paper’s main body) we observe that of all the models studied, only
ViT models are able to provide coverage beyond 30% for such an extreme
constraint. Moreover, we note that the coverage they provide is significantly
larger than that given by models with comparable accuracy or size, and that
ViT models that provide similar coverage to their counterparts do so with less
overall accuracy.

In Figure 17 we see that not only do ViT models provide more coverage than any
other model, but they are also able to do so in any size category. To compare
models fairly by their size, we present Figure 17, which sets the Y axis to be
the logarithm of the number of parameters, so that models sharing the same y
value can be compared solely based on their x value—which is the coverage they
provide for a SAC of 99%. We see that ViT models provide a larger coverage
even when compared with models of a similar size.

##  Appendix K Evaluations of Monte-Carlo Dropout In-Distribution ranking
performance

MC-Dropout gal2016dropout is computed using several dropout-enabled forward
passes to produce uncertainty estimates. In classification, the mean softmax
score of these passes is calculated, and then a predictive entropy score is
used as the final uncertainty estimate. In our evaluations, we use 30 dropout-
enabled forward passes. We do not measure MC-Dropout’s effect on ECE since
entropy scores do not reside in $[0,1]$.

We test this technique using MobileNetV3 Howard et al. [2019], MobileNetv2
sandler2019mobilenetv2 and VGG simonyan2014very, all trained on ImageNet and
taken from the PyTorch repository Paszke et al. [2019].

Table 2: Comparing using MC-Dropout to softmax-response (vanilla).

Architecture | Method | Accuracy | AUROC
---|---|---|---
| Vanilla | 74.04 | 86.88
MobileNetV3 Large | MC-Dropout | 74 | 86.14
| Vanilla | 67.67 | 86.2
MobileNetV3 Small | MC-Dropout | 67.55 | 84.54
| Vanilla | 71.88 | 86.05
MobileNetV2 | MC-Dropout | 71.81 | 84.68
| Vanilla | 70.37 | 86.31
VGG11 | MC-Dropout | 70.21 | 84.3
| Vanilla | 69.02 | 86.19
VGG11 (no BatchNorm) | MC-Dropout | 68.95 | 83.94
| Vanilla | 71.59 | 86.3
VGG13 | MC-Dropout | 71.43 | 84.37
| Vanilla | 69.93 | 86.24
VGG13 (no BatchNorm) | MC-Dropout | 69.71 | 84.3
| Vanilla | 73.36 | 86.76
VGG16 | MC-Dropout | 73.33 | 85.02
| Vanilla | 71.59 | 86.63
VGG16 (no BatchNorm) | MC-Dropout | 71.47 | 84.97
| Vanilla | 74.22 | 86.52
VGG19 | MC-Dropout | 74.17 | 85.06
| Vanilla | 72.38 | 86.55
VGG19 (no BatchNorm) | MC-Dropout | 72.37 | 84.99

The results comparing these models with and without using MC-Dropout are
provided in Table 2.

The table shows that using MC-Dropout causes a consistent drop in both AUROC
and selective performance compared with using the same models with softmax as
the $\kappa$. These results are also visualized in comparison to other methods
in Figure 3 in Section 3. MC-Dropout underperformance in an ID setting was
also previously observed in DBLP:journals/corr/GeifmanE17.

##  Appendix L Constructing C-OOD dataset per model

Given model $f$, confidence function $\kappa$ and ${\cal O}$: a large set of
classes ${\cal Y}_{OOD}$ (in our case, ImageNet-21K), our goal is to build
multiple datasets from ${\cal Y}_{OOD}$ that can be used in evaluating the
model’s performance in C-OOD detection. For that we start by pre-processing
${\cal O}$.

###  L.1 pre-processing ImageNet-21K

Since ImageNet-21K contains our ID dataset (ImageNet-1K), the first step will
be to remove all 1K classes from ImageNet-21K. After that, as a cautionary
step, we remove all classes that are hypernyms or hyponyms of classes in
ImageNet-1K because they might be unfair to include as an OOD class. For
example, ImageNet-1K contains the class “brown bear”, and ImageNet-21K has the
class “bear” which is a hypernym for “brown bear” so it would not be fair to
include it in a C-OOD detection test. After cleaning trivial classes, we
remove classes with a low number of samples; we set our threshold to 200
samples. For classes with more than 200 samples we randomly select 200 samples
and remove the rest. At the end of this process we are left with 12563 classes
containing 200 samples each. We divide the samples in each class $c^{y}$ into
150 ‘estimation’ samples (denoted by $c_{est}^{y}$) and 50 ‘test’ samples
(denoted by $c_{test}^{y}$). $c_{est}^{y}$ will later be used to estimate the
difficulty of the class and $c_{test}^{y}$ will be used to check the C-OOD
performance if the class was chosen.

###  L.2 Construction

Having conducted the pre-processing, we continue to compute the 11 severity
levels (C-OOD datasets) for the given model $f$ (with its given $\kappa$).

We define the severity score of class $y$ to ($f,\kappa$) as follows:

$s(y|f,\kappa)=\frac{1}{|c_{est}^{y}|}\sum_{x\in c_{est}^{y}}\kappa(x|f).$

We also define the severity score for a given group of classes $g$ as:

$s(g|f,\kappa)=\frac{1}{|g|}\sum_{y\in g}s(y|f,\kappa).$

Such a definition of $s$ is intuitive because it assumes that samples that the
model is highly confident of are hard for it to distinguish from ID samples.
We choose the size of each C-OOD dataset to be the same as the size of the of
the ID dataset, 1000 classes. The number of subgroups in ${\cal Y}_{OOD}$ of
size 1000 is huge (${12563\choose 1000}=5.5\times 10^{1513}$ groups), so
instead of going over every possible group of classes, we choose to sort the
classes by their severity score and then use a sliding window of size 1000 to
define resulting in 11564 groups of classes with increasing severities. This
choice for reducing the number of considered groups of classes was chosen for
its simplicity. Using $s(g|f,\kappa)$ we calculate the severity score of each
group, and sort them accordingly. Finally we choose the 11 groups to be the
groups that correspond to the percentiles $\\{10\cdot i\\}_{i=0}^{i=10}$ in
the sorted groups array. This way we build the C-OOD dataset of severity level
$i$ from the test samples of classes in group $i$. This heuristic procedure
for choosing groups allows us to interpret the severity levels with
percentiles. For example, severity level 5 contains classes that have the
median severity among the considered groups.

The reason for choosing the number of classes in each group to be the same as
the number of classes in the ID dataset (1000 classes per group) is because we
wanted the C-OOD dataset to be equal in size to the ID dataset.

##  Appendix M List of C-OOD observations

In this section we list additional findings regarding C-OOD detection.

###  M.1 Per-size Comparison

Figure 18: Number of architecture parameters vs. C-OOD AUROC performance at
severity level 5 (median severity). The pair of numbers next to each
architecture name at the legend correspond to its Spearman correlation and the
number of models tested from that architecture (family), respectively. Note
that ViT transformers are also the best when considering a model size
limitation. Vertical lines indicate the sizes of ResNet-50 (left vertical
line) and ResNet-101 (right vertical line).

The scatter plot in Figure 18 shows the relationship between the # of
architecture parameters and its C-OOD AUROC performance. Overall, there is a
moderate Spearman correlation of 0.45 between #parameters and the C-OOD
performance when considering all tested networks. When grouping the networks
by architecture families, however, we see that some architectures have high
correlation between their model size and their C-OOD AUROC. Architecture
families that exhibit this behavior are, for example, ViTs, Swins,
EffecientNetV2 and ResNets whose correlations are 0.91, 0.94, 0.89, and 0.79,
respectively. Other families exhibit moderate correlations, e.g.,
EffecientNet(V1) with a 0.47 Spearman correlation. Some architectures, on the
other hand, have strong negative correlation, e.g., Twins Chu et al. [2021],
NesT Zhang et al. [2020] and Res2Net Gao et al. [2021], whose correlations are
-0.94,-1.0, and -0.85, respectively.

Additionally, we note that ViT models are also the best even when considering
a model size limitation similar to ResNet-50 or ResNet-101.

###  M.2 High correlation with accuracy

Figure 19: Architecture accuracy vs. C-OOD AUROC performance. In the legend,
the pair of numbers next to each architecture name correspond to the Spearman
correlation and the number of networks tested from that architecture (family),
respectively. Accuracy appears to have a high correlation with the C-OOD
detection performance, with a Spearman correlation of 0.65. Most architectures
also hold this general trend except for Nest and Twins. Next to each
architecture we report the Spearman correlation value and the number of
networks tested from that architecture.

Similarly, the scatter plot in Figure 19 shows the relationship between the
architecture validation accuracy and its C-OOD AUROC performance. Based on
their apparent correlation, increasing accuracy could indicate better C-OOD
detection performance. When grouping the networks by architecture, we notice
that most architectures also hold this trend. In Appendix M.7, we examine how
the correlation between C-OOD AUROC and accuracy (among other metrics) change
with severity.

###  M.3 Training Regime Effects

Figure 20: The average relative improvement when using distillation,
pretraining, semi-supervised learning and adversarial training. The shaded
green area indicates the area of positive improvement.

We evaluate the effect of several training regimes on C-OOD performance at
various severity levels. The regimes we consider are: (1) Training that
involves knowledge distillation in any form; (2) Adversarial training; (3)
Pretraining on ImageNet21k; (4) Various forms of weakly or semi-supervised
learning, including noisy student Xie et al. [2020] and semi-supervised
pretraining Yalniz et al. [2019], which use extra unlabeled data; (5) Weakly
supervised training Mahajan et al. [2018], which uses billions of weakly
labeled Instagram images. The relative improvement generated by each one of
the training regimes is depicted in Figure 20. We observe that distillation
has a positive improvement on C-OOD detection performance on all severity
levels, adversarial training has minimal effects on performance, and, somewhat
surprisingly, training regimes that include pretraining on larger datasets
(pretraining on ImageNet-21K and WSP) does not hinder performance in detecting
OOD classes at lower severity levels, but degrades performance on higher
severities—which is what we originally expected (due to exposure to the OOD
classes in training).

###  M.4 Entropy vs. Softmax

Figure 21: Relative improvement gain in C-OOD detection performance when using
entropy instead of the softmax confidence signal. In median network terms,
entropy offers positive improvement over softmax in most serverities except
$s\in\\{7,8,9\\}$. The green shaded area indicates the area of positive
improvement.

In our research we evaluated entropy as an alternative confidence rate signal
($\kappa$) for C-OOD detection.111Entropy is maximal when the distribution
given by the network for $P(y|x)$ is uniform, which implies high uncertainty.
To convert entropy into a _confidence signal_ we use negative entropy. For
each network $f$ we re-run the algorithm described in Section L for extracting
the classes for the 11 severity levels for $(f,\kappa_{entropy})$ and use them
to benchmark the C-OOD detection for $(f,\kappa_{entropy})$ (Note that using
the same C-OOD groups produced when using softmax might yield an unfair
advantage to entropy). We compare the performance gain from switching to using
entropy instead of the softmax score. The results are depicted using box-plots
in Figure 21. We notice that in most cases using entropy improves the
detection performance.

###  M.5 Monte-Carlo Dropout for C-OOD Detection

Figure 22: Relative improvement gain in C-OOD detection performance when using
MC-Dropout instead of softmax confidence signal. We find that MC-dropout
improves performance, especially so at lower severities. The improvement
becomes less significant as severities increase. Figure 23: Relative
improvement gain in C-OOD detection performance when using MC-Dropout instead
of entropy confidence signal. We find that MC-Dropout improves upon entropy in
most severity levels, and especially at lower ones. This suggests that the
usage of entropy as part of MC-Dropout does not explain on its own MC-
Dropout’s better detection performance.

We evaluate MC-Dropout gal2016dropout in the context of C-OOD detection. We
use 30 dropout-enabled forward passes. The mean softmax score of these passes
is calculated, and then a predictive entropy score is used as the final
uncertainty estimate. We test this technique using MobileNetV3 Howard et al.
[2019], MobileNetv2 sandler2019mobilenetv2 and VGG simonyan2014very, all
trained on ImageNet and taken from the PyTorch repository Paszke et al.
[2019].

For each of these three architectures we re-run the algorithm described in
Section L to extract the classes for all 11 severity levels for
$(f,\kappa_{\text{MC-dropout}})$ and use them to benchmark the C-OOD detection
for $(f,\kappa_{\text{MC-dropout}})$.

The improvements across all severities of using MC-Dropout instead of softmax
are depicted using box-plots in Figure 22. We find that MC-dropout improves
performance, especially so at lower severities. The improvement becomes less
significant as severities increase. We further analyze MC-dropout and recall
that it is composed of two main components: (1) dropout-enabled forward passes
(2) entropy of the mean probability vector from the forward passes. To test
which component contributes the most to the perceived gains, we compare the
C-OOD detection performance when using MC-dropout to the C-OOD detection
performance when using just entropy (with no multiple dropout-enabled forward
passes). We find that MC-Dropout improves upon entropy in most severity
levels, and especially at lower ones. This suggests that the usage of entropy
as part of MC-Dropout does not explain on its own MC-Dropout’s better
detection performance. These results can be seen in Figure 23.

###  M.6 Correlation between Rankings of Multiple Severity Levels

Since we essentially have multiple benchmarks C-OOD datasets (i.e., the 11
severity levels) to test the performance models in C-OOD detection, and each
severity level may rank the models differently, we now consider the question
of how does these ranking change across severity levels. To this end we
calculated the correlations between the rankings obtained at different
severity levels. The resulting correlation matrix can be seen in Figure 24.
Overall we observe high correlations, which means that different severity
levels generally yield similar rankings of the models. We also notice that for
each severity level $s_{i}$, the correlation with $s_{j}$ is higher the closer
$j$ is to $i$. This is not surprising and might be anticipated because
adjacent severity levels have close severity score by design.

Figure 24: Spearman correlation between the rankings of the models given by
different severity level.

###  M.7 Correlations of Various Factors with C-OOD Detection Performance

We searched for factors that could be indicative or correlated with good
performance in C-OOD detection. To this end we measure the correlations of
various factors with the C-OOD detection AUROC performance across all
severities. The results can be seen in the graphs of Figure 25. We observe
that accuracy is typically a good indicator of the model’s performance in
C-OOD detection at most severity levels ($s_{0}-s_{8}$), with Spearman
correlation values in $[0.6,0.7]$ at those levels. The next best indicative
factors are the ID-AUROC performance, number of parameters, and the input
image size (moderate correlations). Finally, the embedding size is only weakly
correlated. Interestingly, ID-AUROC exhibits slightly increasing correlation
up to severity $s_{9}$, and at $s_{10}$ becomes the most indicative factor for
C-OOD detection. In contrast, all other investigated factors lose their
indicative power at the highest severity levels ($s_{9},s_{10})$.

Figure 25: Spearman correlations between C-OOD detection AUROC and Accuracy,
ID-AUROC, #Parameters, Input size, Embedding size across all severity levels.

###  M.8 Analysing the C-OOD classes chosen by each model

When analysing the classes chosen by the algorithm (described in Section L)
for the various models and severity levels, we found that for each severity
level, the union of the classes chosen for all models (for that severity
level) spans all the classes in ${\cal Y}_{OOD}$ (namely, all the classes in
ImageNet-21K after filtration). This implies that hard OOD classes for one
model can easy for others.

In addition, for each severity level (except $s_{10}$) there is no single
class that is shared between all models. However, in $s_{10}$ we find that
there are some shared classes between all models. Moreover, there are specific
instances (images) in those classes that obtain the same wrong prediction from
many models; one interesting example for this shared wrong prediction of 105
models appears in Figure 26, where the butterfly image is wrongly classified
as a fox squirrel.

Figure 26: An instance of viceroy butterfly predicted to be a fox squirrel by
105 models.