# What classifiers know what they don’t?

Mohamed Ishmael Belghazi and David Lopez-Paz
{ishmaelb<EMAIL_ADDRESS>
Facebook AI Research, Paris, France

###### Abstract

Being uncertain when facing the unknown is key to intelligent decision making.
However, machine learning algorithms lack reliable estimates about their
predictive uncertainty. This leads to wrong and overly-confident decisions
when encountering classes unseen during training. Despite the importance of
equipping classifiers with uncertainty estimates ready for the real world,
prior work has focused on small datasets and little or no class discrepancy
between training and testing data. To close this gap, we introduce UIMNET: a
realistic, ImageNet-scale test-bed to evaluate predictive uncertainty
estimates for deep image classifiers. Our benchmark provides implementations
of eight state-of-the-art algorithms, six uncertainty measures, four in-domain
metrics, three out-domain metrics, and a fully automated pipeline to train,
calibrate, ensemble, select, and evaluate models. Our test-bed is open-source
and all of our results are reproducible from a fixed commit in our repository.
Adding new datasets, algorithms, measures, or metrics is a matter of a few
lines of code—in so hoping that UIMNET becomes a stepping stone towards
realistic, rigorous, and reproducible research in uncertainty estimation. Our
results show that ensembles of ERM classifiers as well as single MIMO
classifiers are the two best alternatives currently available to measure
uncertainty about both in-domain and out-domain classes.

##  1 Introduction

_I don’t think I’ve ever seen anything quite like this before_

—HAL 9000 in 2001: A Space Odyssey

Deep image classifiers exceed at discriminating the set of _in-domain_ classes
observed during training. However, when confronting test examples from unseen
_out-domain_ classes, these classifiers can only predict in terms of their
known in-domain categories, leading to wrong and overly-confident decisions
(Hein et al., 2019; Ulmer and Cinà, 2020). In short, machine learning systems
are unaware of their own limits, since “they do not know what they do not
know”. Because of this reason, out-domain data cannot be safely identified and
treated accordingly. Thus, it is reasonable to fear that, when deployed in-
the-wild, the behavior of these classifiers becomes unpredictable and their
performance crumbles by leaps and bounds (Ovadia et al., 2019).

The inability of machine learning systems to estimate their uncertainty and
abstaining to classify out-domain classes is roadblock towards their
implementation in critical applications. These include self-driving
(Michelmore et al., 2018), medicine (Begoli et al., 2019), and the analysis of
satellite imagery (Wadoux, 2019). Good uncertainty estimates are also a key
ingredient in anomaly detection (Chalapathy and Chawla, 2019), active learning
(Settles, 2009), safe reinforcement learning (Henaff et al., 2019), defending
against adversarial examples (Goodfellow et al., 2014), and model
interpretability (Alvarez-Melis and Jaakkola, 2017). For an extensive
literature review on uncertainty estimation and its applications, we refer the
curious reader to the survey of Abdar et al. (2020) and the one of Ruff et al.
(2021). Despite a research effort spanning multiple decades, machine learning
systems still lack trustworthy estimates of their predictive uncertainty. In
our view, one hindrance to this research program is the absence of realistic
benchmarking and evaluation protocols. More specifically, prior attempts are
limited in two fundamental ways. First, these experiment on small datasets
such as SVHN and CIFAR-10 (van Amersfoort et al., 2021). Second, these do not
provide a challenging set of out-domain data. Instead, they construct out-
domain classes by using a second dataset (e.g., using MNIST in-domain versus
FashionMNIST out-domain, cf. Van Amersfoort et al. (2020)) or by perturbing
the in-domain classes using handcrafted transformations (such as Gaussian
noise or blur, see ImageNet-C Hendrycks and Dietterich (2019)). Both
approaches result in simplistic benchmarking, and little is learned about
uncertainty estimation for the real world. The purpose of this work is to
introduce an end-to-end benchmark and evaluation protocol bridging this
disconnect. At the time of writing, UIMNET is the most exhaustive benchmark
for uncertainty estimation in the literature.

Datasets | Algorithms | Uncertainty | In-Domain | Out-Domain | Ablations
---|---|---|---|---|---
|  | measures | metrics | metrics |
ImageNot | ERM | Largest | ACC@1 | AUC | Calibration (y/n)
| Mixup | Gap | ACC@5 | InAsIn | Spectral norm (y/n)
| Soft-labeler | Entropy | ECE | InAsOut | Model size (RN18 / RN50)
| RBF | Jacobian | NLL | OutAsIn |
| RND | GMM |  | OutAsOut |
| OCs | Native |  |  |
| MC-Dropout |  |  |  |
| MIMO |  |  |  |
| (+ Ensembles) |  |  |  |
Table 1: The UIMNET test-bed suite for uncertainty estimation.

#### Formal setup

Following conventional supervised learning, we learn a classifier $f$ using
_in-domain_ data from the distribution $P_{\text{in}}(X,Y)$. After training,
we endow the classifier with a real-valued uncertainty measure
$u(f,x^{\dagger})$. Given a test example $(x^{\dagger},y^{\dagger})\sim P$
with unobserved label $y^{\dagger}$, we declare $x^{\dagger}$ _in-domain_
(hypothesizing $P=P_{\text{in}}$) if $u(f,x^{\dagger})$ is small, whereas we
declare $x^{\dagger}$ _out-domain_ (hypothesizing $P\neq P_{\text{in}}$) if
$u(f,x^{\dagger})$ is large. Using these tools, our goal is to abstain from
classifying out-domain test examples, and to classify with calibrated
probabilities in-domain test examples. The sequel assumes that the difference
between in- and out-domain resides in that the two groups of data concern
disjoint classes.

#### Contributions

We introduce UIMNET, a test-bed for large-scale, realistic evaluation of
uncertainty estimates in deep image classifiers. We outline the components of
the test-bed below, also summarized in Table 1.

  * (Sec. 2)

We construct ImageNot, a perceptual partition of ImageNet into _in-domain_ and
_out-domain_ classes. Unlike prior work focused on small datasets like SVHN
and CIFAR-10, ImageNot provides a benchmark for uncertainty estimators at a
much larger scale. Moreover, both in-domain and out-domain categories in
ImageNot originate from the original ImageNet dataset. This provides realistic
out-domain data, as opposed to prior work relying on a second dataset (e.g.,
MNIST as in-domain versus SVHN as out-domain), or handcrafted perturbations of
in-domain classes (Gaussian noise or blur as out-domain).

  * (Sec. 3)

We re-implement eight state-of-the-art algorithms from scratch, listed in
Table 1. This allows a fair comparison under the exact same experimental
conditions (training/validation splits, hyper-parameter search, neural network
architectures and random initializations). Furthermore, we also study
ensembles of multiple training instances for each algorithm.

  * (Sec. 4)

Each algorithm can be endowed with one out of six possible uncertainty
measures, allowing an exhaustive study of what algorithms play well with what
measures. Listed in Table 1, these are the largest softmax score, the gap
between the two largest softmax scores, the softmax entropy, the norm of the
Jacobian, a per-class Gaussian density model, and (for those available) an
algorithm-specific measure.

  * (Sec. 5)

For each classifier-measure pair, we study four in-domain metrics (top-1 and
top-5 classification accuracy, log-likelihood, expected calibration error) and
three out-domain metrics (the AUC at classifying in-domain versus out-domain
samples using the selected uncertainty measure, as well as the confusion
matrix at a fixed uncertainty threshold computed over an in-domain validation
set).

  * (Sec. 6)

We repeat our entire pipeline to accommodate three popular ablations to
understand the impact of model calibration by temperature scaling, model size,
and the use of spectral normalization.

  * (Sec. 7)

UIMNET is entirely hands-off, since the pipeline from zero to LaTeX tables is
fully automated: this includes hyper-parameter search, model calibration,
model ensembling, and the production of all the tables included in our
experimental results.

  * (Sec. 8)

Our experimental results illustate that no classifier but those trained by
using Multiple-Input and Multiple-Output (MIMO) systematically dominates
calibrated classifiers trained by empirical risk minimization. Interestingly,
a single MIMO classifier _if given enough capacity_ matches ensembles of ERMs
in terms of AUC score and identification of in-domain instances and even edges
them when identifying out-domaim instances. Additionally, our result show that
the use of spectral normalization does not significantly improve out of domain
detection metrics while the effect of calibration remains mild at best.

UIMNET is open sourced at https://github.com/facebookresearch/uimnet. All of
the tables presented in this paper are reproducible by running the main script
in the repository at commit 0xSOON.

##  2 Constructing the ImageNot benchmark

The ImageNet dataset (Russakovsky et al., 2015), a gold standard to conduct
research in supervised learning for computer vision, pertains the
classification of images into $1000$ different classes. Here we will use the
ImageNet dataset to derive ImageNot, a large-scale and realistic benchmark for
uncertainty estimation. In a nutshell, we will divide the $1000$ classes of
the original ImageNet dataset into _in-domain_ classes (used to train and
evaluate algorithms in-distribution) and _out-domain_ classes (used to
evaluate algorithms out-of-distribution).

To partition ImageNet into in-domain and out-domain, we featurize the entire
dataset to understand the perceptual similarity between classes. To this end,
we use a pre-trained ResNet-18 (He et al., 2016) to compute the average last-
layer representation for each of the classes. Next, we use agglomerative
hierarchical clustering Ward Jr (1963) to construct a tree describing the
perceptual similarities between the $1000$ average feature vectors. Such
perceptual tree has $1000$ leafs, each of them being a cluster containing one
of the classes. During each step of the iterative agglomerative clustering
algorithm the two closest clusters are merged, where the distance between two
clusters is computed using the criterion of Ward Jr (1963). The algorithm
halts when there are only two clusters left to merge, forming the root node of
the tree.

At this point, we declare the $266$ classes to the left of the root as _in-
domain_ , and the first $266$ classes to the right of the root as _out-
domain_. In the sequel, we call “training set” and “validation set” to a 90/10
random split from the original ImageNet “train” set. We call “testing set” to
the original ImageNet “val” split. The exact in-/out-domain class partition as
well as the considered train/validation splits are specified in Appendix C.

While inspired by the BREEDS dataset (Santurkar et al., 2020), our benchmark
ImageNot is conceived to tackle a different problem. The aim of the BREEDS
dataset is to classify ImageNet into a small number of super-classes, each of
them containing a number of perceptually-similar sub-classes. The BREEDS
training and testing distributions differ on the sub-class proportions
contributing to their super-classes. Since the BREEDS task is to classify
super-classes, the set of labels remains constant from training to testing
conditions. This is in contrast to ImageNot, where the algorithm observes only
in-domain classes during training, but both in-domain and out-domain classes
during evaluation. While BREEDS studies the important problem of domain
generalization (Gulrajani and Lopez-Paz, 2020), where there is always a right
prediction to make within the in-domain classes during evaluation, here we
focus on measuring uncertainty and abstaining from out-domain classes unseen
during training.

Also related, our benchmark ImageNot is similar to the ImageNet-O dataset of
(Hendrycks et al., 2021). However, their out-domain classes are obtained from
heterogeneous sources outsider of ImageNet, which have distinct statistics
that can be easily picked up by the classifier. In contrast, the starting
point for both in-domain and out-domain classes of our ImageNot is the same
(the original ImageNet dataset) and thus should maximally overlap in terms of
image statistics, leading to a more challenging, realistic benchmark.

##  3 Algorithms

We benchmark eight supervised learning algorithms that are commonly applied to
tasks involving uncertainty estimation. Each algorithm consumes one in-domain
training set of image-label pairs $\\{(x_{i},y_{i})\\}_{i=1}^{n}$ and returns
a _predictor_ 111In the sequel, we denote by _classifier_ the last few layers
in the _predictor_ that follow after the _featurizer_. $f(x)=w(\phi(x))$,
composed by a _featurizer_ $\phi:\mathbb{R}^{3\times 224\times
224}\to\mathbb{R}^{k}$ and a _classifier_ $w:\mathbb{R}^{k}\to\mathbb{R}^{C}$.
We consider predictors implemented using deep convolutional neural networks
(LeCun et al., 2015). Given an input image $x^{\dagger}$, all predictors
return a softmax vector $f(x^{\dagger})=(f(x^{\dagger})_{c})_{c=1}^{C}$ over
$C$ classes. The considered algorithms are:

  * •

Empirical Risk Minimization, or ERM (Vapnik, 1992), or vanilla training.

  * •

Mixup (Zhang et al., 2017) chooses a predictor minimizing the empirical risk
on _mixed_ examples ${(\tilde{x},\tilde{y})}$, built as:

$\displaystyle\lambda$ $\displaystyle\sim\text{Beta}(\alpha,\alpha),$
$\displaystyle\tilde{x}$ $\displaystyle\sim\lambda\cdot x_{i}+(1-\lambda)\cdot
x_{j},$ $\displaystyle\tilde{y}$ $\displaystyle\sim\lambda\cdot
y_{i}+(1-\lambda)\cdot y_{j},$

where $\alpha$ is a mixing parameter, and $((x_{i},y_{i}),(x_{j},y_{j}))$ is a
random pair of training examples. Mixup has been shown to improve both
generalization performance (Zhang et al., 2017) and calibration error
(Thulasidasan et al., 2019).

  * •

Random Network Distillation, or RND (Burda et al., 2018), finds an ERM
predictor $f(x)=w(\phi(x))$, but also trains an auxiliary classifier
$w_{\text{student}}$ to minimize

$\|w_{\text{student}}(\phi(x))-w_{\text{teacher}}(\phi(x))\|^{2}_{2},$

where $w_{\text{teacher}}$ is a fixed classifier with random weights. RND has
shown good performance as a tool for exploration in reinforcement learning.

  * •

Orthogonal Certificates, or OC (Tagasovska and Lopez-Paz, 2018), is analogous
to RND for $w_{\text{teacher}}(\phi(x))=\vec{0}_{k}$ for all $x$. That is, the
goal of $w_{\text{student}}$ is to map all the in-domain training examples to
zero in $k$ different ways (or _certificates_). To ensure diverse and non-
trivial certificates, we regularize each weight matrix $W$ of
$w_{\text{student}}$ to be orthogonal by adding a regularization term
$\|W^{\top}W-I\|_{2}^{2}$. OCs have shown good performance at the task of
estimating uncertainty across a variety of classification tasks.

  * •

MC-Dropout (Gal and Ghahramani, 2016) uses ERM over a family of predictors
with one or more dropout layers (Srivastava et al., 2014). These stochastic
dropout layers remain active at test time, allowing the predictor to produce
multiple softmax vectors $\\{f(x^{\dagger},\text{dropout}_{t})\\}_{t=1}^{T}$
for each test example $x^{\dagger}$. Here, $\text{dropout}_{t}$ is a random
dropout mask sampled anew. MCDropout is one of the most popular baselines to
estimate uncertainty.

  * •

MIMO (Havasi et al., 2021) is a variant of ERM over predictors accepting $T$
images and producing $T$ softmax vectors. For example, MIMO with $T=3$ is
trained to predict jointly the label vector $(y_{i},y_{j},y_{k})$ using a
predictor $h(x_{i},x_{j},x_{k})$, where $((x_{t},y_{t}))_{i=1}^{3}$ is a
random triplet of training examples. Given a test point $x^{\dagger}$, we form
predictions by replicating and averaging, that is
$f(x^{\dagger})=\frac{1}{3}\sum_{t=1}^{3}h(x^{\dagger},x^{\dagger},x^{\dagger})_{t}$.

  * •

Radial Basis Function, or RBF (Broomhead and Lowe, 1988), is a variant of ERM
where we transform the logit vector $z\mapsto e^{-z^{2}}$ before passing them
to the final softmax layer. In such a way, as the logit norm $\|z\|\to\infty$,
the predicted softmax vector tends to the maximum entropy solution
$(\frac{1}{C})_{c=1}^{C}$, signaling high uncertainty far away from the
training data. RBFs have been proposed as defense to adversarial examples
(Goodfellow et al., 2014), but they remain under-explored given the
difficulties involved in their training.

  * •

Soft labeler (Hinton et al., 2015; Szegedy et al., 2016) is a variant of ERM
where the one-hot vector labels $y_{i}$ are _smoothed_ such that every zero
becomes $\ell_{\text{min}}>0$ and the solitary one becomes
$\ell_{\text{max}}<1$. Softening labels avoids saturation in the final softmax
layer in neural network predictors, one of the main causes of overly-confident
predictors. Using soft labels, we can identify softmax vectors with entries
exceeding $\ell_{\text{max}}$ as “over-shoots”, regarding them as uncertain
extrapolations.

#### Ensembles of predictors

We also consider ensembles of predictors trained by each of the algorithms
above. Ensembles are commonly regarded as the state-of-the-art in uncertainty
estimation (Lakshminarayanan et al., 2016). In particular, and for each
algorithm, we construct bagging ensembles by (i) selecting the best
$K\in\\{1,5\\}$ predictors $\\{f^{k}\\}_{k=1}^{K}$ from all considered random
initializations and hyper-parameters, and (ii) returning the average function
$f(x^{\dagger}):=\frac{1}{K}\sum_{k=1}^{M}f^{k}(x^{\dagger})$.

##  4 Uncertainty measures

Using UIMNET, we can equip a trained predictor $f$ with six different
uncertainty measures. An uncertainty measure is a real-valued function
$u(f,x^{\dagger})$ designed to return small values for _in-domain_ instances
$x^{\dagger}$, and large values for _out-domain_ instances $x^{\dagger}$. To
describe the different measures, let $\\{s_{(1)},\ldots,s_{(C)}\\}$ be the
softmax scores returned by $f(x^{\dagger})$ sorted in decreasing order.

  * •

Largest (Hendrycks and Gimpel, 2016) returns minus the largest softmax score,
$-s_{(1)}$

  * •

Softmax gap (Tagasovska and Lopez-Paz, 2018) returns $s_{(2)}-s_{(1)}$.

  * •

Entropy (Shannon, 1948) returns
$-\sum_{c=1}^{C}s_{(c)}\mathbb{I}\\{s_{(c)}>0\\}\log s_{(c)}$.

  * •

Norm of the Jacobian (Novak et al., 2018) returns
$\|\nabla_{x}f(x^{\dagger})\|^{2}_{2}$.

  * •

GMM (Mukhoti et al., 2021) estimates one Gaussian density
$\mathcal{N}(\phi(x);\mu_{c},\Sigma_{c})$ per-class, on top of the feature
vectors $\phi(x)$ collected from a in-domain validation set. Given a test
example $x^{\dagger}$, return
$-\sum_{c=1}^{C}\lambda_{c}\cdot\mathcal{N}(\phi(x^{\dagger});\mu_{c},\Sigma_{c})$,
where $\lambda_{c}$ is the proportion of in-domain validation examples from
class $c$.

  * •

Test-time augmentation (Ashukha et al., 2020) returns
$-\max_{c}(\frac{1}{A}\sum_{a=1}^{A}f(x^{\dagger}_{a}))_{c}$. This is the
measure “Largest” about the average prediction over $A$ random data
augmentations $\\{x^{\dagger}_{a}\\}_{a=1}^{A}$ of the test instance
$x^{\dagger}$.

These uncertainty measures are applicable to all the algorithms considered in
Section 3. Additionally, some algorithms provide their Native uncertainty
measures, outlined below.

  * •

For Mixup, we return $\|\lambda\cdot
f(x^{\dagger})+(1-\lambda)\cdot\bar{y}-f(\lambda\cdot
x^{\dagger}+(1-\lambda)\cdot\bar{x})\|_{2}^{2}$, where we recall that
$\lambda\sim\text{Beta}(\alpha,\alpha)$, and $(\bar{x},\bar{y})$ is the
average image and label from the training set. This measures if the test
example $x^{\dagger}$ violates the Mixup criterion wrt the training dataset
average.

  * •

For RND and OC, we return
$\|w_{\text{student}}(\phi(x^{\dagger}))-w_{\text{teacher}}(\phi(x^{\dagger}))\|_{2}^{2}$,
that is, we consider a prediction uncertain if the outputs of the student and
teacher disagree. We expect this disagreement to be related predictive
uncertainty, as the student did not observe the behaviour of the teacher at
out-domain instances $x^{\dagger}$.

  * •

For Soft labeler we return $(s_{(1)}-\ell_{\text{max}})^{2}$. This measures
the discrepancy between the largest softmax and the positive soft label
target, able to signal overly-confident predictions.

  * •

For MC-Dropout and Ensembles, and following (Lakshminarayanan et al., 2016),
we return the Jensen-Shannon divergence between the $K$ members (or stochastic
forward passes) $f^{1},\ldots,f^{K}$ of the ensemble:

$H\left(\frac{1}{K}\sum_{k=1}^{K}f^{k}(x^{\dagger})\right)-\frac{1}{K}\sum_{k=1}^{K}H(f^{k}(x^{\dagger})).$

Note that the models ERM, OC, and RND are equivalent in all aspects except in
comparisons involving the uncertainty measure Native. This is because the only
difference between these three models is the training of an external student
for RND and OC, used only in their Native uncertainty measures.

##  5 Evaluation metrics

For each algorithm-measure pair, we evaluate several metrics both _in-domain_
and _out-domain_.

###  5.1 In-domain metrics

Following (Havasi et al., 2021), we implement four metrics to assess the
performance and calibration of predictors when facing in-domain test examples.

  * •

Top-1 and Top-5 classification accuracy (Russakovsky et al., 2015).

  * •

Expected Calibration Error or ECE (Guo et al., 2017):

$\frac{1}{B}\sum_{b=1}^{B}\frac{|B_{b}|}{n}\left|\text{acc}(f,B_{b})-\text{conf}(f,B_{b})\right|,$

where $B_{b}$ contains the examples where the algorithm predicts a softmax
score of $b$. The functions _acc_ and _conf_ compute the average
classification accuracy and largest softmax score of $f$ over $B_{b}$. In a
nutshell, ECE is minimized when $f$ is calibrated, that is, $f$ is wrong $p\%$
of the times it predicts a largest softmax score $p$. Following (Guo et al.,
2017), we discrete $b\in[0,1]$ into $15$ equally-spaced bins.

  * •

Negative Log Likelihood (NLL) Also known as the cross-entropy loss, this is
the objective minimized during the training process of the algorithms.

###  5.2 Out-domain metrics

After measuring the performance of a predictor in-domain, we equip it with an
uncertainty measure. We assess the uncertainty estimates of each predictor-
measure pair using three metrics:

  * •

Area Under the Curve, or AUC (Tagasovska and Lopez-Paz, 2018), describes how
well does the predictor-measure pair distinguish between in-domain and out-
domain examples over all thresholds of the uncertainty measure.

  * •

Confusion matrix at fixed threshold. To reject out-domain examples in real
scenarios, one must fix a threshold $\theta$ for the selected uncertainty
measure. We do so by computing the 95% quantile of the uncertainty measure,
computed over an in-domain validation set. Then, at testing time, we declare
one example out-domain if the uncertainty measure exceeds $\theta$. This
strategy is equivalent to the statistical hypothesis test with null “$H_{0}$:
the observed example is in-domain”. To understand where does the uncertainty
measure hit or miss, we monitor two metrics: InAsIn (percentage of in-domain
examples classified as in-domain), OutAsOut (percentage of in-domain examples
classified as out-domain). Please note that From these two metrics, we can
deduce: InAsOut (percentage in-domain examples classified as out-domain, also
known as false positives or type-I errors), OutAsIn (percentage out-domain
examples classified as in-domain, also known as false negatives or type-II
errors).

##  6 Ablations

We execute our entire test-bed under three additional ablations often
discussed in the literature of uncertainty estimation.

  * •

We study the effect of calibration by temperature scaling (Platt et al.,
1999). To this end, we introduce a temperature scaling $\tau>0$ before the
softmax layer, resulting in predictions $\text{Softmax}(\frac{z}{\tau})$ about
the logit vector $z$. We estimate the optimal temperature $\hat{\tau}$ by
minimizing the NLL of the predictor across an in-domain validation set. We
evaluate all metrics for both the un-calibrated ($\tau=1)$ and calibrated
($\tau=\hat{\tau}$) predictors. According to previous literature (Guo et al.,
2017), calibrated models provide better in-domain uncertainty estimates.

  * •

We analyze the impact of spectral normalization applied to the featurizer
$\phi$. Several recent works (Liu et al., 2020; Van Amersfoort et al., 2020;
van Amersfoort et al., 2021; Mukhoti et al., 2021) have highlighted the
importance of controlling both the _smoothness_ and _sensitivity_ of the
feature extraction process to achieve high-quality uncertainty estimates. On
the one hand, enforcing _smoothness_ _upper_ _bounds_ the Lipschitz constant
of $\phi$, limiting its reaction to changes in the input. Smoothness is often
enforced by normalizing each weight matrix in $\phi$ by its spectral norm
(Miyato et al., 2018). On the other hand, enforcing _sensitivity_ _lower_
_bounds_ the Lipschitz constant of $\phi$, ensuring that the feature space
reacts in some amount when the input changes. Sensitivity is often enforced by
residual connections (He et al., 2016), present in the ResNet models that we
will use throughout our experiments.

  * •

We analyze the impact of the model size, running experiments with both
ResNet-18 and ResNet-50 models (He et al., 2016).

##  7 Experimental protocol

We are now ready to conduct experiments on the ImageNot benchmark for
uncertainty estimation (Section 2) for all combinations of algorithms (Section
3) and measures (Section 4), that we will evaluate under all metrics (Section
5) and ablations (Section 6).

_Hyper-parameter search._ We train each algorithm sixty times, arising from a
combination of (i) ResNet-18 or ResNet-50 architectures (ii) using or not
spectral normalization, (iii) five hyper-parameter trials, and (iv) three
random train/validation splits of the in-domain data (data seeds). We opt for
a random hyper-parameter search (Bergstra and Bengio, 2012), where the search
grid for each algorithm is detailed in Table 2. More specifically, while the
first trial uses the default hyper-parameter configuration suggested by the
authors of each algorithm, the additional four trials explore random hyper-
parameters.

Algorithm | Hyper-parameter | Default value | Random search distribution
---|---|---|---
all | learning rate | 0.1 | $10^{\text{Uniform}(-2,-0.3)}$
momentum | 0.9 | $\text{Choice}([0.5,0.9,0.99])$
weight decay | $10^{-4}$ | $10^{\text{Uniform}(-5,-3)}$
Mixup | mixing parameter | 0.3 | $\text{Choice}([0.1,0.2,0.3,1,2])$
MC-Dropout | dropout rate | 0.05 | $\text{Choice}([0.05,0.1,0.2])$
number of passes | 10 | $\text{Choice}([10])$
MIMO | number of subnetworks | 2 | $\text{RandInt}(2,5)$
prob. input repetition | 0.6 | $\text{Uniform}(0,1)$
batch repetition | 2 | $\text{RandInt}(1,5)$
RND, OC | teacher width | 128 | $\text{Choice}([64,128,256])$
teacher depth | 3 | $\text{Choice}([2,3,4])$
regularization | 0 | $10^{\text{Uniform}(-2,1)}$
Soft labeler | soft label value | 128 | $\text{Choice}([0.7,0.8,0.9])$
Table 2: Default hyper-parameters and random search grids for all algorithms.

_Model selection._ After training all instances of a given algorithm, we
report the average and standard deviation (over data seeds) for all metrics of
two chosen models. On the one hand, we report metrics for the _best model_
($k=1$), which is the one minimizing the average (over data seeds) log-
likelihood in the in-domain validation set. On the other hand, we report
metrics also for the _ensemble model_ formed by the five different hyper-
parameter trials. Finally, we report all results of both the best and the
ensemble model with and without calibration and for all model sizes (Section
6).

_Software and hardware._ All algorithms are implemented in PyTorch (Paszke et
al., 2019) and train ResNet-18 or ResNet-50 backbones (He et al., 2016)
initialized by following (Glorot and Bengio, 2010). In addition, the Gaussian
process in DUE is implemented in GPyTorch (Gardner et al., 2018). These models
are trained by stochastic gradient descent (Robbins and Monro, 1951; Bottou,
2012) for 100 epochs, with mini-batches of 256 examples distributed over 8
NVIDIA Tesla V100 GPUs (yielding an effective mini-batch of size 32 per GPU).
During training, we decay the learning rate by a factor of 10 every 30 epochs.

##  8 Conclusions

Table 3 summarizes our results for in-domain metrics (Section 5.1), and Table
4 summarizes our results for out-domain metrics (Section 5.2) and the best
performing measure (Entropy, see Section 4). These tables contain the results
for all algorithms (Section 3), ablations (use of spectral normalization, use
of temperature calibration; see Section 6), and ensembling (whether bagging
the best $k=1$ or $k=5$ models) for the largest model (ResNet50) considered.
Appendix A contains the in-domain resutls for ResNet18, and Appendix B
contains the out-domain result tables for all the other measures. From the in-
domain results summarized in Table 3, we identify the following key takeaways:

  * •

No algorithm significantly outperforms ERM in any in-domain metric.

  * •

Ensembling multiple models ($k=5$) improves all in-domain metrics.

  * •

Temperature calibration helps, decreasing the expected calibration error by up
to $30\%$.

  * •

Spectral normalization has a neutral and marginal effect on in-domain metrics.

  * •

Most algorithms are able to achieve similar performances, as their hyper-
parameter searches allow them to behave like ERM.

Next, from the out-domain results summarized in Table 4, we identify the
following key takeaways:

  * •

A single MIMO is the only classifier outperforming ensembles of ERM
classifiers on the OutAsOut metric, or the rate of correct identification of
out-domain instances.

  * •

No other algorithm significantly outperforms ERM in any out-domain metric.

  * •

Ensembling models ($k=5$) improves all out-domain metrics for all classifiers
but MIMO.

  * •

Temperature calibration has a neutral and marginal effect on out-domain
metrics.

  * •

Spectral normalization has a marginal negative effect on out-domain metrics.

  * •

Most algorithms are able to achieve similar performances, as their hyper-
parameter searches allow them to behave like ERM.

Appendix B, summarizing the out-domain performance of all other uncertainty
measures, provides us with additional takeaways, in particular:

  * •

The best performing uncertainty measures, are Entropy, Largest, and Gap, with
Entropy taking the lead with a generous gap.

  * •

The uncertainty measures Augmentations and Jacobian exhibit a poor
performance.

  * •

Increasing model size leads to large improvements in the out-domain metric
OutAsOut.

  * •

Given enough capacity, a single MIMO classifier surpasses ensembles of 5 ERM
classifiers on the out-domain metric OutAsOut.

  * •

The Native uncertainty measures (those specific measures provided by some
algorithms) do not exhibit a good performance.

From all of these results, our recommendation us to _use calibrated ensembles
of ERM classifiers_ or _a single MIMO classifier_ , with no need for spectral
normalization, and allowing the largest model size. When computational
resources do not allow large model sizes or the use of ensembles, we recommend
to use _a single calibrated ERM models_.

#### Other results

Despite our best efforts, we were unable to obtain competitive performances
when using the algorithms RBF, (Broomhead and Lowe, 1988) or the uncertainty
measure GMM (Mukhoti et al., 2021). We believe that training RBFs at this
large scale are challenging optimization problems that deserve further study
in our community. Furthermore, we believe that the large number of classes in
our study (266 ImageNET classes instead of the 10 CIFAR-10 classes often
considered) makes for a difficult problem for density-based uncertainty
measures such as GMM.

algorithm | spectral | calibration | k | ACC@1 | ACC@5 | ECE | NLL
---|---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.762\pm 0.004$ | $0.859\pm 0.001$ | $0.050\pm 0.003$ | $0.952\pm 0.013$
5.0 | $0.782\pm 0.001$ | $0.871\pm 0.002$ | $0.030\pm 0.001$ | $0.855\pm 0.001$
learned | 1.0 | $0.763\pm 0.001$ | $0.859\pm 0.004$ | $0.046\pm 0.001$ | $0.945\pm 0.009$
5.0 | $0.782\pm 0.001$ | $0.871\pm 0.002$ | $0.034\pm 0.007$ | $0.864\pm 0.016$
True | initial | 1.0 | $0.764\pm 0.002$ | $0.859\pm 0.001$ | $0.054\pm 0.002$ | $0.957\pm 0.009$
5.0 | $0.782\pm 0.001$ | $0.872\pm 0.002$ | $0.031\pm 0.001$ | $0.853\pm 0.003$
learned | 1.0 | $0.764\pm 0.002$ | $0.859\pm 0.001$ | $0.045\pm 0.001$ | $0.947\pm 0.011$
5.0 | $0.782\pm 0.001$ | $0.872\pm 0.002$ | $0.033\pm 0.005$ | $0.861\pm 0.011$
MCDropout | False | initial | 1.0 | $0.764\pm 0.002$ | $0.860\pm 0.002$ | $0.054\pm 0.001$ | $0.951\pm 0.005$
5.0 | $0.782\pm 0.001$ | $0.872\pm 0.002$ | $0.030\pm 0.001$ | $0.853\pm 0.005$
learned | 1.0 | $0.764\pm 0.002$ | $0.860\pm 0.002$ | $0.045\pm 0.001$ | $0.941\pm 0.004$
5.0 | $0.782\pm 0.001$ | $0.872\pm 0.002$ | $0.033\pm 0.004$ | $0.861\pm 0.008$
True | initial | 1.0 | $0.764\pm 0.004$ | $0.860\pm 0.002$ | $0.054\pm 0.002$ | $0.956\pm 0.009$
5.0 | $0.780\pm 0.002$ | $0.872\pm 0.002$ | $0.029\pm 0.003$ | $0.856\pm 0.004$
learned | 1.0 | $0.764\pm 0.004$ | $0.860\pm 0.002$ | $0.045\pm 0.001$ | $0.946\pm 0.009$
5.0 | $0.780\pm 0.002$ | $0.872\pm 0.002$ | $0.033\pm 0.007$ | $0.863\pm 0.012$
MIMO | False | initial | 1.0 | $0.768\pm 0.005$ | $0.862\pm 0.003$ | $0.050\pm 0.002$ | $0.932\pm 0.017$
5.0 | $0.769\pm 0.001$ | $0.862\pm 0.001$ | $0.057\pm 0.004$ | $0.925\pm 0.005$
learned | 1.0 | $0.768\pm 0.005$ | $0.862\pm 0.003$ | $0.054\pm 0.029$ | $0.949\pm 0.050$
5.0 | $0.769\pm 0.001$ | $0.862\pm 0.001$ | $0.052\pm 0.010$ | $0.930\pm 0.004$
True | initial | 1.0 | $0.767\pm 0.002$ | $0.863\pm 0.002$ | $0.052\pm 0.002$ | $0.937\pm 0.005$
5.0 | $0.769\pm 0.003$ | $0.862\pm 0.001$ | $0.060\pm 0.001$ | $0.931\pm 0.003$
learned | 1.0 | $0.766\pm 0.003$ | $0.862\pm 0.003$ | $0.088\pm 0.024$ | $1.026\pm 0.058$
5.0 | $0.769\pm 0.003$ | $0.862\pm 0.001$ | $0.053\pm 0.011$ | $0.935\pm 0.011$
Mixup | False | initial | 1.0 | $0.767\pm 0.003$ | $0.863\pm 0.002$ | $0.037\pm 0.001$ | $0.921\pm 0.014$
5.0 | $0.778\pm 0.001$ | $0.871\pm 0.001$ | $0.073\pm 0.002$ | $0.900\pm 0.006$
learned | 1.0 | $0.768\pm 0.002$ | $0.863\pm 0.001$ | $0.074\pm 0.005$ | $0.970\pm 0.005$
5.0 | $0.778\pm 0.001$ | $0.871\pm 0.001$ | $0.070\pm 0.006$ | $0.900\pm 0.005$
True | initial | 1.0 | $0.763\pm 0.004$ | $0.861\pm 0.002$ | $0.025\pm 0.003$ | $0.933\pm 0.010$
5.0 | $0.778\pm 0.004$ | $0.869\pm 0.002$ | $0.074\pm 0.002$ | $0.902\pm 0.010$
learned | 1.0 | $0.767\pm 0.003$ | $0.862\pm 0.001$ | $0.072\pm 0.004$ | $0.969\pm 0.006$
5.0 | $0.778\pm 0.004$ | $0.869\pm 0.002$ | $0.066\pm 0.007$ | $0.902\pm 0.009$
OC | False | initial | 1.0 | $0.765\pm 0.004$ | $0.860\pm 0.002$ | $0.054\pm 0.003$ | $0.956\pm 0.016$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.001$ | $0.029\pm 0.001$ | $0.855\pm 0.004$
learned | 1.0 | $0.765\pm 0.004$ | $0.860\pm 0.002$ | $0.044\pm 0.003$ | $0.945\pm 0.015$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.001$ | $0.041\pm 0.000$ | $0.881\pm 0.004$
True | initial | 1.0 | $0.763\pm 0.003$ | $0.860\pm 0.003$ | $0.055\pm 0.002$ | $0.956\pm 0.007$
5.0 | $0.780\pm 0.003$ | $0.872\pm 0.002$ | $0.028\pm 0.004$ | $0.856\pm 0.001$
learned | 1.0 | $0.763\pm 0.003$ | $0.860\pm 0.003$ | $0.046\pm 0.001$ | $0.946\pm 0.006$
5.0 | $0.780\pm 0.003$ | $0.872\pm 0.002$ | $0.042\pm 0.001$ | $0.882\pm 0.001$
RND | False | initial | 1.0 | $0.765\pm 0.001$ | $0.860\pm 0.001$ | $0.053\pm 0.000$ | $0.953\pm 0.001$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.001$ | $0.030\pm 0.003$ | $0.856\pm 0.002$
learned | 1.0 | $0.765\pm 0.001$ | $0.860\pm 0.001$ | $0.044\pm 0.001$ | $0.943\pm 0.002$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.001$ | $0.040\pm 0.001$ | $0.880\pm 0.001$
True | initial | 1.0 | $0.762\pm 0.001$ | $0.859\pm 0.001$ | $0.054\pm 0.002$ | $0.961\pm 0.006$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.002$ | $0.029\pm 0.002$ | $0.859\pm 0.003$
learned | 1.0 | $0.762\pm 0.001$ | $0.859\pm 0.001$ | $0.045\pm 0.002$ | $0.950\pm 0.004$
5.0 | $0.781\pm 0.001$ | $0.871\pm 0.002$ | $0.040\pm 0.000$ | $0.884\pm 0.004$
SoftLabeler | False | initial | 1.0 | $0.763\pm 0.002$ | $0.857\pm 0.001$ | $0.035\pm 0.004$ | $1.001\pm 0.005$
5.0 | $0.781\pm 0.001$ | $0.874\pm 0.002$ | $0.131\pm 0.003$ | $1.003\pm 0.002$
learned | 1.0 | $0.763\pm 0.002$ | $0.857\pm 0.001$ | $0.092\pm 0.002$ | $1.021\pm 0.002$
5.0 | $0.781\pm 0.001$ | $0.874\pm 0.002$ | $0.092\pm 0.001$ | $0.944\pm 0.003$
True | initial | 1.0 | $0.761\pm 0.003$ | $0.858\pm 0.002$ | $0.031\pm 0.004$ | $1.002\pm 0.002$
5.0 | $0.781\pm 0.002$ | $0.873\pm 0.002$ | $0.129\pm 0.003$ | $1.001\pm 0.002$
learned | 1.0 | $0.761\pm 0.003$ | $0.858\pm 0.002$ | $0.094\pm 0.004$ | $1.027\pm 0.004$
5.0 | $0.781\pm 0.002$ | $0.873\pm 0.002$ | $0.092\pm 0.002$ | $0.943\pm 0.006$

Table 3: In-domain results for backbone ResNet50.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.869\pm 0.003$ | $0.941\pm 0.001$ | $0.382\pm 0.012$
5.0 | $0.874\pm 0.001$ | $0.941\pm 0.002$ | $0.375\pm 0.008$
learned | 1.0 | $0.864\pm 0.004$ | $0.942\pm 0.001$ | $0.359\pm 0.009$
5.0 | $0.864\pm 0.000$ | $0.940\pm 0.002$ | $0.366\pm 0.006$
True | initial | 1.0 | $0.858\pm 0.007$ | $0.943\pm 0.001$ | $0.354\pm 0.022$
5.0 | $0.872\pm 0.002$ | $0.941\pm 0.002$ | $0.371\pm 0.007$
learned | 1.0 | $0.860\pm 0.008$ | $0.943\pm 0.001$ | $0.358\pm 0.023$
5.0 | $0.862\pm 0.001$ | $0.940\pm 0.002$ | $0.361\pm 0.009$
MCDropout | False | initial | 1.0 | $0.859\pm 0.004$ | $0.942\pm 0.003$ | $0.357\pm 0.005$
5.0 | $0.872\pm 0.002$ | $0.940\pm 0.001$ | $0.373\pm 0.008$
learned | 1.0 | $0.861\pm 0.004$ | $0.943\pm 0.003$ | $0.359\pm 0.006$
5.0 | $0.862\pm 0.001$ | $0.940\pm 0.001$ | $0.366\pm 0.003$
True | initial | 1.0 | $0.861\pm 0.005$ | $0.942\pm 0.002$ | $0.357\pm 0.015$
5.0 | $0.872\pm 0.002$ | $0.939\pm 0.002$ | $0.373\pm 0.009$
learned | 1.0 | $0.863\pm 0.005$ | $0.942\pm 0.002$ | $0.358\pm 0.017$
5.0 | $0.862\pm 0.002$ | $0.941\pm 0.001$ | $0.363\pm 0.006$
MIMO | False | initial | 1.0 | $0.871\pm 0.002$ | $0.942\pm 0.001$ | $0.403\pm 0.013$
5.0 | $0.858\pm 0.003$ | $0.939\pm 0.002$ | $0.336\pm 0.015$
learned | 1.0 | $0.869\pm 0.007$ | $0.942\pm 0.001$ | $0.400\pm 0.003$
5.0 | $0.846\pm 0.003$ | $0.938\pm 0.002$ | $0.333\pm 0.014$
True | initial | 1.0 | $0.868\pm 0.002$ | $0.942\pm 0.004$ | $0.386\pm 0.005$
5.0 | $0.860\pm 0.002$ | $0.938\pm 0.002$ | $0.340\pm 0.013$
learned | 1.0 | $0.846\pm 0.012$ | $0.943\pm 0.004$ | $0.340\pm 0.028$
5.0 | $0.848\pm 0.002$ | $0.938\pm 0.002$ | $0.334\pm 0.009$
Mixup | False | initial | 1.0 | $0.860\pm 0.008$ | $0.939\pm 0.002$ | $0.364\pm 0.018$
5.0 | $0.870\pm 0.001$ | $0.938\pm 0.003$ | $0.370\pm 0.007$
learned | 1.0 | $0.839\pm 0.009$ | $0.941\pm 0.002$ | $0.332\pm 0.026$
5.0 | $0.852\pm 0.003$ | $0.938\pm 0.004$ | $0.351\pm 0.011$
True | initial | 1.0 | $0.859\pm 0.006$ | $0.942\pm 0.003$ | $0.359\pm 0.020$
5.0 | $0.872\pm 0.001$ | $0.939\pm 0.003$ | $0.374\pm 0.009$
learned | 1.0 | $0.845\pm 0.005$ | $0.942\pm 0.002$ | $0.341\pm 0.011$
5.0 | $0.855\pm 0.002$ | $0.941\pm 0.002$ | $0.354\pm 0.008$
OC | False | initial | 1.0 | $0.860\pm 0.002$ | $0.942\pm 0.001$ | $0.358\pm 0.013$
5.0 | $0.871\pm 0.003$ | $0.941\pm 0.000$ | $0.367\pm 0.006$
learned | 1.0 | $0.862\pm 0.001$ | $0.942\pm 0.001$ | $0.359\pm 0.014$
5.0 | $0.861\pm 0.003$ | $0.942\pm 0.002$ | $0.359\pm 0.009$
True | initial | 1.0 | $0.860\pm 0.006$ | $0.942\pm 0.001$ | $0.355\pm 0.011$
5.0 | $0.871\pm 0.004$ | $0.939\pm 0.001$ | $0.364\pm 0.014$
learned | 1.0 | $0.862\pm 0.006$ | $0.942\pm 0.001$ | $0.358\pm 0.013$
5.0 | $0.860\pm 0.003$ | $0.940\pm 0.001$ | $0.354\pm 0.018$
RND | False | initial | 1.0 | $0.859\pm 0.003$ | $0.941\pm 0.002$ | $0.365\pm 0.003$
5.0 | $0.871\pm 0.001$ | $0.940\pm 0.002$ | $0.371\pm 0.013$
learned | 1.0 | $0.861\pm 0.003$ | $0.941\pm 0.002$ | $0.365\pm 0.005$
5.0 | $0.861\pm 0.002$ | $0.940\pm 0.002$ | $0.363\pm 0.015$
True | initial | 1.0 | $0.858\pm 0.007$ | $0.941\pm 0.004$ | $0.358\pm 0.011$
5.0 | $0.872\pm 0.004$ | $0.940\pm 0.001$ | $0.374\pm 0.015$
learned | 1.0 | $0.860\pm 0.007$ | $0.941\pm 0.004$ | $0.359\pm 0.010$
5.0 | $0.862\pm 0.003$ | $0.939\pm 0.001$ | $0.368\pm 0.016$
SoftLabeler | False | initial | 1.0 | $0.843\pm 0.005$ | $0.941\pm 0.001$ | $0.318\pm 0.026$
5.0 | $0.856\pm 0.002$ | $0.939\pm 0.001$ | $0.335\pm 0.017$
learned | 1.0 | $0.839\pm 0.002$ | $0.942\pm 0.002$ | $0.322\pm 0.017$
5.0 | $0.846\pm 0.001$ | $0.941\pm 0.001$ | $0.333\pm 0.015$
True | initial | 1.0 | $0.839\pm 0.002$ | $0.943\pm 0.002$ | $0.308\pm 0.010$
5.0 | $0.855\pm 0.010$ | $0.941\pm 0.002$ | $0.331\pm 0.032$
learned | 1.0 | $0.835\pm 0.002$ | $0.942\pm 0.001$ | $0.315\pm 0.008$
5.0 | $0.845\pm 0.006$ | $0.941\pm 0.001$ | $0.329\pm 0.015$

Table 4: Out-domain results for measure Entropy and backbone ResNet50.

## References

  * Abdar et al. (2020) Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Abbas Khosravi, U Rajendra Acharya, Vladimir Makarenkov, et al.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  _arXiv_ , 2020.
  * Alvarez-Melis and Jaakkola (2017) David Alvarez-Melis and Tommi S Jaakkola.  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  _arXiv_ , 2017.
  * Ashukha et al. (2020) Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov.  Pitfalls of in-domain uncertainty estimation and ensembling in deep learning.  _arXiv_ , 2020.
  * Begoli et al. (2019) Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov.  The need for uncertainty quantification in machine-assisted medical decision making.  _Nature Machine Intelligence_ , 2019.
  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bottou (2012) Léon Bottou.  Stochastic gradient descent tricks.  In _Neural networks: Tricks of the trade_ , pages 421–436. Springer, 2012.
  * Broomhead and Lowe (1988) David S Broomhead and David Lowe.  Radial basis functions, multi-variable functional interpolation and adaptive networks.  Technical report, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988.
  * Burda et al. (2018) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  _arXiv_ , 2018.
  * Chalapathy and Chawla (2019) Raghavendra Chalapathy and Sanjay Chawla.  Deep learning for anomaly detection: A survey.  _arXiv_ , 2019.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , 2016.
  * Gardner et al. (2018) Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson.  Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration.  _arXiv preprint arXiv:1809.11165_ , 2018.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv_ , 2014.
  * Gulrajani and Lopez-Paz (2020) Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  _arXiv_ , 2020.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _ICML_ , 2017.
  * Havasi et al. (2021) Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran.  Training independent subnetworks for robust prediction.  In _ICLR_ , 2021.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hein et al. (2019) Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf.  Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.  In _CVPR_ , 2019.
  * Henaff et al. (2019) Mikael Henaff, Alfredo Canziani, and Yann LeCun.  Model-predictive policy learning with uncertainty regularization for driving in dense traffic.  _arXiv_ , 2019.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _arXiv_ , 2019.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv_ , 2016.
  * Hendrycks et al. (2021) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271, 2021.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv_ , 2015.
  * Lakshminarayanan et al. (2016) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _NeurIPS_ , 2016.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 2015.
  * Liu et al. (2020) Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan.  Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.  _arXiv_ , 2020.
  * Michelmore et al. (2018) Rhiannon Michelmore, Marta Kwiatkowska, and Yarin Gal.  Evaluating uncertainty quantification in end-to-end autonomous driving control.  _arXiv_ , 2018.
  * Miyato et al. (2018) Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  _arXiv_ , 2018.
  * Mukhoti et al. (2021) Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal.  Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty.  _arXiv_ , 2021.
  * Novak et al. (2018) Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein.  Sensitivity and generalization in neural networks: an empirical study.  _arXiv_ , 2018.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _arXiv_ , 2019.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _arXiv_ , 2019.
  * Platt et al. (1999) John Platt et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in Large Margin Classifiers_ , 1999.
  * Robbins and Monro (1951) Herbert Robbins and Sutton Monro.  A stochastic approximation method.  _The annals of mathematical statistics_ , pages 400–407, 1951.
  * Ruff et al. (2021) Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller.  A unifying review of deep and shallow anomaly detection.  _Proceedings of the IEEE_ , 2021.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _IJCV_ , 2015.
  * Santurkar et al. (2020) Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry.  Breeds: Benchmarks for subpopulation shift.  _arXiv_ , 2020.
  * Settles (2009) Burr Settles.  Active learning literature survey.  2009\.
  * Shannon (1948) Claude Elwood Shannon.  A mathematical theory of communication.  _Bell System Technical Journal_ , 1948.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 2014.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , 2016.
  * Tagasovska and Lopez-Paz (2018) Natasa Tagasovska and David Lopez-Paz.  Single-model uncertainties for deep learning.  _arXiv_ , 2018.
  * Thulasidasan et al. (2019) Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  _arXiv_ , 2019.
  * Ulmer and Cinà (2020) Dennis Ulmer and Giovanni Cinà.  Know your limits: Monotonicity & softmax make neural classifiers overconfident on ood data.  _arXiv_ , 2020.
  * Van Amersfoort et al. (2020) Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal.  Uncertainty estimation using a single deep deterministic neural network.  In _ICML_ , 2020.
  * van Amersfoort et al. (2021) Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal.  Improving deterministic uncertainty estimation in deep learning for classification and regression.  _arXiv_ , 2021.
  * Vapnik (1992) Vladimir Vapnik.  Principles of risk minimization for learning theory.  In _NeurIPS_ , 1992.
  * Wadoux (2019) Alexandre MJ-C Wadoux.  Using deep learning for multivariate mapping of soil with quantified uncertainty.  _Geoderma_ , 2019.
  * Ward Jr (1963) Joe H Ward Jr.  Hierarchical grouping to optimize an objective function.  _Journal of the American statistical association_ , 1963.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv_ , 2017.

##  Appendix A In-domain results

algorithm | spectral | calibration | k | ACC@1 | ACC@5 | ECE | NLL
---|---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.718\pm 0.001$ | $0.821\pm 0.001$ | $0.044\pm 0.001$ | $1.139\pm 0.002$
5.0 | $0.745\pm 0.001$ | $0.843\pm 0.002$ | $0.044\pm 0.001$ | $1.017\pm 0.001$
learned | 1.0 | $0.718\pm 0.001$ | $0.821\pm 0.001$ | $0.050\pm 0.002$ | $1.145\pm 0.003$
5.0 | $0.745\pm 0.001$ | $0.843\pm 0.002$ | $0.049\pm 0.003$ | $1.048\pm 0.004$
True | initial | 1.0 | $0.720\pm 0.001$ | $0.822\pm 0.001$ | $0.046\pm 0.004$ | $1.144\pm 0.004$
5.0 | $0.744\pm 0.001$ | $0.843\pm 0.001$ | $0.045\pm 0.000$ | $1.020\pm 0.003$
learned | 1.0 | $0.720\pm 0.001$ | $0.822\pm 0.001$ | $0.052\pm 0.003$ | $1.151\pm 0.003$
5.0 | $0.744\pm 0.001$ | $0.843\pm 0.001$ | $0.047\pm 0.003$ | $1.048\pm 0.007$
MCDropout | False | initial | 1.0 | $0.720\pm 0.002$ | $0.825\pm 0.003$ | $0.051\pm 0.003$ | $1.142\pm 0.007$
5.0 | $0.747\pm 0.001$ | $0.843\pm 0.001$ | $0.045\pm 0.001$ | $1.017\pm 0.004$
learned | 1.0 | $0.722\pm 0.001$ | $0.824\pm 0.002$ | $0.049\pm 0.005$ | $1.137\pm 0.005$
5.0 | $0.747\pm 0.001$ | $0.843\pm 0.001$ | $0.049\pm 0.004$ | $1.050\pm 0.007$
True | initial | 1.0 | $0.719\pm 0.003$ | $0.822\pm 0.004$ | $0.047\pm 0.004$ | $1.149\pm 0.013$
5.0 | $0.746\pm 0.001$ | $0.842\pm 0.001$ | $0.047\pm 0.002$ | $1.020\pm 0.005$
learned | 1.0 | $0.719\pm 0.003$ | $0.822\pm 0.004$ | $0.051\pm 0.004$ | $1.155\pm 0.014$
5.0 | $0.746\pm 0.001$ | $0.842\pm 0.001$ | $0.047\pm 0.003$ | $1.050\pm 0.003$
MIMO | False | initial | 1.0 | $0.722\pm 0.002$ | $0.825\pm 0.000$ | $0.046\pm 0.002$ | $1.123\pm 0.005$
5.0 | $0.723\pm 0.002$ | $0.823\pm 0.003$ | $0.090\pm 0.002$ | $1.161\pm 0.005$
learned | 1.0 | $0.722\pm 0.002$ | $0.825\pm 0.000$ | $0.050\pm 0.004$ | $1.127\pm 0.007$
5.0 | $0.723\pm 0.002$ | $0.823\pm 0.003$ | $0.047\pm 0.001$ | $1.164\pm 0.005$
True | initial | 1.0 | $0.722\pm 0.003$ | $0.824\pm 0.001$ | $0.047\pm 0.004$ | $1.124\pm 0.000$
5.0 | $0.724\pm 0.001$ | $0.825\pm 0.001$ | $0.090\pm 0.001$ | $1.159\pm 0.003$
learned | 1.0 | $0.722\pm 0.003$ | $0.824\pm 0.001$ | $0.050\pm 0.005$ | $1.127\pm 0.002$
5.0 | $0.724\pm 0.001$ | $0.825\pm 0.001$ | $0.046\pm 0.001$ | $1.163\pm 0.005$
Mixup | False | initial | 1.0 | $0.723\pm 0.002$ | $0.824\pm 0.004$ | $0.033\pm 0.001$ | $1.120\pm 0.007$
5.0 | $0.739\pm 0.003$ | $0.838\pm 0.001$ | $0.083\pm 0.002$ | $1.082\pm 0.001$
learned | 1.0 | $0.703\pm 0.002$ | $0.809\pm 0.002$ | $0.075\pm 0.002$ | $1.246\pm 0.006$
5.0 | $0.739\pm 0.003$ | $0.838\pm 0.001$ | $0.105\pm 0.009$ | $1.193\pm 0.030$
True | initial | 1.0 | $0.722\pm 0.001$ | $0.826\pm 0.001$ | $0.032\pm 0.002$ | $1.117\pm 0.006$
5.0 | $0.740\pm 0.002$ | $0.837\pm 0.001$ | $0.092\pm 0.002$ | $1.091\pm 0.003$
learned | 1.0 | $0.701\pm 0.001$ | $0.804\pm 0.003$ | $0.078\pm 0.000$ | $1.255\pm 0.008$
5.0 | $0.740\pm 0.002$ | $0.837\pm 0.001$ | $0.071\pm 0.003$ | $1.089\pm 0.005$
OC | False | initial | 1.0 | $0.720\pm 0.001$ | $0.823\pm 0.001$ | $0.046\pm 0.002$ | $1.141\pm 0.006$
5.0 | $0.746\pm 0.002$ | $0.842\pm 0.002$ | $0.046\pm 0.002$ | $1.019\pm 0.001$
learned | 1.0 | $0.720\pm 0.001$ | $0.823\pm 0.001$ | $0.052\pm 0.002$ | $1.147\pm 0.006$
5.0 | $0.746\pm 0.002$ | $0.842\pm 0.002$ | $0.049\pm 0.002$ | $1.051\pm 0.005$
True | initial | 1.0 | $0.721\pm 0.002$ | $0.823\pm 0.003$ | $0.048\pm 0.003$ | $1.141\pm 0.003$
5.0 | $0.746\pm 0.001$ | $0.843\pm 0.001$ | $0.045\pm 0.003$ | $1.016\pm 0.002$
learned | 1.0 | $0.721\pm 0.002$ | $0.823\pm 0.003$ | $0.054\pm 0.004$ | $1.149\pm 0.005$
5.0 | $0.746\pm 0.001$ | $0.843\pm 0.001$ | $0.049\pm 0.004$ | $1.048\pm 0.006$
RND | False | initial | 1.0 | $0.720\pm 0.001$ | $0.821\pm 0.002$ | $0.045\pm 0.003$ | $1.146\pm 0.009$
5.0 | $0.745\pm 0.001$ | $0.842\pm 0.001$ | $0.046\pm 0.002$ | $1.019\pm 0.004$
learned | 1.0 | $0.720\pm 0.001$ | $0.821\pm 0.002$ | $0.050\pm 0.002$ | $1.152\pm 0.007$
5.0 | $0.745\pm 0.001$ | $0.842\pm 0.001$ | $0.048\pm 0.004$ | $1.049\pm 0.001$
True | initial | 1.0 | $0.723\pm 0.004$ | $0.826\pm 0.002$ | $0.052\pm 0.005$ | $1.138\pm 0.010$
5.0 | $0.744\pm 0.002$ | $0.843\pm 0.002$ | $0.045\pm 0.003$ | $1.018\pm 0.006$
learned | 1.0 | $0.718\pm 0.002$ | $0.821\pm 0.001$ | $0.050\pm 0.003$ | $1.148\pm 0.003$
5.0 | $0.744\pm 0.002$ | $0.843\pm 0.002$ | $0.050\pm 0.005$ | $1.049\pm 0.009$
SoftLabeler | False | initial | 1.0 | $0.719\pm 0.003$ | $0.824\pm 0.001$ | $0.033\pm 0.002$ | $1.180\pm 0.005$
5.0 | $0.743\pm 0.002$ | $0.842\pm 0.002$ | $0.137\pm 0.001$ | $1.179\pm 0.003$
learned | 1.0 | $0.719\pm 0.003$ | $0.824\pm 0.001$ | $0.098\pm 0.004$ | $1.214\pm 0.006$
5.0 | $0.743\pm 0.002$ | $0.842\pm 0.002$ | $0.102\pm 0.003$ | $1.128\pm 0.008$
True | initial | 1.0 | $0.720\pm 0.003$ | $0.824\pm 0.001$ | $0.036\pm 0.004$ | $1.183\pm 0.007$
5.0 | $0.744\pm 0.003$ | $0.843\pm 0.001$ | $0.138\pm 0.003$ | $1.179\pm 0.004$
learned | 1.0 | $0.720\pm 0.003$ | $0.824\pm 0.001$ | $0.097\pm 0.004$ | $1.216\pm 0.013$
5.0 | $0.744\pm 0.003$ | $0.843\pm 0.001$ | $0.103\pm 0.005$ | $1.132\pm 0.008$

Table 5: In-domain results for backbone ResNet18.

##  Appendix B Out-domain results

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.644\pm 0.017$ | $0.949\pm 0.001$ | $0.093\pm 0.016$
5.0 | $0.668\pm 0.011$ | $0.966\pm 0.030$ | $0.076\pm 0.066$
learned | 1.0 | $0.649\pm 0.017$ | $0.952\pm 0.001$ | $0.090\pm 0.017$
5.0 | $0.658\pm 0.009$ | $0.949\pm 0.001$ | $0.105\pm 0.011$
True | initial | 1.0 | $0.648\pm 0.010$ | $0.949\pm 0.000$ | $0.097\pm 0.004$
5.0 | $0.664\pm 0.005$ | $0.948\pm 0.003$ | $0.105\pm 0.006$
learned | 1.0 | $0.649\pm 0.010$ | $0.950\pm 0.001$ | $0.097\pm 0.003$
5.0 | $0.657\pm 0.006$ | $0.951\pm 0.002$ | $0.100\pm 0.004$
MCDropout | False | initial | 1.0 | $0.658\pm 0.008$ | $0.965\pm 0.030$ | $0.080\pm 0.070$
5.0 | $0.668\pm 0.003$ | $0.966\pm 0.030$ | $0.073\pm 0.063$
learned | 1.0 | $0.654\pm 0.010$ | $0.949\pm 0.002$ | $0.101\pm 0.008$
5.0 | $0.660\pm 0.002$ | $0.950\pm 0.001$ | $0.104\pm 0.002$
True | initial | 1.0 | $0.655\pm 0.024$ | $0.953\pm 0.003$ | $0.095\pm 0.015$
5.0 | $0.673\pm 0.006$ | $0.949\pm 0.003$ | $0.107\pm 0.007$
learned | 1.0 | $0.656\pm 0.027$ | $0.951\pm 0.003$ | $0.093\pm 0.014$
5.0 | $0.668\pm 0.007$ | $0.951\pm 0.000$ | $0.105\pm 0.003$
MIMO | False | initial | 1.0 | $0.660\pm 0.034$ | $0.950\pm 0.001$ | $0.101\pm 0.026$
5.0 | $0.673\pm 0.013$ | $0.948\pm 0.002$ | $0.116\pm 0.014$
learned | 1.0 | $0.661\pm 0.033$ | $0.951\pm 0.001$ | $0.100\pm 0.025$
5.0 | $0.660\pm 0.012$ | $0.949\pm 0.000$ | $0.112\pm 0.010$
True | initial | 1.0 | $0.655\pm 0.016$ | $0.966\pm 0.029$ | $0.064\pm 0.058$
5.0 | $0.676\pm 0.004$ | $0.948\pm 0.003$ | $0.117\pm 0.002$
learned | 1.0 | $0.652\pm 0.015$ | $0.947\pm 0.001$ | $0.101\pm 0.014$
5.0 | $0.666\pm 0.003$ | $0.948\pm 0.001$ | $0.114\pm 0.005$
Mixup | False | initial | 1.0 | $0.683\pm 0.007$ | $0.966\pm 0.029$ | $0.080\pm 0.069$
5.0 | $0.691\pm 0.001$ | $0.964\pm 0.031$ | $0.086\pm 0.074$
learned | 1.0 | $0.669\pm 0.020$ | $0.951\pm 0.002$ | $0.136\pm 0.019$
5.0 | $0.676\pm 0.009$ | $0.949\pm 0.001$ | $0.122\pm 0.007$
True | initial | 1.0 | $0.688\pm 0.011$ | $0.951\pm 0.003$ | $0.119\pm 0.009$
5.0 | $0.689\pm 0.010$ | $0.947\pm 0.002$ | $0.126\pm 0.014$
learned | 1.0 | $0.648\pm 0.021$ | $0.950\pm 0.003$ | $0.114\pm 0.022$
5.0 | $0.679\pm 0.006$ | $0.948\pm 0.003$ | $0.121\pm 0.010$
OC | False | initial | 1.0 | $0.641\pm 0.017$ | $0.949\pm 0.003$ | $0.092\pm 0.012$
5.0 | $0.667\pm 0.003$ | $0.950\pm 0.001$ | $0.104\pm 0.003$
learned | 1.0 | $0.640\pm 0.017$ | $0.950\pm 0.003$ | $0.089\pm 0.013$
5.0 | $0.662\pm 0.000$ | $0.950\pm 0.003$ | $0.106\pm 0.005$
True | initial | 1.0 | $0.649\pm 0.030$ | $0.947\pm 0.003$ | $0.099\pm 0.011$
5.0 | $0.670\pm 0.011$ | $0.950\pm 0.001$ | $0.106\pm 0.009$
learned | 1.0 | $0.653\pm 0.029$ | $0.950\pm 0.002$ | $0.100\pm 0.012$
5.0 | $0.659\pm 0.013$ | $0.949\pm 0.002$ | $0.106\pm 0.006$
RND | False | initial | 1.0 | $0.650\pm 0.030$ | $0.951\pm 0.002$ | $0.094\pm 0.023$
5.0 | $0.671\pm 0.012$ | $0.950\pm 0.003$ | $0.107\pm 0.012$
learned | 1.0 | $0.655\pm 0.031$ | $0.967\pm 0.029$ | $0.074\pm 0.066$
5.0 | $0.661\pm 0.011$ | $0.950\pm 0.000$ | $0.108\pm 0.007$
True | initial | 5.0 | $0.664\pm 0.007$ | $0.948\pm 0.002$ | $0.102\pm 0.006$
learned | 1.0 | $0.645\pm 0.017$ | $0.965\pm 0.031$ | $0.062\pm 0.054$
5.0 | $0.650\pm 0.007$ | $0.948\pm 0.001$ | $0.101\pm 0.004$
SoftLabeler | False | initial | 1.0 | $0.630\pm 0.020$ | $0.951\pm 0.003$ | $0.089\pm 0.015$
5.0 | $0.644\pm 0.006$ | $0.968\pm 0.028$ | $0.062\pm 0.054$
learned | 1.0 | $0.633\pm 0.021$ | $0.950\pm 0.003$ | $0.090\pm 0.016$
5.0 | $0.646\pm 0.009$ | $0.950\pm 0.001$ | $0.097\pm 0.007$
True | initial | 1.0 | $0.642\pm 0.013$ | $0.950\pm 0.004$ | $0.089\pm 0.005$
5.0 | $0.656\pm 0.010$ | $0.947\pm 0.002$ | $0.103\pm 0.009$
learned | 1.0 | $0.641\pm 0.011$ | $0.950\pm 0.001$ | $0.093\pm 0.004$
5.0 | $0.658\pm 0.008$ | $0.949\pm 0.001$ | $0.105\pm 0.004$

Table 6: Out-domain results for measure Augmentations and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.663\pm 0.004$ | $0.951\pm 0.001$ | $0.106\pm 0.010$
5.0 | $0.682\pm 0.004$ | $0.966\pm 0.030$ | $0.079\pm 0.068$
learned | 1.0 | $0.657\pm 0.010$ | $0.951\pm 0.002$ | $0.105\pm 0.017$
5.0 | $0.662\pm 0.004$ | $0.948\pm 0.002$ | $0.106\pm 0.003$
True | initial | 1.0 | $0.655\pm 0.002$ | $0.951\pm 0.001$ | $0.105\pm 0.006$
5.0 | $0.662\pm 0.007$ | $0.948\pm 0.002$ | $0.101\pm 0.011$
learned | 1.0 | $0.657\pm 0.001$ | $0.947\pm 0.003$ | $0.113\pm 0.009$
5.0 | $0.652\pm 0.007$ | $0.948\pm 0.002$ | $0.099\pm 0.010$
MCDropout | False | initial | 1.0 | $0.635\pm 0.042$ | $0.950\pm 0.002$ | $0.089\pm 0.023$
5.0 | $0.674\pm 0.010$ | $0.967\pm 0.029$ | $0.066\pm 0.057$
learned | 1.0 | $0.638\pm 0.040$ | $0.951\pm 0.001$ | $0.089\pm 0.017$
5.0 | $0.654\pm 0.011$ | $0.949\pm 0.001$ | $0.096\pm 0.004$
True | initial | 1.0 | $0.652\pm 0.020$ | $0.949\pm 0.002$ | $0.102\pm 0.009$
5.0 | $0.666\pm 0.013$ | $0.947\pm 0.001$ | $0.106\pm 0.010$
learned | 1.0 | $0.656\pm 0.021$ | $0.949\pm 0.001$ | $0.107\pm 0.012$
5.0 | $0.659\pm 0.013$ | $0.949\pm 0.001$ | $0.103\pm 0.012$
MIMO | False | initial | 1.0 | $0.664\pm 0.016$ | $0.950\pm 0.001$ | $0.110\pm 0.015$
5.0 | $0.700\pm 0.006$ | $0.949\pm 0.001$ | $0.121\pm 0.005$
learned | 1.0 | $0.665\pm 0.016$ | $0.950\pm 0.001$ | $0.113\pm 0.013$
5.0 | $0.679\pm 0.006$ | $0.950\pm 0.001$ | $0.114\pm 0.005$
True | initial | 1.0 | $0.668\pm 0.020$ | $0.950\pm 0.004$ | $0.107\pm 0.017$
5.0 | $0.686\pm 0.018$ | $0.949\pm 0.002$ | $0.111\pm 0.013$
learned | 1.0 | $0.643\pm 0.027$ | $0.967\pm 0.029$ | $0.061\pm 0.054$
5.0 | $0.671\pm 0.015$ | $0.946\pm 0.001$ | $0.109\pm 0.010$
Mixup | False | initial | 1.0 | $0.672\pm 0.021$ | $0.945\pm 0.004$ | $0.115\pm 0.012$
5.0 | $0.700\pm 0.012$ | $0.965\pm 0.031$ | $0.084\pm 0.073$
learned | 1.0 | $0.700\pm 0.027$ | $0.943\pm 0.001$ | $0.151\pm 0.031$
5.0 | $0.682\pm 0.010$ | $0.948\pm 0.000$ | $0.118\pm 0.007$
True | initial | 1.0 | $0.682\pm 0.018$ | $0.946\pm 0.003$ | $0.124\pm 0.029$
5.0 | $0.693\pm 0.003$ | $0.945\pm 0.002$ | $0.123\pm 0.009$
learned | 1.0 | $0.701\pm 0.024$ | $0.963\pm 0.032$ | $0.117\pm 0.101$
5.0 | $0.682\pm 0.002$ | $0.946\pm 0.003$ | $0.118\pm 0.010$
OC | False | initial | 1.0 | $0.645\pm 0.025$ | $0.951\pm 0.001$ | $0.096\pm 0.014$
5.0 | $0.664\pm 0.022$ | $0.949\pm 0.002$ | $0.102\pm 0.012$
learned | 1.0 | $0.651\pm 0.025$ | $0.949\pm 0.002$ | $0.104\pm 0.015$
5.0 | $0.654\pm 0.017$ | $0.948\pm 0.004$ | $0.101\pm 0.012$
True | initial | 1.0 | $0.666\pm 0.010$ | $0.950\pm 0.002$ | $0.108\pm 0.009$
5.0 | $0.673\pm 0.011$ | $0.949\pm 0.003$ | $0.107\pm 0.012$
learned | 1.0 | $0.666\pm 0.009$ | $0.967\pm 0.029$ | $0.074\pm 0.064$
5.0 | $0.663\pm 0.011$ | $0.950\pm 0.004$ | $0.106\pm 0.013$
RND | False | initial | 1.0 | $0.653\pm 0.021$ | $0.966\pm 0.030$ | $0.064\pm 0.057$
5.0 | $0.675\pm 0.006$ | $0.966\pm 0.030$ | $0.074\pm 0.064$
learned | 1.0 | $0.661\pm 0.020$ | $0.949\pm 0.001$ | $0.117\pm 0.026$
5.0 | $0.664\pm 0.005$ | $0.949\pm 0.002$ | $0.110\pm 0.010$
True | initial | 1.0 | $0.655\pm 0.012$ | $0.948\pm 0.001$ | $0.104\pm 0.009$
5.0 | $0.670\pm 0.013$ | $0.948\pm 0.001$ | $0.109\pm 0.006$
learned | 1.0 | $0.655\pm 0.013$ | $0.951\pm 0.002$ | $0.102\pm 0.008$
5.0 | $0.660\pm 0.013$ | $0.949\pm 0.002$ | $0.102\pm 0.004$
SoftLabeler | False | initial | 1.0 | $0.630\pm 0.007$ | $0.954\pm 0.003$ | $0.083\pm 0.008$
5.0 | $0.649\pm 0.007$ | $0.963\pm 0.032$ | $0.066\pm 0.057$
learned | 1.0 | $0.625\pm 0.002$ | $0.950\pm 0.005$ | $0.089\pm 0.009$
5.0 | $0.647\pm 0.007$ | $0.946\pm 0.001$ | $0.096\pm 0.004$
True | initial | 1.0 | $0.635\pm 0.044$ | $0.950\pm 0.002$ | $0.086\pm 0.021$
5.0 | $0.644\pm 0.020$ | $0.948\pm 0.002$ | $0.087\pm 0.014$
learned | 1.0 | $0.624\pm 0.043$ | $0.951\pm 0.002$ | $0.086\pm 0.021$
5.0 | $0.632\pm 0.018$ | $0.982\pm 0.031$ | $0.034\pm 0.059$

Table 7: Out-domain results for measure Augmentations and backbone ResNet50.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.811\pm 0.009$ | $0.941\pm 0.001$ | $0.258\pm 0.011$
5.0 | $0.831\pm 0.007$ | $0.939\pm 0.002$ | $0.273\pm 0.012$
learned | 1.0 | $0.810\pm 0.009$ | $0.941\pm 0.001$ | $0.257\pm 0.011$
5.0 | $0.822\pm 0.006$ | $0.937\pm 0.002$ | $0.274\pm 0.012$
True | initial | 1.0 | $0.809\pm 0.006$ | $0.941\pm 0.001$ | $0.244\pm 0.008$
5.0 | $0.830\pm 0.006$ | $0.938\pm 0.001$ | $0.267\pm 0.013$
learned | 1.0 | $0.808\pm 0.005$ | $0.941\pm 0.001$ | $0.245\pm 0.009$
5.0 | $0.822\pm 0.006$ | $0.937\pm 0.002$ | $0.271\pm 0.011$
MCDropout | False | initial | 1.0 | $0.805\pm 0.008$ | $0.941\pm 0.002$ | $0.250\pm 0.014$
5.0 | $0.831\pm 0.004$ | $0.940\pm 0.002$ | $0.269\pm 0.012$
learned | 1.0 | $0.811\pm 0.006$ | $0.939\pm 0.001$ | $0.260\pm 0.011$
5.0 | $0.822\pm 0.004$ | $0.939\pm 0.001$ | $0.269\pm 0.015$
True | initial | 1.0 | $0.810\pm 0.004$ | $0.940\pm 0.004$ | $0.251\pm 0.008$
5.0 | $0.833\pm 0.004$ | $0.940\pm 0.002$ | $0.273\pm 0.007$
learned | 1.0 | $0.809\pm 0.004$ | $0.940\pm 0.004$ | $0.252\pm 0.007$
5.0 | $0.824\pm 0.005$ | $0.939\pm 0.002$ | $0.276\pm 0.010$
MIMO | False | initial | 1.0 | $0.816\pm 0.013$ | $0.939\pm 0.002$ | $0.259\pm 0.023$
5.0 | $0.822\pm 0.005$ | $0.939\pm 0.001$ | $0.247\pm 0.013$
learned | 1.0 | $0.816\pm 0.013$ | $0.939\pm 0.002$ | $0.259\pm 0.023$
5.0 | $0.814\pm 0.003$ | $0.939\pm 0.001$ | $0.250\pm 0.012$
True | initial | 1.0 | $0.813\pm 0.009$ | $0.940\pm 0.003$ | $0.263\pm 0.017$
5.0 | $0.822\pm 0.006$ | $0.940\pm 0.002$ | $0.245\pm 0.012$
learned | 1.0 | $0.813\pm 0.009$ | $0.940\pm 0.003$ | $0.263\pm 0.017$
5.0 | $0.813\pm 0.005$ | $0.940\pm 0.001$ | $0.243\pm 0.013$
Mixup | False | initial | 1.0 | $0.806\pm 0.011$ | $0.941\pm 0.000$ | $0.243\pm 0.013$
5.0 | $0.820\pm 0.008$ | $0.938\pm 0.001$ | $0.252\pm 0.011$
learned | 1.0 | $0.783\pm 0.010$ | $0.938\pm 0.002$ | $0.234\pm 0.004$
5.0 | $0.803\pm 0.004$ | $0.938\pm 0.002$ | $0.253\pm 0.009$
True | initial | 1.0 | $0.808\pm 0.005$ | $0.941\pm 0.002$ | $0.249\pm 0.009$
5.0 | $0.820\pm 0.004$ | $0.939\pm 0.002$ | $0.253\pm 0.006$
learned | 1.0 | $0.787\pm 0.009$ | $0.940\pm 0.000$ | $0.232\pm 0.014$
5.0 | $0.811\pm 0.002$ | $0.938\pm 0.001$ | $0.262\pm 0.009$
OC | False | initial | 1.0 | $0.811\pm 0.003$ | $0.940\pm 0.001$ | $0.257\pm 0.009$
5.0 | $0.830\pm 0.005$ | $0.939\pm 0.001$ | $0.268\pm 0.009$
learned | 1.0 | $0.810\pm 0.003$ | $0.941\pm 0.001$ | $0.257\pm 0.009$
5.0 | $0.821\pm 0.004$ | $0.938\pm 0.001$ | $0.274\pm 0.010$
True | initial | 1.0 | $0.808\pm 0.003$ | $0.941\pm 0.002$ | $0.253\pm 0.005$
5.0 | $0.830\pm 0.001$ | $0.939\pm 0.002$ | $0.270\pm 0.010$
learned | 1.0 | $0.807\pm 0.003$ | $0.941\pm 0.003$ | $0.254\pm 0.005$
5.0 | $0.822\pm 0.001$ | $0.938\pm 0.003$ | $0.272\pm 0.011$
RND | False | initial | 1.0 | $0.813\pm 0.002$ | $0.940\pm 0.000$ | $0.248\pm 0.011$
5.0 | $0.832\pm 0.001$ | $0.941\pm 0.002$ | $0.265\pm 0.012$
learned | 1.0 | $0.812\pm 0.002$ | $0.940\pm 0.000$ | $0.248\pm 0.011$
5.0 | $0.824\pm 0.001$ | $0.941\pm 0.001$ | $0.267\pm 0.008$
True | initial | 1.0 | $0.817\pm 0.006$ | $0.939\pm 0.002$ | $0.267\pm 0.017$
5.0 | $0.832\pm 0.003$ | $0.938\pm 0.003$ | $0.271\pm 0.012$
learned | 1.0 | $0.813\pm 0.004$ | $0.939\pm 0.001$ | $0.254\pm 0.007$
5.0 | $0.824\pm 0.001$ | $0.938\pm 0.002$ | $0.275\pm 0.007$
SoftLabeler | False | initial | 1.0 | $0.786\pm 0.021$ | $0.941\pm 0.001$ | $0.219\pm 0.022$
5.0 | $0.802\pm 0.004$ | $0.939\pm 0.002$ | $0.236\pm 0.007$
learned | 1.0 | $0.789\pm 0.019$ | $0.940\pm 0.002$ | $0.231\pm 0.018$
5.0 | $0.799\pm 0.002$ | $0.938\pm 0.001$ | $0.249\pm 0.003$
True | initial | 1.0 | $0.787\pm 0.007$ | $0.941\pm 0.001$ | $0.212\pm 0.015$
5.0 | $0.809\pm 0.008$ | $0.940\pm 0.001$ | $0.235\pm 0.011$
learned | 1.0 | $0.790\pm 0.003$ | $0.940\pm 0.002$ | $0.224\pm 0.011$
5.0 | $0.805\pm 0.004$ | $0.939\pm 0.001$ | $0.247\pm 0.002$

Table 8: Out-domain results for measure Entropy and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.775\pm 0.007$ | $0.944\pm 0.002$ | $0.177\pm 0.010$
5.0 | $0.795\pm 0.004$ | $0.942\pm 0.001$ | $0.192\pm 0.008$
learned | 1.0 | $0.774\pm 0.007$ | $0.944\pm 0.002$ | $0.176\pm 0.010$
5.0 | $0.785\pm 0.004$ | $0.942\pm 0.002$ | $0.178\pm 0.008$
True | initial | 1.0 | $0.778\pm 0.003$ | $0.944\pm 0.003$ | $0.172\pm 0.011$
5.0 | $0.794\pm 0.004$ | $0.939\pm 0.001$ | $0.186\pm 0.009$
learned | 1.0 | $0.778\pm 0.002$ | $0.944\pm 0.003$ | $0.171\pm 0.011$
5.0 | $0.784\pm 0.003$ | $0.939\pm 0.001$ | $0.174\pm 0.008$
MCDropout | False | initial | 1.0 | $0.774\pm 0.007$ | $0.944\pm 0.001$ | $0.172\pm 0.007$
5.0 | $0.796\pm 0.003$ | $0.944\pm 0.002$ | $0.183\pm 0.010$
learned | 1.0 | $0.777\pm 0.004$ | $0.945\pm 0.002$ | $0.171\pm 0.008$
5.0 | $0.786\pm 0.003$ | $0.944\pm 0.001$ | $0.171\pm 0.009$
True | initial | 1.0 | $0.779\pm 0.004$ | $0.943\pm 0.002$ | $0.174\pm 0.007$
5.0 | $0.796\pm 0.004$ | $0.941\pm 0.002$ | $0.187\pm 0.007$
learned | 1.0 | $0.778\pm 0.004$ | $0.943\pm 0.002$ | $0.173\pm 0.007$
5.0 | $0.786\pm 0.004$ | $0.942\pm 0.002$ | $0.172\pm 0.006$
MIMO | False | initial | 1.0 | $0.783\pm 0.007$ | $0.944\pm 0.001$ | $0.179\pm 0.010$
5.0 | $0.788\pm 0.002$ | $0.943\pm 0.001$ | $0.179\pm 0.004$
learned | 1.0 | $0.782\pm 0.007$ | $0.944\pm 0.001$ | $0.178\pm 0.010$
5.0 | $0.775\pm 0.002$ | $0.944\pm 0.001$ | $0.166\pm 0.004$
True | initial | 1.0 | $0.782\pm 0.006$ | $0.944\pm 0.002$ | $0.176\pm 0.008$
5.0 | $0.788\pm 0.004$ | $0.943\pm 0.001$ | $0.175\pm 0.006$
learned | 1.0 | $0.782\pm 0.006$ | $0.944\pm 0.002$ | $0.175\pm 0.007$
5.0 | $0.775\pm 0.004$ | $0.944\pm 0.001$ | $0.162\pm 0.007$
Mixup | False | initial | 1.0 | $0.783\pm 0.007$ | $0.947\pm 0.001$ | $0.172\pm 0.009$
5.0 | $0.790\pm 0.004$ | $0.942\pm 0.002$ | $0.185\pm 0.009$
learned | 1.0 | $0.750\pm 0.009$ | $0.944\pm 0.003$ | $0.151\pm 0.007$
5.0 | $0.772\pm 0.001$ | $0.943\pm 0.002$ | $0.159\pm 0.004$
True | initial | 1.0 | $0.779\pm 0.001$ | $0.944\pm 0.002$ | $0.181\pm 0.003$
5.0 | $0.792\pm 0.001$ | $0.942\pm 0.003$ | $0.184\pm 0.005$
learned | 1.0 | $0.755\pm 0.006$ | $0.942\pm 0.002$ | $0.157\pm 0.004$
5.0 | $0.778\pm 0.002$ | $0.943\pm 0.002$ | $0.167\pm 0.007$
OC | False | initial | 1.0 | $0.778\pm 0.003$ | $0.946\pm 0.002$ | $0.174\pm 0.003$
5.0 | $0.794\pm 0.003$ | $0.941\pm 0.001$ | $0.184\pm 0.001$
learned | 1.0 | $0.777\pm 0.003$ | $0.946\pm 0.002$ | $0.174\pm 0.004$
5.0 | $0.784\pm 0.002$ | $0.943\pm 0.001$ | $0.172\pm 0.002$
True | initial | 1.0 | $0.778\pm 0.003$ | $0.942\pm 0.002$ | $0.177\pm 0.002$
5.0 | $0.796\pm 0.002$ | $0.945\pm 0.001$ | $0.182\pm 0.003$
learned | 1.0 | $0.777\pm 0.003$ | $0.942\pm 0.003$ | $0.175\pm 0.003$
5.0 | $0.787\pm 0.003$ | $0.945\pm 0.001$ | $0.171\pm 0.004$
RND | False | initial | 1.0 | $0.780\pm 0.005$ | $0.944\pm 0.002$ | $0.174\pm 0.004$
5.0 | $0.797\pm 0.002$ | $0.942\pm 0.001$ | $0.188\pm 0.003$
learned | 1.0 | $0.780\pm 0.005$ | $0.944\pm 0.002$ | $0.173\pm 0.004$
5.0 | $0.788\pm 0.002$ | $0.943\pm 0.002$ | $0.174\pm 0.005$
True | initial | 1.0 | $0.785\pm 0.004$ | $0.945\pm 0.001$ | $0.179\pm 0.009$
5.0 | $0.797\pm 0.002$ | $0.943\pm 0.002$ | $0.189\pm 0.004$
learned | 1.0 | $0.779\pm 0.004$ | $0.943\pm 0.002$ | $0.171\pm 0.011$
5.0 | $0.787\pm 0.003$ | $0.943\pm 0.003$ | $0.177\pm 0.003$
SoftLabeler | False | initial | 1.0 | $0.771\pm 0.018$ | $0.944\pm 0.002$ | $0.175\pm 0.012$
5.0 | $0.786\pm 0.000$ | $0.942\pm 0.003$ | $0.183\pm 0.004$
learned | 1.0 | $0.764\pm 0.017$ | $0.944\pm 0.001$ | $0.165\pm 0.012$
5.0 | $0.771\pm 0.003$ | $0.944\pm 0.004$ | $0.162\pm 0.003$
True | initial | 1.0 | $0.772\pm 0.003$ | $0.943\pm 0.001$ | $0.170\pm 0.002$
5.0 | $0.791\pm 0.006$ | $0.944\pm 0.001$ | $0.181\pm 0.003$
learned | 1.0 | $0.765\pm 0.004$ | $0.944\pm 0.001$ | $0.160\pm 0.001$
5.0 | $0.775\pm 0.004$ | $0.944\pm 0.001$ | $0.160\pm 0.004$

Table 9: Out-domain results for measure Gap and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.822\pm 0.001$ | $0.945\pm 0.002$ | $0.223\pm 0.000$
5.0 | $0.829\pm 0.002$ | $0.944\pm 0.001$ | $0.225\pm 0.005$
learned | 1.0 | $0.819\pm 0.003$ | $0.945\pm 0.001$ | $0.216\pm 0.006$
5.0 | $0.820\pm 0.003$ | $0.944\pm 0.002$ | $0.209\pm 0.007$
True | initial | 1.0 | $0.815\pm 0.004$ | $0.944\pm 0.001$ | $0.214\pm 0.005$
5.0 | $0.827\pm 0.002$ | $0.942\pm 0.003$ | $0.223\pm 0.005$
learned | 1.0 | $0.817\pm 0.005$ | $0.944\pm 0.001$ | $0.218\pm 0.006$
5.0 | $0.818\pm 0.003$ | $0.943\pm 0.003$ | $0.207\pm 0.004$
MCDropout | False | initial | 1.0 | $0.817\pm 0.004$ | $0.945\pm 0.003$ | $0.218\pm 0.003$
5.0 | $0.827\pm 0.003$ | $0.943\pm 0.002$ | $0.224\pm 0.008$
learned | 1.0 | $0.818\pm 0.005$ | $0.945\pm 0.003$ | $0.221\pm 0.003$
5.0 | $0.818\pm 0.004$ | $0.943\pm 0.002$ | $0.208\pm 0.009$
True | initial | 1.0 | $0.817\pm 0.005$ | $0.946\pm 0.004$ | $0.212\pm 0.004$
5.0 | $0.826\pm 0.002$ | $0.941\pm 0.003$ | $0.226\pm 0.004$
learned | 1.0 | $0.818\pm 0.005$ | $0.946\pm 0.004$ | $0.215\pm 0.004$
5.0 | $0.817\pm 0.002$ | $0.940\pm 0.003$ | $0.210\pm 0.003$
MIMO | False | initial | 1.0 | $0.822\pm 0.004$ | $0.944\pm 0.001$ | $0.229\pm 0.010$
5.0 | $0.818\pm 0.003$ | $0.941\pm 0.003$ | $0.217\pm 0.005$
learned | 1.0 | $0.822\pm 0.006$ | $0.944\pm 0.001$ | $0.226\pm 0.006$
5.0 | $0.806\pm 0.003$ | $0.942\pm 0.002$ | $0.197\pm 0.006$
True | initial | 1.0 | $0.820\pm 0.002$ | $0.944\pm 0.002$ | $0.223\pm 0.006$
5.0 | $0.819\pm 0.003$ | $0.942\pm 0.004$ | $0.217\pm 0.007$
learned | 1.0 | $0.809\pm 0.010$ | $0.945\pm 0.003$ | $0.198\pm 0.014$
5.0 | $0.806\pm 0.003$ | $0.943\pm 0.004$ | $0.198\pm 0.006$
Mixup | False | initial | 1.0 | $0.819\pm 0.004$ | $0.944\pm 0.003$ | $0.221\pm 0.010$
5.0 | $0.828\pm 0.002$ | $0.943\pm 0.002$ | $0.230\pm 0.005$
learned | 1.0 | $0.802\pm 0.007$ | $0.946\pm 0.003$ | $0.194\pm 0.011$
5.0 | $0.811\pm 0.003$ | $0.944\pm 0.002$ | $0.201\pm 0.007$
True | initial | 1.0 | $0.818\pm 0.003$ | $0.947\pm 0.001$ | $0.219\pm 0.003$
5.0 | $0.830\pm 0.001$ | $0.945\pm 0.003$ | $0.231\pm 0.008$
learned | 1.0 | $0.808\pm 0.006$ | $0.945\pm 0.001$ | $0.202\pm 0.007$
5.0 | $0.813\pm 0.001$ | $0.946\pm 0.003$ | $0.201\pm 0.009$
OC | False | initial | 1.0 | $0.818\pm 0.002$ | $0.945\pm 0.003$ | $0.217\pm 0.008$
5.0 | $0.827\pm 0.003$ | $0.945\pm 0.001$ | $0.220\pm 0.002$
learned | 1.0 | $0.819\pm 0.002$ | $0.945\pm 0.003$ | $0.220\pm 0.007$
5.0 | $0.818\pm 0.003$ | $0.945\pm 0.001$ | $0.204\pm 0.002$
True | initial | 1.0 | $0.816\pm 0.001$ | $0.945\pm 0.004$ | $0.211\pm 0.010$
5.0 | $0.825\pm 0.004$ | $0.942\pm 0.002$ | $0.224\pm 0.013$
learned | 1.0 | $0.818\pm 0.002$ | $0.945\pm 0.004$ | $0.214\pm 0.011$
5.0 | $0.816\pm 0.004$ | $0.943\pm 0.001$ | $0.208\pm 0.011$
RND | False | initial | 1.0 | $0.814\pm 0.002$ | $0.944\pm 0.002$ | $0.216\pm 0.007$
5.0 | $0.826\pm 0.001$ | $0.942\pm 0.001$ | $0.225\pm 0.003$
learned | 1.0 | $0.816\pm 0.002$ | $0.944\pm 0.002$ | $0.220\pm 0.008$
5.0 | $0.817\pm 0.002$ | $0.943\pm 0.002$ | $0.208\pm 0.002$
True | initial | 1.0 | $0.813\pm 0.005$ | $0.943\pm 0.003$ | $0.214\pm 0.008$
5.0 | $0.827\pm 0.003$ | $0.943\pm 0.004$ | $0.231\pm 0.009$
learned | 1.0 | $0.815\pm 0.005$ | $0.943\pm 0.003$ | $0.218\pm 0.008$
5.0 | $0.818\pm 0.003$ | $0.943\pm 0.003$ | $0.214\pm 0.009$
SoftLabeler | False | initial | 1.0 | $0.814\pm 0.002$ | $0.944\pm 0.002$ | $0.218\pm 0.002$
5.0 | $0.828\pm 0.002$ | $0.943\pm 0.002$ | $0.228\pm 0.008$
learned | 1.0 | $0.805\pm 0.002$ | $0.944\pm 0.000$ | $0.201\pm 0.004$
5.0 | $0.812\pm 0.002$ | $0.944\pm 0.002$ | $0.195\pm 0.003$
True | initial | 1.0 | $0.810\pm 0.001$ | $0.945\pm 0.001$ | $0.214\pm 0.003$
5.0 | $0.827\pm 0.005$ | $0.943\pm 0.002$ | $0.228\pm 0.007$
learned | 1.0 | $0.802\pm 0.002$ | $0.946\pm 0.001$ | $0.198\pm 0.004$
5.0 | $0.811\pm 0.004$ | $0.945\pm 0.002$ | $0.192\pm 0.003$

Table 10: Out-domain results for measure Gap and backbone ResNet50.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.552\pm 0.007$ | $0.956\pm 0.001$ | $0.045\pm 0.002$
5.0 | $0.218\pm 0.012$ | $0.960\pm 0.000$ | $0.001\pm 0.000$
learned | 1.0 | $0.552\pm 0.007$ | $0.956\pm 0.001$ | $0.045\pm 0.002$
5.0 | $0.218\pm 0.012$ | $0.960\pm 0.000$ | $0.001\pm 0.000$
True | initial | 1.0 | $0.566\pm 0.029$ | $0.957\pm 0.001$ | $0.051\pm 0.009$
5.0 | $0.219\pm 0.009$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.566\pm 0.029$ | $0.957\pm 0.001$ | $0.051\pm 0.009$
5.0 | $0.219\pm 0.009$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
MIMO | False | initial | 1.0 | $0.269\pm 0.027$ | $0.961\pm 0.001$ | $0.003\pm 0.002$
5.0 | $0.244\pm 0.004$ | $0.961\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.269\pm 0.027$ | $0.961\pm 0.001$ | $0.003\pm 0.002$
5.0 | $0.244\pm 0.004$ | $0.961\pm 0.001$ | $0.001\pm 0.000$
True | initial | 1.0 | $0.273\pm 0.018$ | $0.961\pm 0.000$ | $0.002\pm 0.001$
5.0 | $0.237\pm 0.007$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.273\pm 0.018$ | $0.961\pm 0.000$ | $0.002\pm 0.001$
5.0 | $0.237\pm 0.007$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
Mixup | False | initial | 1.0 | $0.565\pm 0.005$ | $0.954\pm 0.002$ | $0.056\pm 0.002$
5.0 | $0.259\pm 0.013$ | $0.960\pm 0.001$ | $0.002\pm 0.000$
learned | 1.0 | $0.539\pm 0.006$ | $0.955\pm 0.002$ | $0.041\pm 0.006$
5.0 | $0.259\pm 0.013$ | $0.960\pm 0.001$ | $0.002\pm 0.000$
True | initial | 1.0 | $0.566\pm 0.043$ | $0.956\pm 0.002$ | $0.060\pm 0.019$
5.0 | $0.260\pm 0.006$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.520\pm 0.022$ | $0.955\pm 0.001$ | $0.035\pm 0.007$
5.0 | $0.260\pm 0.006$ | $0.959\pm 0.001$ | $0.001\pm 0.000$
OC | False | initial | 1.0 | $0.569\pm 0.016$ | $0.956\pm 0.001$ | $0.051\pm 0.006$
5.0 | $0.216\pm 0.008$ | $0.959\pm 0.000$ | $0.000\pm 0.000$
learned | 1.0 | $0.569\pm 0.016$ | $0.956\pm 0.001$ | $0.051\pm 0.006$
5.0 | $0.216\pm 0.008$ | $0.959\pm 0.000$ | $0.000\pm 0.000$
True | initial | 1.0 | $0.539\pm 0.013$ | $0.956\pm 0.001$ | $0.044\pm 0.006$
5.0 | $0.218\pm 0.005$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
learned | 1.0 | $0.539\pm 0.013$ | $0.956\pm 0.001$ | $0.044\pm 0.006$
5.0 | $0.218\pm 0.005$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
RND | False | initial | 1.0 | $0.557\pm 0.017$ | $0.955\pm 0.003$ | $0.047\pm 0.007$
5.0 | $0.217\pm 0.003$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
learned | 1.0 | $0.557\pm 0.017$ | $0.955\pm 0.003$ | $0.047\pm 0.007$
5.0 | $0.217\pm 0.003$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
True | initial | 1.0 | $0.537\pm 0.013$ | $0.956\pm 0.001$ | $0.041\pm 0.005$
5.0 | $0.220\pm 0.006$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
learned | 1.0 | $0.565\pm 0.024$ | $0.957\pm 0.003$ | $0.050\pm 0.007$
5.0 | $0.220\pm 0.006$ | $0.959\pm 0.002$ | $0.001\pm 0.000$
SoftLabeler | False | initial | 1.0 | $0.587\pm 0.005$ | $0.953\pm 0.001$ | $0.049\pm 0.009$
5.0 | $0.406\pm 0.005$ | $0.958\pm 0.002$ | $0.020\pm 0.001$
learned | 1.0 | $0.587\pm 0.005$ | $0.953\pm 0.001$ | $0.049\pm 0.009$
5.0 | $0.406\pm 0.005$ | $0.958\pm 0.002$ | $0.020\pm 0.001$
True | initial | 1.0 | $0.586\pm 0.006$ | $0.953\pm 0.003$ | $0.047\pm 0.004$
5.0 | $0.394\pm 0.009$ | $0.957\pm 0.002$ | $0.016\pm 0.002$
learned | 1.0 | $0.586\pm 0.006$ | $0.953\pm 0.003$ | $0.047\pm 0.004$
5.0 | $0.394\pm 0.009$ | $0.957\pm 0.002$ | $0.016\pm 0.002$

Table 11: Out-domain results for measure Jacobian and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.545\pm 0.036$ | $0.951\pm 0.003$ | $0.058\pm 0.013$
learned | 1.0 | $0.534\pm 0.029$ | $0.951\pm 0.001$ | $0.059\pm 0.012$
True | initial | 1.0 | $0.522\pm 0.020$ | $0.950\pm 0.003$ | $0.053\pm 0.004$
learned | 1.0 | $0.522\pm 0.020$ | $0.950\pm 0.003$ | $0.053\pm 0.004$
MIMO | False | initial | 1.0 | $0.192\pm 0.007$ | $0.955\pm 0.002$ | $0.003\pm 0.001$
learned | 1.0 | $0.192\pm 0.007$ | $0.955\pm 0.002$ | $0.003\pm 0.001$
True | initial | 1.0 | $0.194\pm 0.009$ | $0.955\pm 0.001$ | $0.003\pm 0.000$
learned | 1.0 | $0.230\pm 0.009$ | $0.955\pm 0.002$ | $0.005\pm 0.001$
Mixup | False | initial | 1.0 | $0.561\pm 0.054$ | $0.951\pm 0.003$ | $0.061\pm 0.021$
learned | 1.0 | $0.509\pm 0.009$ | $0.948\pm 0.001$ | $0.053\pm 0.006$
True | initial | 1.0 | $0.595\pm 0.029$ | $0.948\pm 0.002$ | $0.072\pm 0.018$
learned | 1.0 | $0.504\pm 0.008$ | $0.949\pm 0.002$ | $0.049\pm 0.006$
OC | False | initial | 1.0 | $0.529\pm 0.018$ | $0.951\pm 0.001$ | $0.050\pm 0.005$
5.0 | $0.162\pm 0.000$ | $0.958\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.529\pm 0.018$ | $0.951\pm 0.001$ | $0.050\pm 0.005$
5.0 | $0.162\pm 0.000$ | $0.958\pm 0.001$ | $0.001\pm 0.000$
True | initial | 1.0 | $0.532\pm 0.021$ | $0.949\pm 0.002$ | $0.057\pm 0.006$
5.0 | $0.162\pm 0.009$ | $0.957\pm 0.000$ | $0.001\pm 0.000$
learned | 1.0 | $0.532\pm 0.021$ | $0.949\pm 0.002$ | $0.057\pm 0.006$
5.0 | $0.162\pm 0.009$ | $0.957\pm 0.000$ | $0.001\pm 0.000$
RND | False | initial | 1.0 | $0.528\pm 0.018$ | $0.949\pm 0.001$ | $0.052\pm 0.006$
5.0 | $0.166\pm 0.001$ | $0.958\pm 0.001$ | $0.001\pm 0.000$
learned | 1.0 | $0.528\pm 0.018$ | $0.949\pm 0.001$ | $0.052\pm 0.006$
5.0 | $0.166\pm 0.001$ | $0.958\pm 0.001$ | $0.001\pm 0.000$
True | initial | 1.0 | $0.543\pm 0.013$ | $0.956\pm 0.003$ | $0.054\pm 0.008$
5.0 | $0.161\pm 0.009$ | $0.958\pm 0.003$ | $0.001\pm 0.000$
learned | 1.0 | $0.543\pm 0.013$ | $0.956\pm 0.003$ | $0.054\pm 0.008$
5.0 | $0.161\pm 0.009$ | $0.958\pm 0.003$ | $0.001\pm 0.000$
SoftLabeler | False | initial | 1.0 | $0.525\pm 0.009$ | $0.950\pm 0.002$ | $0.045\pm 0.002$
5.0 | $0.380\pm 0.003$ | $0.953\pm 0.001$ | $0.016\pm 0.001$
learned | 1.0 | $0.525\pm 0.009$ | $0.950\pm 0.002$ | $0.045\pm 0.002$
5.0 | $0.380\pm 0.003$ | $0.953\pm 0.001$ | $0.016\pm 0.001$
True | initial | 1.0 | $0.536\pm 0.005$ | $0.950\pm 0.001$ | $0.045\pm 0.003$
5.0 | $0.379\pm 0.011$ | $0.953\pm 0.002$ | $0.015\pm 0.001$
learned | 1.0 | $0.536\pm 0.005$ | $0.950\pm 0.001$ | $0.045\pm 0.003$
5.0 | $0.379\pm 0.011$ | $0.953\pm 0.002$ | $0.015\pm 0.001$

Table 12: Out-domain results for measure Jacobian and backbone ResNet50.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.795\pm 0.009$ | $0.940\pm 0.002$ | $0.245\pm 0.006$
5.0 | $0.819\pm 0.005$ | $0.937\pm 0.002$ | $0.271\pm 0.009$
learned | 1.0 | $0.794\pm 0.009$ | $0.940\pm 0.002$ | $0.244\pm 0.006$
5.0 | $0.804\pm 0.005$ | $0.937\pm 0.000$ | $0.253\pm 0.005$
True | initial | 1.0 | $0.796\pm 0.004$ | $0.941\pm 0.000$ | $0.235\pm 0.007$
5.0 | $0.818\pm 0.005$ | $0.937\pm 0.001$ | $0.263\pm 0.011$
learned | 1.0 | $0.795\pm 0.004$ | $0.941\pm 0.000$ | $0.234\pm 0.007$
5.0 | $0.804\pm 0.005$ | $0.937\pm 0.002$ | $0.247\pm 0.012$
MCDropout | False | initial | 1.0 | $0.792\pm 0.008$ | $0.940\pm 0.001$ | $0.239\pm 0.017$
5.0 | $0.819\pm 0.004$ | $0.939\pm 0.003$ | $0.260\pm 0.017$
learned | 1.0 | $0.796\pm 0.005$ | $0.941\pm 0.001$ | $0.243\pm 0.012$
5.0 | $0.805\pm 0.004$ | $0.939\pm 0.001$ | $0.243\pm 0.016$
True | initial | 1.0 | $0.798\pm 0.004$ | $0.940\pm 0.001$ | $0.244\pm 0.005$
5.0 | $0.821\pm 0.005$ | $0.939\pm 0.001$ | $0.268\pm 0.006$
learned | 1.0 | $0.797\pm 0.004$ | $0.940\pm 0.001$ | $0.243\pm 0.006$
5.0 | $0.806\pm 0.005$ | $0.939\pm 0.001$ | $0.248\pm 0.011$
MIMO | False | initial | 1.0 | $0.802\pm 0.010$ | $0.939\pm 0.001$ | $0.251\pm 0.018$
5.0 | $0.814\pm 0.003$ | $0.939\pm 0.001$ | $0.247\pm 0.012$
learned | 1.0 | $0.801\pm 0.010$ | $0.939\pm 0.001$ | $0.251\pm 0.018$
5.0 | $0.796\pm 0.002$ | $0.940\pm 0.000$ | $0.231\pm 0.012$
True | initial | 1.0 | $0.801\pm 0.007$ | $0.942\pm 0.002$ | $0.244\pm 0.014$
5.0 | $0.814\pm 0.005$ | $0.939\pm 0.001$ | $0.242\pm 0.011$
learned | 1.0 | $0.800\pm 0.007$ | $0.942\pm 0.002$ | $0.243\pm 0.014$
5.0 | $0.794\pm 0.005$ | $0.939\pm 0.000$ | $0.231\pm 0.010$
Mixup | False | initial | 1.0 | $0.799\pm 0.009$ | $0.941\pm 0.002$ | $0.237\pm 0.015$
5.0 | $0.814\pm 0.006$ | $0.938\pm 0.003$ | $0.254\pm 0.013$
learned | 1.0 | $0.768\pm 0.009$ | $0.939\pm 0.001$ | $0.216\pm 0.006$
5.0 | $0.786\pm 0.002$ | $0.940\pm 0.001$ | $0.220\pm 0.007$
True | initial | 1.0 | $0.798\pm 0.002$ | $0.942\pm 0.002$ | $0.242\pm 0.009$
5.0 | $0.815\pm 0.002$ | $0.938\pm 0.001$ | $0.257\pm 0.009$
learned | 1.0 | $0.773\pm 0.007$ | $0.942\pm 0.001$ | $0.213\pm 0.017$
5.0 | $0.794\pm 0.002$ | $0.940\pm 0.001$ | $0.233\pm 0.005$
OC | False | initial | 1.0 | $0.797\pm 0.003$ | $0.940\pm 0.003$ | $0.246\pm 0.011$
5.0 | $0.818\pm 0.003$ | $0.939\pm 0.001$ | $0.265\pm 0.009$
learned | 1.0 | $0.796\pm 0.003$ | $0.940\pm 0.003$ | $0.244\pm 0.011$
5.0 | $0.803\pm 0.003$ | $0.939\pm 0.002$ | $0.247\pm 0.005$
True | initial | 1.0 | $0.796\pm 0.002$ | $0.941\pm 0.002$ | $0.243\pm 0.001$
5.0 | $0.819\pm 0.001$ | $0.940\pm 0.001$ | $0.263\pm 0.008$
learned | 1.0 | $0.795\pm 0.002$ | $0.941\pm 0.001$ | $0.241\pm 0.002$
5.0 | $0.805\pm 0.002$ | $0.940\pm 0.001$ | $0.244\pm 0.004$
RND | False | initial | 1.0 | $0.800\pm 0.003$ | $0.940\pm 0.002$ | $0.239\pm 0.004$
5.0 | $0.821\pm 0.001$ | $0.939\pm 0.001$ | $0.265\pm 0.003$
learned | 1.0 | $0.798\pm 0.003$ | $0.940\pm 0.002$ | $0.237\pm 0.004$
5.0 | $0.806\pm 0.002$ | $0.939\pm 0.001$ | $0.248\pm 0.003$
True | initial | 1.0 | $0.804\pm 0.005$ | $0.940\pm 0.002$ | $0.251\pm 0.016$
5.0 | $0.821\pm 0.001$ | $0.938\pm 0.002$ | $0.269\pm 0.004$
learned | 1.0 | $0.799\pm 0.004$ | $0.939\pm 0.001$ | $0.241\pm 0.007$
5.0 | $0.806\pm 0.002$ | $0.939\pm 0.003$ | $0.251\pm 0.002$
SoftLabeler | False | initial | 1.0 | $0.786\pm 0.020$ | $0.941\pm 0.002$ | $0.228\pm 0.019$
5.0 | $0.804\pm 0.001$ | $0.939\pm 0.001$ | $0.242\pm 0.004$
learned | 1.0 | $0.778\pm 0.018$ | $0.941\pm 0.001$ | $0.224\pm 0.019$
5.0 | $0.784\pm 0.003$ | $0.940\pm 0.001$ | $0.222\pm 0.001$
True | initial | 1.0 | $0.787\pm 0.003$ | $0.940\pm 0.000$ | $0.221\pm 0.010$
5.0 | $0.811\pm 0.006$ | $0.940\pm 0.002$ | $0.241\pm 0.001$
learned | 1.0 | $0.778\pm 0.003$ | $0.939\pm 0.001$ | $0.214\pm 0.009$
5.0 | $0.789\pm 0.004$ | $0.940\pm 0.002$ | $0.224\pm 0.003$

Table 13: Out-domain results for measure Largest and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 1.0 | $0.846\pm 0.002$ | $0.942\pm 0.001$ | $0.336\pm 0.010$
5.0 | $0.856\pm 0.001$ | $0.940\pm 0.000$ | $0.342\pm 0.003$
learned | 1.0 | $0.843\pm 0.003$ | $0.943\pm 0.003$ | $0.323\pm 0.004$
5.0 | $0.841\pm 0.002$ | $0.941\pm 0.001$ | $0.311\pm 0.006$
True | initial | 1.0 | $0.837\pm 0.006$ | $0.942\pm 0.000$ | $0.313\pm 0.014$
5.0 | $0.854\pm 0.001$ | $0.941\pm 0.003$ | $0.334\pm 0.005$
learned | 1.0 | $0.840\pm 0.007$ | $0.943\pm 0.001$ | $0.317\pm 0.014$
5.0 | $0.839\pm 0.001$ | $0.942\pm 0.003$ | $0.306\pm 0.006$
MCDropout | False | initial | 1.0 | $0.838\pm 0.004$ | $0.943\pm 0.001$ | $0.315\pm 0.005$
5.0 | $0.854\pm 0.002$ | $0.940\pm 0.003$ | $0.337\pm 0.003$
learned | 1.0 | $0.841\pm 0.004$ | $0.943\pm 0.001$ | $0.318\pm 0.005$
5.0 | $0.839\pm 0.002$ | $0.941\pm 0.002$ | $0.309\pm 0.003$
True | initial | 1.0 | $0.839\pm 0.005$ | $0.943\pm 0.004$ | $0.318\pm 0.013$
5.0 | $0.853\pm 0.002$ | $0.940\pm 0.001$ | $0.339\pm 0.004$
learned | 1.0 | $0.842\pm 0.005$ | $0.943\pm 0.004$ | $0.323\pm 0.014$
5.0 | $0.839\pm 0.002$ | $0.939\pm 0.001$ | $0.312\pm 0.002$
MIMO | False | initial | 1.0 | $0.847\pm 0.003$ | $0.941\pm 0.000$ | $0.348\pm 0.012$
5.0 | $0.844\pm 0.003$ | $0.938\pm 0.001$ | $0.321\pm 0.009$
learned | 1.0 | $0.846\pm 0.008$ | $0.941\pm 0.000$ | $0.345\pm 0.012$
5.0 | $0.826\pm 0.003$ | $0.938\pm 0.001$ | $0.288\pm 0.008$
True | initial | 1.0 | $0.845\pm 0.002$ | $0.942\pm 0.002$ | $0.330\pm 0.002$
5.0 | $0.846\pm 0.003$ | $0.938\pm 0.003$ | $0.326\pm 0.013$
learned | 1.0 | $0.826\pm 0.011$ | $0.943\pm 0.002$ | $0.293\pm 0.026$
5.0 | $0.827\pm 0.003$ | $0.939\pm 0.003$ | $0.292\pm 0.011$
Mixup | False | initial | 1.0 | $0.841\pm 0.005$ | $0.940\pm 0.001$ | $0.327\pm 0.010$
5.0 | $0.856\pm 0.002$ | $0.940\pm 0.004$ | $0.343\pm 0.009$
learned | 1.0 | $0.820\pm 0.008$ | $0.942\pm 0.003$ | $0.290\pm 0.020$
5.0 | $0.830\pm 0.003$ | $0.941\pm 0.003$ | $0.294\pm 0.007$
True | initial | 1.0 | $0.842\pm 0.004$ | $0.942\pm 0.003$ | $0.325\pm 0.015$
5.0 | $0.859\pm 0.001$ | $0.941\pm 0.001$ | $0.346\pm 0.006$
learned | 1.0 | $0.827\pm 0.006$ | $0.943\pm 0.002$ | $0.294\pm 0.014$
5.0 | $0.832\pm 0.002$ | $0.943\pm 0.003$ | $0.294\pm 0.012$
OC | False | initial | 1.0 | $0.840\pm 0.002$ | $0.943\pm 0.003$ | $0.319\pm 0.008$
5.0 | $0.853\pm 0.002$ | $0.942\pm 0.001$ | $0.331\pm 0.008$
learned | 1.0 | $0.842\pm 0.002$ | $0.943\pm 0.003$ | $0.323\pm 0.011$
5.0 | $0.839\pm 0.003$ | $0.942\pm 0.002$ | $0.304\pm 0.005$
True | initial | 1.0 | $0.839\pm 0.003$ | $0.941\pm 0.002$ | $0.316\pm 0.008$
5.0 | $0.852\pm 0.003$ | $0.941\pm 0.003$ | $0.331\pm 0.017$
learned | 1.0 | $0.841\pm 0.004$ | $0.942\pm 0.001$ | $0.321\pm 0.009$
5.0 | $0.837\pm 0.003$ | $0.941\pm 0.001$ | $0.304\pm 0.015$
RND | False | initial | 1.0 | $0.837\pm 0.002$ | $0.941\pm 0.001$ | $0.317\pm 0.005$
5.0 | $0.853\pm 0.002$ | $0.940\pm 0.003$ | $0.338\pm 0.017$
learned | 1.0 | $0.840\pm 0.002$ | $0.941\pm 0.001$ | $0.322\pm 0.005$
5.0 | $0.839\pm 0.002$ | $0.939\pm 0.003$ | $0.309\pm 0.014$
True | initial | 1.0 | $0.836\pm 0.006$ | $0.942\pm 0.003$ | $0.314\pm 0.012$
5.0 | $0.854\pm 0.003$ | $0.938\pm 0.002$ | $0.346\pm 0.014$
learned | 1.0 | $0.839\pm 0.006$ | $0.941\pm 0.003$ | $0.319\pm 0.010$
5.0 | $0.839\pm 0.003$ | $0.938\pm 0.002$ | $0.318\pm 0.010$
SoftLabeler | False | initial | 1.0 | $0.835\pm 0.003$ | $0.942\pm 0.001$ | $0.310\pm 0.013$
5.0 | $0.851\pm 0.001$ | $0.941\pm 0.000$ | $0.325\pm 0.011$
learned | 1.0 | $0.821\pm 0.002$ | $0.941\pm 0.003$ | $0.284\pm 0.011$
5.0 | $0.826\pm 0.002$ | $0.941\pm 0.001$ | $0.282\pm 0.004$
True | initial | 1.0 | $0.831\pm 0.002$ | $0.942\pm 0.001$ | $0.305\pm 0.009$
5.0 | $0.850\pm 0.007$ | $0.941\pm 0.001$ | $0.323\pm 0.015$
learned | 1.0 | $0.818\pm 0.002$ | $0.942\pm 0.002$ | $0.283\pm 0.001$
5.0 | $0.825\pm 0.004$ | $0.942\pm 0.001$ | $0.277\pm 0.007$

Table 14: Out-domain results for measure Largest and backbone ResNet50.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 5.0 | $0.768\pm 0.004$ | $0.942\pm 0.002$ | $0.126\pm 0.002$
learned | 5.0 | $0.768\pm 0.004$ | $0.942\pm 0.002$ | $0.126\pm 0.002$
True | initial | 5.0 | $0.771\pm 0.007$ | $0.943\pm 0.003$ | $0.128\pm 0.009$
learned | 5.0 | $0.771\pm 0.007$ | $0.943\pm 0.003$ | $0.128\pm 0.009$
MCDropout | False | initial | 1.0 | $0.519\pm 0.000$ | $0.960\pm 0.001$ | $0.094\pm 0.003$
5.0 | $0.772\pm 0.003$ | $0.945\pm 0.002$ | $0.126\pm 0.002$
learned | 1.0 | $0.517\pm 0.004$ | $0.958\pm 0.003$ | $0.099\pm 0.004$
5.0 | $0.772\pm 0.003$ | $0.945\pm 0.002$ | $0.126\pm 0.002$
True | initial | 1.0 | $0.521\pm 0.003$ | $0.959\pm 0.001$ | $0.099\pm 0.000$
5.0 | $0.770\pm 0.002$ | $0.942\pm 0.002$ | $0.126\pm 0.003$
learned | 1.0 | $0.521\pm 0.003$ | $0.959\pm 0.001$ | $0.099\pm 0.000$
5.0 | $0.770\pm 0.002$ | $0.942\pm 0.002$ | $0.126\pm 0.003$
MIMO | False | initial | 5.0 | $0.730\pm 0.008$ | $0.946\pm 0.004$ | $0.088\pm 0.014$
learned | 5.0 | $0.730\pm 0.008$ | $0.946\pm 0.004$ | $0.088\pm 0.014$
True | initial | 5.0 | $0.733\pm 0.003$ | $0.947\pm 0.002$ | $0.086\pm 0.005$
learned | 5.0 | $0.733\pm 0.003$ | $0.947\pm 0.002$ | $0.086\pm 0.005$
Mixup | False | initial | 1.0 | $0.438\pm 0.009$ | $0.942\pm 0.007$ | $0.041\pm 0.002$
5.0 | $0.754\pm 0.002$ | $0.945\pm 0.001$ | $0.109\pm 0.005$
learned | 1.0 | $0.498\pm 0.008$ | $0.948\pm 0.003$ | $0.055\pm 0.002$
5.0 | $0.754\pm 0.002$ | $0.945\pm 0.001$ | $0.109\pm 0.005$
True | initial | 1.0 | $0.477\pm 0.003$ | $0.944\pm 0.003$ | $0.050\pm 0.001$
5.0 | $0.745\pm 0.002$ | $0.946\pm 0.001$ | $0.098\pm 0.002$
learned | 1.0 | $0.476\pm 0.007$ | $0.951\pm 0.003$ | $0.041\pm 0.003$
5.0 | $0.745\pm 0.002$ | $0.946\pm 0.001$ | $0.098\pm 0.002$
OC | False | initial | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.769\pm 0.004$ | $0.942\pm 0.003$ | $0.125\pm 0.007$
learned | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.769\pm 0.004$ | $0.942\pm 0.003$ | $0.125\pm 0.007$
True | initial | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.769\pm 0.004$ | $0.944\pm 0.003$ | $0.120\pm 0.004$
learned | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.769\pm 0.004$ | $0.944\pm 0.003$ | $0.120\pm 0.004$
RND | False | initial | 1.0 | $0.390\pm 0.011$ | $0.957\pm 0.001$ | $0.011\pm 0.005$
5.0 | $0.771\pm 0.012$ | $0.943\pm 0.002$ | $0.125\pm 0.009$
learned | 1.0 | $0.390\pm 0.011$ | $0.957\pm 0.001$ | $0.011\pm 0.005$
5.0 | $0.771\pm 0.012$ | $0.943\pm 0.002$ | $0.125\pm 0.009$
True | initial | 1.0 | $0.468\pm 0.049$ | $0.956\pm 0.001$ | $0.031\pm 0.014$
5.0 | $0.769\pm 0.002$ | $0.944\pm 0.003$ | $0.123\pm 0.005$
learned | 1.0 | $0.370\pm 0.042$ | $0.956\pm 0.000$ | $0.009\pm 0.005$
5.0 | $0.769\pm 0.002$ | $0.944\pm 0.003$ | $0.123\pm 0.005$
SoftLabeler | False | initial | 1.0 | $0.769\pm 0.024$ | $0.941\pm 0.002$ | $0.228\pm 0.019$
5.0 | $0.735\pm 0.007$ | $0.947\pm 0.002$ | $0.103\pm 0.008$
learned | 1.0 | $0.683\pm 0.027$ | $0.941\pm 0.001$ | $0.224\pm 0.019$
5.0 | $0.735\pm 0.007$ | $0.947\pm 0.002$ | $0.103\pm 0.008$
True | initial | 1.0 | $0.771\pm 0.004$ | $0.940\pm 0.000$ | $0.221\pm 0.010$
5.0 | $0.734\pm 0.004$ | $0.948\pm 0.001$ | $0.096\pm 0.004$
learned | 1.0 | $0.683\pm 0.002$ | $0.939\pm 0.001$ | $0.214\pm 0.009$
5.0 | $0.734\pm 0.004$ | $0.948\pm 0.001$ | $0.096\pm 0.004$

Table 15: Out-domain results for measure Native and backbone ResNet18.

algorithm | spectral | calibration | k | AUC | InAsIn | OutAsOut
---|---|---|---|---|---|---
ERM | False | initial | 5.0 | $0.792\pm 0.005$ | $0.941\pm 0.001$ | $0.144\pm 0.003$
learned | 5.0 | $0.792\pm 0.005$ | $0.941\pm 0.001$ | $0.144\pm 0.003$
True | initial | 5.0 | $0.793\pm 0.002$ | $0.942\pm 0.000$ | $0.143\pm 0.006$
learned | 5.0 | $0.793\pm 0.002$ | $0.942\pm 0.000$ | $0.143\pm 0.006$
MCDropout | False | initial | 1.0 | $0.523\pm 0.003$ | $0.968\pm 0.001$ | $0.094\pm 0.004$
5.0 | $0.792\pm 0.005$ | $0.942\pm 0.003$ | $0.148\pm 0.009$
learned | 1.0 | $0.523\pm 0.003$ | $0.968\pm 0.001$ | $0.094\pm 0.004$
5.0 | $0.792\pm 0.005$ | $0.942\pm 0.003$ | $0.148\pm 0.009$
True | initial | 1.0 | $0.525\pm 0.003$ | $0.968\pm 0.002$ | $0.097\pm 0.002$
5.0 | $0.793\pm 0.004$ | $0.941\pm 0.004$ | $0.147\pm 0.002$
learned | 1.0 | $0.525\pm 0.003$ | $0.968\pm 0.002$ | $0.097\pm 0.002$
5.0 | $0.793\pm 0.004$ | $0.941\pm 0.004$ | $0.147\pm 0.002$
MIMO | False | initial | 5.0 | $0.766\pm 0.005$ | $0.943\pm 0.002$ | $0.097\pm 0.007$
learned | 5.0 | $0.766\pm 0.005$ | $0.943\pm 0.002$ | $0.097\pm 0.007$
True | initial | 5.0 | $0.765\pm 0.002$ | $0.943\pm 0.001$ | $0.098\pm 0.004$
learned | 5.0 | $0.765\pm 0.002$ | $0.943\pm 0.001$ | $0.098\pm 0.004$
Mixup | False | initial | 1.0 | $0.457\pm 0.003$ | $0.928\pm 0.003$ | $0.053\pm 0.004$
5.0 | $0.774\pm 0.008$ | $0.945\pm 0.000$ | $0.115\pm 0.003$
learned | 1.0 | $0.471\pm 0.019$ | $0.948\pm 0.004$ | $0.041\pm 0.003$
5.0 | $0.774\pm 0.008$ | $0.945\pm 0.000$ | $0.115\pm 0.003$
True | initial | 1.0 | $0.458\pm 0.015$ | $0.945\pm 0.000$ | $0.036\pm 0.002$
5.0 | $0.776\pm 0.002$ | $0.943\pm 0.002$ | $0.119\pm 0.007$
learned | 1.0 | $0.497\pm 0.008$ | $0.945\pm 0.002$ | $0.049\pm 0.005$
5.0 | $0.776\pm 0.002$ | $0.943\pm 0.002$ | $0.119\pm 0.007$
OC | False | initial | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.794\pm 0.007$ | $0.941\pm 0.000$ | $0.147\pm 0.010$
learned | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.794\pm 0.007$ | $0.941\pm 0.000$ | $0.147\pm 0.010$
True | initial | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.791\pm 0.006$ | $0.940\pm 0.003$ | $0.144\pm 0.006$
learned | 1.0 | $0.500\pm 0.000$ | $1.000\pm 0.000$ | $0.000\pm 0.000$
5.0 | $0.791\pm 0.006$ | $0.940\pm 0.003$ | $0.144\pm 0.006$
RND | False | initial | 1.0 | $0.488\pm 0.023$ | $0.957\pm 0.001$ | $0.019\pm 0.004$
5.0 | $0.793\pm 0.003$ | $0.940\pm 0.002$ | $0.146\pm 0.005$
learned | 1.0 | $0.488\pm 0.023$ | $0.957\pm 0.001$ | $0.019\pm 0.004$
5.0 | $0.793\pm 0.003$ | $0.940\pm 0.002$ | $0.146\pm 0.005$
True | initial | 1.0 | $0.463\pm 0.049$ | $0.956\pm 0.004$ | $0.022\pm 0.016$
5.0 | $0.790\pm 0.005$ | $0.940\pm 0.001$ | $0.145\pm 0.011$
learned | 1.0 | $0.463\pm 0.049$ | $0.956\pm 0.004$ | $0.022\pm 0.016$
5.0 | $0.790\pm 0.005$ | $0.940\pm 0.001$ | $0.145\pm 0.011$
SoftLabeler | False | initial | 1.0 | $0.826\pm 0.004$ | $0.942\pm 0.001$ | $0.310\pm 0.013$
5.0 | $0.766\pm 0.003$ | $0.947\pm 0.000$ | $0.114\pm 0.003$
learned | 1.0 | $0.716\pm 0.003$ | $0.941\pm 0.003$ | $0.284\pm 0.011$
5.0 | $0.766\pm 0.003$ | $0.947\pm 0.000$ | $0.114\pm 0.003$
True | initial | 1.0 | $0.821\pm 0.002$ | $0.942\pm 0.001$ | $0.305\pm 0.009$
5.0 | $0.769\pm 0.008$ | $0.944\pm 0.002$ | $0.124\pm 0.012$
learned | 1.0 | $0.708\pm 0.005$ | $0.942\pm 0.002$ | $0.283\pm 0.001$
5.0 | $0.769\pm 0.008$ | $0.944\pm 0.002$ | $0.124\pm 0.012$

Table 16: Out-domain results for measure Native and backbone ResNet50.

##  Appendix C ImageNot dataset

Our ImageNot dataset is a class partition of the ImageNet ILSVRC2012 dataset
(Russakovsky et al., 2015).

###  C.1 In-domain classes

    n02666196
    n02669723
    n02672831
    n02690373
    n02699494
    n02776631
    n02783161
    n02786058
    n02791124
    n02793495
    n02794156
    n02797295
    n02799071
    n02804610
    n02808304
    n02808440
    n02814860
    n02817516
    n02834397
    n02835271
    n02837789
    n02840245
    n02859443
    n02860847
    n02869837
    n02871525
    n02877765
    n02883205
    n02892767
    n02894605
    n02895154
    n02909870
    n02927161
    n02951358
    n02951585
    n02966687
    n02974003
    n02977058
    n02979186
    n02980441
    n02981792
    n02992211
    n03000684
    n03026506
    n03028079
    n03032252
    n03042490
    n03045698
    n03063599
    n03075370
    n03085013
    n03095699
    n03124170
    n03127747
    n03127925
    n03131574
    n03160309
    n03180011
    n03187595
    n03201208
    n03207743
    n03207941
    n03216828
    n03223299
    n03240683
    n03249569
    n03272010
    n03272562
    n03290653
    n03291819
    n03325584
    n03337140
    n03344393
    n03347037
    n03372029
    n03384352
    n03388549
    n03393912
    n03394916
    n03417042
    n03425413
    n03445777
    n03445924
    n03447721
    n03452741
    n03459775
    n03461385
    n03467068
    n03478589
    n03481172
    n03482405
    n03495258
    n03496892
    n03498962
    n03527444
    n03534580
    n03535780
    n03584254
    n03590841
    n03594734
    n03594945
    n03627232
    n03633091
    n03657121
    n03661043
    n03670208
    n03680355
    n03690938
    n03706229
    n03709823
    n03710193
    n03710721
    n03717622
    n03720891
    n03721384
    n03729826
    n03733281
    n03733805
    n03742115
    n03743016
    n03759954
    n03761084
    n03763968
    n03769881
    n03773504
    n03775071
    n03775546
    n03777568
    n03781244
    n03782006
    n03785016
    n03786901
    n03787032
    n03788365
    n03791053
    n03793489
    n03794056
    n03803284
    n03814639
    n03841143
    n03843555
    n03857828
    n03866082
    n03868863
    n03874293
    n03874599
    n03877845
    n03884397
    n03887697
    n03891332
    n03895866
    n03903868
    n03920288
    n03924679
    n03930313
    n03933933
    n03938244
    n03947888
    n03950228
    n03956157
    n03958227
    n03967562
    n03976657
    n03982430
    n03983396
    n03995372
    n04004767
    n04008634
    n04019541
    n04023962
    n04033995
    n04037443
    n04039381
    n04041544
    n04065272
    n04067472
    n04070727
    n04081281
    n04086273
    n04090263
    n04118776
    n04131690
    n04141327
    n04146614
    n04153751
    n04154565
    n04162706
    n04179913
    n04192698
    n04201297
    n04208210
    n04209133
    n04228054
    n04235860
    n04238763
    n04243546
    n04254120
    n04254777
    n04258138
    n04259630
    n04264628
    n04266014
    n04270147
    n04273569
    n04286575
    n04311004
    n04311174
    n04317175
    n04325704
    n04326547
    n04330267
    n04332243
    n04346328
    n04347754
    n04355933
    n04356056
    n04357314
    n04371430
    n04371774
    n04372370
    n04376876
    n04399382
    n04404412
    n04409515
    n04418357
    n04447861
    n04456115
    n04458633
    n04462240
    n04465501
    n04476259
    n04482393
    n04485082
    n04487394
    n04501370
    n04517823
    n04522168
    n04525038
    n04525305
    n04540053
    n04548280
    n04548362
    n04550184
    n04552348
    n04553703
    n04560804
    n04579145
    n04584207
    n04590129
    n04591157
    n04591713
    n04592741
    n04604644
    n04612504
    n04613696
    n06359193
    n07802026
    n07930864
    n09193705
    n09246464
    n09288635
    n09332890
    n09421951
    n09472597
    n10148035
    n15075141

###  C.2 Out-domain classes

    n01440764
    n01443537
    n01484850
    n01491361
    n01494475
    n01496331
    n01498041
    n01514668
    n01514859
    n01518878
    n01530575
    n01531178
    n01532829
    n01534433
    n01537544
    n01558993
    n01560419
    n01580077
    n01582220
    n01592084
    n01601694
    n01608432
    n01614925
    n01616318
    n01622779
    n01629819
    n01630670
    n01631663
    n01632458
    n01632777
    n01641577
    n01644373
    n01644900
    n01664065
    n01665541
    n01667114
    n01667778
    n01669191
    n01675722
    n01677366
    n01682714
    n01685808
    n01687978
    n01688243
    n01689811
    n01692333
    n01693334
    n01694178
    n01695060
    n01697457
    n01698640
    n01704323
    n01728572
    n01728920
    n01729322
    n01729977
    n01734418
    n01735189
    n01737021
    n01739381
    n01740131
    n01742172
    n01744401
    n01748264
    n01749939
    n01751748
    n01753488
    n01755581
    n01756291
    n01768244
    n01770081
    n01770393
    n01773157
    n01773549
    n01773797
    n01774384
    n01774750
    n01775062
    n01776313
    n01784675
    n01795545
    n01796340
    n01797886
    n01798484
    n01806143
    n01806567
    n01807496
    n01817953
    n01818515
    n01819313
    n01820546
    n01824575
    n01828970
    n01829413
    n01833805
    n01843065
    n01843383
    n01847000
    n01855032
    n01855672
    n01860187
    n01871265
    n01872401
    n01873310
    n01877812
    n01882714
    n01883070
    n01910747
    n01914609
    n01917289
    n01924916
    n01930112
    n01943899
    n01944390
    n01945685
    n01950731
    n01955084
    n01968897
    n01978287
    n01978455
    n01980166
    n01981276
    n01983481
    n01984695
    n01985128
    n01986214
    n01990800
    n02002556
    n02002724
    n02006656
    n02007558
    n02009229
    n02009912
    n02011460
    n02012849
    n02013706
    n02017213
    n02018207
    n02018795
    n02025239
    n02027492
    n02028035
    n02033041
    n02037110
    n02051845
    n02056570
    n02058221
    n02066245
    n02071294
    n02074367
    n02077923
    n02085620
    n02085782
    n02085936
    n02086079
    n02086240
    n02086646
    n02086910
    n02087046
    n02087394
    n02088094
    n02088238
    n02088364
    n02088466
    n02088632
    n02089078
    n02089867
    n02089973
    n02090379
    n02090622
    n02090721
    n02091032
    n02091134
    n02091244
    n02091467
    n02091635
    n02091831
    n02092002
    n02092339
    n02093256
    n02093428
    n02093647
    n02093754
    n02093859
    n02093991
    n02094114
    n02094258
    n02094433
    n02095314
    n02095570
    n02095889
    n02096051
    n02096177
    n02096294
    n02096437
    n02096585
    n02097047
    n02097130
    n02097209
    n02097298
    n02097474
    n02097658
    n02098105
    n02098286
    n02098413
    n02099267
    n02099429
    n02099601
    n02099712
    n02099849
    n02100236
    n02100583
    n02100735
    n02100877
    n02101006
    n02101388
    n02101556
    n02102040
    n02102177
    n02102318
    n02102480
    n02102973
    n02104029
    n02104365
    n02105056
    n02105162
    n02105251
    n02105412
    n02105505
    n02105641
    n02105855
    n02106030
    n02106166
    n02106382
    n02106550
    n02106662
    n02107142
    n02107312
    n02107574
    n02107683
    n02107908
    n02108000
    n02108089
    n02108422
    n02108551
    n02108915
    n02109047
    n02109525
    n02109961
    n02110063
    n02110185
    n02110341
    n02110627
    n02110806
    n02110958
    n02111129
    n02111277
    n02111500
    n02111889
    n02112018
    n02112137
    n02112350
    n02112706
    n02113023
    n02113186
    n02113624