area under the ROC curve.

##  V Calibration

A predictor is called well-calibrated if the derived predictive confidence
represents a good approximation of the actual probability of correctness [15].
Therefore, in order to make use of uncertainty quantification methods, one has
to be sure that the network is well calibrated. Formally, for classification
tasks a neural network $f_{\theta}$ is calibrated [265] if it holds that

$\displaystyle\forall
p\in[0,1]:\quad\sum_{i=1}^{N}\sum_{k=1}^{K}\frac{y_{i,k}\cdot\mathbb{I}\\{f_{\theta}(x_{i})_{k}=p\\}}{\mathbb{I}\\{f_{\theta}(x_{i})_{k}=p\\}}\xrightarrow[]{N\to\infty}p~{}.$
(37)

Here, $\mathbb{I}\\{\cdot\\}$ is the indicator function that is either 1 if
the condition is true or 0 if it is false and $y_{i,k}$ is the $k$-th entry in
the one-hot encoded groundtruth vector of a training sample $(x_{i},y_{i})$.
This formulation means that for example $30\%$ of all predictions with a
predictive confidence of $70\%$ should actually be false. For regression tasks
the calibration can be defined such that predicted confidence intervals should
match the confidence intervals empirically computed from the data set [265],
i.e.

$\forall
p\in[0,1]:\quad\sum_{i=1}^{N}\frac{\mathbb{I}\left\\{y_{i}\in\text{conf}_{p}(f_{\theta}(x_{i}))\right\\}}{N}\xrightarrow[]{N\to\infty}p,$
(38)

where $\text{conf}_{p}$ is the confidence interval that covers $p$ percent of
a distribution.

A DNN is called under-confident if the left hand side of (37) and (38) are
larger than $p$. Equivalently, it is under-confident if the terms are smaller
than $p$. The calibration property of a DNN can be visualized using a
reliability diagram, as shown in Figure 8.

In general, calibration errors are caused by factors related to model
uncertainty [15]. This is intuitively clear, since as discussed in Section II,
data uncertainty represents the underlying uncertainty that an input $x$ and a
target $y$ represent the same real world information. Following, correctly
predicted data uncertainty would lead to a perfectly calibrated neural
network. In practice, several works pointed out that deeper networks tend to
be more overconfident than shallower ones [15, 266, 267].
Several methods for uncertainty estimation presented in Section III also
improve the networks calibration [31, 20]. This is clear, since these methods
quantify model and data uncertainty separately and aim at reducing the model
uncertainty on the predictions. Besides the methods that improve the
calibration by reducing the model uncertainty, a large and growing body of
literature has investigated methods for explicitly reducing calibration
errors. These methods are presented in the following, followed by measures to
quantify the calibration error. It is important to note that these methods do
not reduce the model uncertainty, but propagate the model uncertainty onto the
representation of the data uncertainty. For example, if a binary classifier is
overfitted and predicts all samples of a test set as class A with probability
1, while half of the test samples are actually class B, the recalibration
methods might map the network output to 0.5 in order to have a reliable
confidence. This probability of 0.5 is not equivalent to the data uncertainty
but represents the model uncertainty propagated onto the predicted data
uncertainty.

(a) Underconfidence

(b) Overconfidence

(c) Calibrated classifier

Figure 8: (a) Reliability diagram showing an overconfident classifier: The
bin-wise accuracy is smaller than the corresponding confidence. (b)
Reliability diagram of an underconfident classifier: The bin-wise accuracy is
larger than the corresponding confidence. (c) Reliability diagram of a well
calibrated classifier: The confidence fits the actual accuracy for the single
bins.

###  V-A Calibration Methods

Calibration methods can be classified into three main groups according to the
step when they are applied:

  * •

Regularization methods applied during the training phase [268, 269, 11, 270,
271]
These methods modify the objective, optimization and/or regularization
procedure in order to build DNNs that are inherently calibrated.

  * •

Post-processing methods applied after the training process of the DNN [15, 47]
These methods require a held-out calibration data set to adjust the prediction
scores for recalibration. They only work under the assumption that the
distribution of the left-out validation set is equivalent to the distribution,
on which inference is done. Hence, also the size of the validation data set
can influence the calibration result.

  * •

Neural network uncertainty estimation methods
Approaches, as presented in Section III, that reduce the amount of model
uncertainty on a neural network’s confidence prediction, also lead to a better
calibrated predictor. This is because the remaining predicted data uncertainty
better represents the actual uncertainty on the prediction. Such methods are
based for example on Bayesian methods [272, 209, 273, 274, 16] or deep
ensembles [31, 275].

In the following, we present the three types of calibration methods in more
detail.

Neural Network Calibration Methods Regularization Methods Objective Function
Modification11 Data Augmentation10 Label Smoothing9 Exposure to OOD examples8
Uncertainty Estimation Approaches Bayesian Neural Networks6 Ensemble of Neural
Networks5 Post-Processing Methods Temperature Scaling4 Histogram Binning3
Gaussian Processes2 Ensemble of Post-Processing Models1 1 [48] 2 [47] 3 [276]
4[15, 68, 274] 5 [31, 275] 6 [272, 209, 273, 274, 16] 8 [277] 9 [268, 270] 10
[278, 279, 234, 280] 11 [269, 266, 11, 271, 279]

Figure 9: Visualization of the different types of uncertainty calibration
methods presented in this paper.

####  V-A1 Regularization Methods

Regularization methods for calibrating confidences manipulate the training of
DNNs by modifying the objective function or by augmenting the training data
set. The goal and idea of regularization methods is very similar to the
methods presented in Section III-A where the methods mainly quantify model and
data uncertainty separately within a single forward pass. However, the methods
in Section III-A quantify the model and data uncertainty, while these
calibration methods are regularized in order to minimize the model
uncertainty. Following, at inference, the model uncertainty cannot be obtained
anymore. This is the main motivation for us to separate the approaches
presented below from the approaches presented in Section III-A.
One popular regularization based calibration method is label smoothing [268].
For label smoothing, the labels of the training examples are modified by
taking a small portion $\alpha$ of the true class’ probability mass and assign
it uniformly to the false classes. For hard, non-smoothed labels, the optimum
cannot be reached in practice, as the gradient of the output with respect to
the logit vector $z$,

$\displaystyle\begin{split}\nabla_{z}\text{CE}(y,\hat{y}(z))&=\text{softmax}(z)-y\\\
&=\frac{\exp(z)}{\sum_{i=1}^{K}\exp(z_{i})}-y~{},\end{split}$ (39)

can only converge to zero with increasing distance between the true and false
classes’ logits. As a result, the logits of the correct class are much larger
than the logits for the incorrect classes and the logits of the incorrect
classes can be very different to each other. Label-smoothing avoids this and
while it generally leads to a higher training loss, the calibration error
decreases and the accuracy often increases as well [270].
Seo et al. [266] extended the idea of label smoothing and directly aimed at
reducing the model uncertainty. For this, they sampled $T$ forward passes of a
stochastic neural network already at training time. Based on the $T$ forward
passes of a training sample $(x_{i},y_{i})$, a normalized model variance
$\alpha_{i}$ is derived as the mean of the Bhattacharyya coefficients [281]
between the $T$ individual predictions $\hat{y}_{1},...,\hat{y}_{T}$ and the
average prediction $\bar{y}=\frac{1}{T}\sum_{t=1}^{T}\hat{y}_{t}$,

$\displaystyle\begin{split}\alpha_{i}&=\frac{1}{T}\sum_{t=1}^{T}BC(\bar{y}_{i},\hat{y}_{t})\\\
&=\frac{1}{T}\sum_{t=1}^{T}\sum_{k=1}^{K}\sqrt{\bar{y}_{i,k}\cdot\hat{y}_{i,t,k}}~{}.\end{split}$
(40)

Based on this $\alpha_{i}$, Seo et al. [266] introduced the variance-weighted
confidence-integrated loss function that is a convex combination of two
contradictive loss functions,

$\displaystyle\begin{split}L^{\text{VWCI}}(\theta)=-\sum_{i=1}^{N}(1-\alpha_{i})L_{\text{GT}}^{(i)}(\theta)+\alpha_{i}L_{\text{U}}^{(i)}(\theta)~{},\end{split}$
(41)

where $L_{\text{GT}}^{(i)}$ is the mean cross-entropy computed for the
training sample $x_{i}$ with given ground-truth $y_{i}$. $L_{\text{U}}$
represents the mean KL-divergence between a uniform target probability vector
and the computed prediction. The adaptive smoothing parameter ${\alpha}_{i}$
pushes predictions of training samples with high model uncertainty (given by
high variances) towards a uniform distribution while increasing the prediction
scores of samples with low model uncertainty. As a result, variances in the
predictions of a single sample are reduced and the network can then be applied
with a single forward pass at inference.

Pereyra et al. [269] combated the overconfidence issue by adding the negative
entropy to the standard loss function and therefore a penalty that increases
with the network’s predicted confidence. This results in the entropy-based
objective function $L^{H}$, which is defined as

$L^{H}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\log\hat{y}_{i}-\alpha_{i}H(\hat{y}_{i})~{},$
(42)

where $H(\hat{y}_{i})$ is the entropy of the output and $\alpha_{i}$ a
parameter that controls the strength of the confidence penalty and is computed
equivalently as for the VWCI loss.

Instead of regularizing the training process by modifying the objective
function, Thulasidasan et al. [278] regularized it by using a data-agnostic
data augmentation technique named mixup [282]. In mixup training, the network
is not only trained on the training data, but also on virtual training samples
$(\tilde{x},\tilde{y})$ generated by a convex combination of two random
training pairs $(x_{i},y_{i})$ and $(x_{j},y_{j})$, i.e.

$\tilde{x}=\lambda x_{i}+(1-\lambda)x_{j}$ (43) $\tilde{y}=\lambda
y_{i}+(1-\lambda)y_{j}~{}.$ (44)

According to [278], the label smoothing resulting from mixup training can be
viewed as a form of entropy-based regularization resulting in inherent
calibration of networks trained with mixup. Maro$\tilde{\text{n}}$as et al.
[279] see mixup training among the most popular data augmentation
regularization techniques due to its ability to improve the calibration as
well as the accuracy. However, they argued that in mixup training the data
uncertainty in mixed inputs affects the calibration and therefore mixup does
not necessarily improve the calibration. They also underlined this claim
empirically. Similarly, Rahaman and Thiery [234] experimentally showed that
the distributional-shift induced by data augmentation techniques such as mixup
training can negatively affect the confidence calibration. Based on this
observation, Maro$\tilde{\text{n}}$as et al. [279] proposed a new objective
function that explicitly takes the calibration performance on the unmixed
input samples into account. Inspired by the expected calibration error (ECE,
see Section V-B) Naeini et al. [283] measured the calibration performance on
the unmixed samples for each batch $b$ by the differentiable squared
differences between the batch accuracy and the mean confidence on the batch
samples. The total loss is given as a weighted combination of the original
loss on mixed and unmixed samples and the calibration measure evaluated only
on the unmixed samples:

$L^{ECE}(\theta)=\frac{1}{B}\sum_{b\in B}L^{b}(\theta)+\beta ECE_{b}~{},$ (45)

where $L^{b}(\theta)$ is the original unregularized loss using training and
mixed samples included in batch $b$ and $\beta$ is a hyperparameter
controlling the relative importance given to the batchwise expected
calibration error $ECE_{b}$. By adding the batchwise calibration error for
each batch $b\in B$ to the standard loss function, the miscalibration induced
by mixup training is regularized.

In the context of data augmentation, Patel et al. [280] improved the
calibration of uncertainty estimates by using on-manifold data augmentation.
While mixup training combines training samples, on-manifold adversarial
training generate out-of-domain samples using adversarial attack. They
experimentally showed that on-manifold adversarial training outperforms mixup
training in improving the calibration. Similar to [280], Hendrycks et al.
[277] showed that exposing classifiers to out-of-distribution examples at
training can help to improve the calibration.

####  V-A2 Post-Processing Methods

Post-processing (or post-hoc) methods are applied after the training process
and aim at learning a re-calibration function. For this, a subset of the
training data is held-out during the training process and used as a
calibration set. The re-calibration function is applied to the network’s
outputs (e.g. the logit vector) and yields an improved calibration learned on
the left-out calibration set. Zhang et al. [48] discussed three requirements
that should be satisfied by post-hoc calibration methods. They should

  1. 1.

preserve the accuracy, i.e. should not affect the predictors performance.

  2. 2.

be data efficient, i.e. only a small fraction of the training data set should
be left out for the calibration.

  3. 3.

be able to approximate the correct re-calibration map as long as there is
enough data available for calibration.

Furthermore, they pointed out that none of the existing approaches fulfills
all three requirements.
For classification tasks, the most basic but still very efficient way of post-
hoc calibration is temperature scaling [15]. For temperature scaling, the
temperature $T>0$ of the softmax function

$\text{softmax}(z_{i})=\frac{\exp^{z_{i}/T}}{\sum_{j=1}^{K}\exp^{z_{j}/T}}~{},$
(46)

is optimized. For $T=1$ the function remains the regular softmax function. For
$T>1$ the output changes such that its entropy increases, i.e. the predicted
confidence decreases. For $T\in(0,1)$ the entropy decreases and following, the
predicted confidence increases. As already mentioned above, a perfect
calibrated neural network outputs MAP estimates. Since the learned
transformation can only affect the uncertainty, the log-likelihood based
losses as cross-entropy do not have to be replaced by a special calibration
loss. While the data efficiency and the preservation of the accuracy is given,
the expressiveness of basic temperature scaling is limited [48]. To overcome
this, Zhang et al. [48] investigated an ensemble of several temperature
scaling models. Doing so, they achieved better calibrated predictions, while
preserving the classification accuracy and improving the data efficiency and
the expressive power. Kull et al. [284] were motivated by non-neural network
calibration methods, where the calibration is performed class-wise as a one-
vs-all binary calibration. They showed that this approach can be interpreted
as learning a linear transformation of the predicted log-likelihoods followed
by a softmax function. This again is equivalent to train a dense layer on the
log-probabilities and hence the method is also very easy to implement and
apply. Obviously, the original predictions are not guaranteed to be preserved.

Analogous to temperature scaling for classification networks, Levi et al.
[285] introduced standard deviation scaling (std-scaling) for regression
networks. As the name indicates, the method is trained to rescale the
predicted standard deviations of a given network. Equivalently to the
motivation of optimizing temperature scaling with the cross-entropy loss, std-
scaling can be trained using the Gaussian log-likelihood function as loss,
which is in general also used for the training of regression networks, which
also give a prediction for the data uncertainty.

Wenger et al. [47] proposed a Gaussian process (GP) based method, which can be
used to calibrate any multi-class classifier that outputs confidence values
and presented their methodology by calibrating neural networks. The main idea
behind their work is to learn the calibration map by a Gaussian process that
is trained on the networks confidence predictions and the corresponding
ground-truths in the left out calibration set. For this approach, the
preservation of the predictions is also not assured.

####  V-A3 Calibration with Uncertainty Estimation Approaches

As already discussed above, removing the model uncertainty and receiving an
accurate estimation of the data uncertainty leads to a well calibrated
predictor. Following several works based on deep ensembles [31, 275] and BNNs,
[272, 209, 204] also compared their performance to other methods based on the
resulting calibration. Lakshminarayanan et al. [31] and Mehrtash et al. [275]
reported an improved calibration by applying deep ensembles compared to single
networks. However, Rahaman and Thiery[234] showed that for specific
configurations as the usage of mixup-regularization, deep ensembles can even
increase the calibration error. On the other side they showed that applying
temperature scaling on the averaged predictions can give a significant
improvement on the calibration.
For the Bayesian approaches, [204] showed that restricting the Bayesian
approximation to the weights of the last fully connected layer of a DNN is
already enough to improve the calibration significantly. Zhang et al. [273]
and Laves et al. [274] showed that confidence estimates computed with MC
dropout can be poorly calibrated. To overcome this, Zhang et al. [273]
proposed structured dropout, which consists of dropping channel, blocks or
layers, to promote model diversity and reduce calibration errors.

###  V-B Evaluating Calibration Quality

Evaluating calibration consists of measuring the statistical consistency
between the predictive distributions and the observations [286]. For
classification tasks, several calibration measures are based on binning. For
that, the predictions are ordered by the predicted confidence $\hat{p}_{i}$
and grouped into $M$ bins $b_{1},...,b_{M}$. Following, the calibration of the
single bins is evaluated by setting the average bin confidence

$\text{conf}(b_{m})=\frac{1}{|b_{m}|}\sum_{s\in b_{m}}\hat{p}_{s}$ (47)

in relation to the average bin accuracy

$\text{acc}(b_{m})=\frac{1}{|b_{m}|}\sum_{s\in
b_{m}}\mathbbm{1}(\hat{y}_{s}=y_{s})~{},$ (48)

where $\hat{y}_{s}$, $y_{s}$ and $\hat{p}_{s}$ refer to the predicted and true
class label of a sample $s$. As noted in [15], confidences are well-calibrated
when for each bin $\text{acc}(b_{m})=\text{conf}(b_{m})$. For a visual
evaluation of a model’s calibration, the reliability diagram introduced by
[287] is widely used. For a reliability diagram, the $\text{conf}(b_{m})$ is
plotted against $\text{acc}(b_{m})$. For a well-calibrated model, the plot
should be close to the diagonal, as visualized in Figure 8. The basic
reliability diagram visualization does not distinguish between different
classes. In order to do so and hence to improve the interpretability of the
calibration error, Vaicenavicius et al. [286] used an alternative
visualization named multidimensional reliability diagram.

For a quantitative evaluation of a model’s calibration, different calibration
measures can be considered.
The Expected Calibration Error (ECE) is a widely used binning based
calibration measure [283, 15, 274, 275, 278, 47]. For the ECE, $M$ equally-
spaced bins $b_{1},...,b_{M}$ are considered, where $b_{m}$ denotes the set of
indices of samples whose confidences fall into the interval
$I_{m}=]\frac{m-1}{M},\frac{m}{M}]$. The ECE is then computed as the weighted
average of the bin-wise calibration errors, i.e.

$\text{ECE}=\sum_{m=1}^{M}\frac{|b_{m}|}{N}|\text{acc}(b_{m})-\text{conf}(b_{m})|~{}.$
(49)

For the ECE, only the predicted confidence score (top-label) is considered. In
contrast to this, the Static Calibration Error (SCE) [288, 289] considers the
predictions of all classes (all-labels). For each class, the SCE computes the
calibration error within the bins and then averages across all the bins, i.e.

$\text{SCE}=\frac{1}{K}\sum_{k=1}^{K}\sum_{m=1}^{M}\frac{|b_{m_{k}}|}{N}|\text{conf}(b_{m_{k}})-\text{acc}(b_{m_{k}})|~{}.$
(50)

Here $conf(b_{m_{k}})$ and $acc(b_{m_{k}})$ are the confidence and accuracy of
bin $b_{m}$ for class label $k$, respectively. Nixon et al. [288] empirically
showed that all-labels calibration measures such as the SCE are more effective
in assessing the calibration error than the top-label calibration measures as
the ECE.

In contrast to the ECE and SCE, which group predictions into $M$ equally-
spaced bins (what in general leads to different numbers of evaluation samples
per bin), the adaptive calibration error [288, 289] adaptively groups
predictions into $R$ bins with different width but equal number of
predictions. With this adaptive bin size, the adaptive Expected Calibration
Error (aECE)

$\text{aECE}=\frac{1}{R}\sum_{r=1}^{R}|\text{conf}(b_{r})-\text{acc}(b_{r})|~{},$
(51)

and the adaptive Static Calibration Error (aSCE)

$\text{aSCE}=\frac{1}{KR}\sum_{k=1}^{K}\sum_{r=1}^{R}|\text{conf}(b_{r_{k}})-\text{acc}(b_{r_{k}})|$
(52)

are defined as extensions of the ECE and the SCE.
As has been empirically shown in [280] and [288], the adaptive binning
calibration measures aECE and aSCE are more robust to the number of bins than
the corresponding equal-width binning calibration measures ECE and SCE.

It is important to make clear that in a multi-class setting, the calibration
measures can suffer from imbalance in the test data. Even when then
calibration is computed classwise, the computed errors are weighted by the
number of samples in the classes. Following, larger classes can shadow the bad
calibration on small classes, comparable to accuracy values in classification
tasks [290].

##  VI Data Sets and Baselines

TABLE IV: Overview of frequently compared benchmark approaches, tasks and their data sets among existing works organized according to the taxonomy of this paper. |  Tasks |  Task index: Data sets |  Baselines
---|---|---|---
Bayesian Neural Networks |  1\. Regression1,3,4,7,8,11,12,15-17
2\. Calibration6,10,13,14
3\. OOD Detection4,6,8,10,12,13
4\. Adversarial Attacks4,8,12
5\. Active Learning7,12,14
6\. Continual Learning10
7\. Reinforcement Learning (Intrinsic Motivation, Contextual Bandits)1,14,15 |  1: UCI
2, 3, 4: (not)MNIST,
CIFAR10/100, SHVN,
ImageNet
5: UCI
6: Permuted MNIST |  Softmax44,
MCdropout1, DeepEnsemble2,
BBB3, NormalizingFlow4,
PBP7, SWAG6,
KFAC8, DVI11
HMC9,
VOGN10, INF12
1[20] 3[30] 4[149] 5[75] 6[213] 7[129] 8[63] 9[80] 10[150] 11[143] 12[84]
13[235] 14[145] 15[135] 16[146] 17[272]
Ensembles |  1\. Regression2,20
2\. Calibration2,20,24-32
3\. OOD Detection2,20,25,27-30,32-34
4\. Active Learning31 |  1: Toy7, UCI
2, 3: Toy, (not)MNIST, SVHN, LSUN, CIFAR10/100,
(Tiny)ImageNet, Diabetic Retinopathy
4: MNIST |  Softmax44, MFVI5, SGLD55,
MCdropout1, DeepEnsemble2,
BBB3, PBP7, NormalizingFlow4,
TemperatureScaling38,54
2[31] 20[90] 24[234] 25[85] 26[86] 27[94] 28[39] 29[235] 30[40] 31[240] 32[13]
33[241] 34[246]
Single Deterministic Models |  1\. Regression21-23
2\. Calibration21-23,39,40,41,43
3\. OOD Detection21,22,38,39,41-53
4\. Adversarial Attacks21,41,48 |  1\. Toy7, UCI, NYU Depth
2, 3: (E/Fashion/not)MNIST,
Toy, CIFAR10/100, SVHN,
LSUN, (Tiny)ImageNet, IMDB,
Diabetic Retinopathy, Omniglot
4: MNIST, CIFAR10, NYU Depth, Omniglot |  Softmax44, GAN27, Dirichlet48, BBB3,
MCdropout1, DeepEnsemble2,57,
Mahalanobis56, TemperatureScaling38,54
NormalizingFlow4
21[101] 22[103] 23[104] 38[68] 39[143] 40[98] 41[43] 42[96] 43[95] 44[67]
45[64] 46[92] 47[72] 48[44] 49[94] 50[35] 51[36] 52[46] 53[71]
Test-Time Data Augmentation |  1\. Semantic Segmentation36, 37
2\. Calibration35
3\. OOD Detection35-37 |  1, 2, 3: Medical data, Diabetic Retinopathy |  Softmax44, MCdropout1
35[14] 36[248] 37[249]
|  |  |
54[15] 55[81] 56[291] 57[259]

In this section, we collect commonly used tasks and data sets for evaluating
uncertainty estimation among existing works. Besides, a variety of baseline
approaches commonly used as comparison against the methods proposed by the
researchers are also presented. By providing a review on the relevant
information of these experiments, we hope that both researchers and
practitioners can benefit from it. While the former can gain a basic
understanding of recent benchmarks tasks, data sets and baselines so that they
can design appropriate experiments to validate their ideas more efficiently,
the latter might use the provided information to select more relevant
approaches to start based on a concise overview on the tasks and data sets on
which the approach has been validated.

In the following, we will introduce the data sets and baselines summarized in
table IV according to the taxonomy used throughout this review.

The structure of the table is designed to organize the main contents of this
section concisely, hoping to provide a clear overview of the relevant works.
We group the approaches of each category into one of four blocks and extract
the most commonly used tasks, data sets and provided baselines for each column
respectively. The corresponding literature is listed at the bottom of each
block to facilitate lookup. Note that we focus on methodological comparison
here, but not the choice of architecture for different methods which has an
impact on performance as well. Due to the space limitation and visual density,
we only show the most important elements (task, data set, baselines) ranked
according to the frequency of use in the literature we have researched.

The main results are as follows. One of the most frequent tasks for evaluating
uncertainty estimation methods are regression tasks, where samples close and
far away from the training distribution are studied. Furthermore, the
calibration of uncertainty estimates in the case of classification problems is
very often investigated. Further noteworthy tasks are out-of-distribution
(OOD) detection and robustness against adversarial attacks. In the medical
domain, calibration of semantic segmentation results is the predominant use
case.

The choice of data sets is mostly consistent among all reviewed works. For
regression, toy data sets are employed for visualization of uncertainty
intervals while the UCI data sets are studied in light of (negative) log-
likelihood comparison. The most common data sets for calibration and OOD
detection are MNIST, CIFAR10 and 100 as well as SVHN while ImageNet and its
tiny variant are also studied frequently. These form distinct pairs when OOD
detection is studied where models trained on CIFAR variants are evaluated on
SVHN and visa versa while MNIST is paired with variants of itself like
notMNIST and FashionMNIST. Classification data sets are also commonly
distorted and corrupted to study the effects on calibration, blurring the line
between OOD detection and adversarial attacks.

Finally, the most commonly used baselines by far are Monte Carlo (MC) Dropout
and deep ensembles while the softmax output of deterministic models is almost
always employed as a kind of surrogate baseline. It is interesting to note
that inside each approach–BNNs, Ensembles, Single Deterministic Models and
Input Augmentation–some baselines are preferred over others. BNNs are most
frequently compared against variational inference methods like Bayes’ by
Backprop (BBB) or Probabilistic Backpropagation (PBP) while for Single
Deterministic Models it is more common to compare them against distance-based
methods in the case of OOD detection. Overall, BNN methods show a more diverse
set of tasks considered while being less frequently evaluated on large data
sets like ImageNet.

To further facilitate access for practitioners, we provide web-links to the
authors’ official implementations (marked by a star) of all common baselines
as identified in the baselines column. Where no official implementation is
provided, we instead link to the highest ranked implementations found on
GitHub at the time of this survey. The list can be also found within our
GitHub repository on available
implementations555https://github.com/JakobCode/UncertaintyInNeuralNetworks_Resources.
The relevant baselines are Softmax* (TensorFlow, PyTorch), MCdropout
(TensorFlow*; PyTorch: 1, 2), DeepEnsembles (TensorFlow: 1, 2, 3; PyTorch: 1,
2), BBB (PyTorch: 1, 2, 3, 4), NormalizingFlow (TensorFlow, PyTorch), PBP,
SWAG (1*, 2), KFAC (PyTorch: 1, 2, 3; TensorFlow), DVI (TensorFlow*, PyTorch),
HMC, VOGN*, INF*, MFVI, SGLD, TemperatureScaling (1*, 2, 3), GAN*, Dirichlet
and Mahalanobis*.

##  VII Applications of Uncertainty Estimates

From a practical point of view, the main motivation for quantifying
uncertainties in DNNs is to be able to classify the received predictions and
to make more confident decisions. This section gives a brief overview and
examples of the aforementioned motivations. In the first part, we discuss how
uncertainty is used within active learning and reinforcement learning.
Subsequently, we discuss the interest of the communities working on domain
fields like medical image analysis, robotics, and earth observation. These
application fields are used representatively for the large number of domains
where the uncertainty quantification plays an important role. The challenges
and concepts could (and should) be transferred to any application domain of
interest.

####  VII-1 Active Learning

The process of collecting labeled data for supervised training of a DNN can be
laborious, time-consuming, and costly. To reduce the annotation effort, the
active learning framework shown in Figure 10 trains the DNN sequentially on
different labelled data sets increasing in size over time [292]. In
particular, given a small labelled data set and a large unlabeled data set, a
deep neural network trained in the setting of active learning learns from the
small labeled data set and decides based on the acquisition function, which
samples to select from the pool of unlabeled data. The selected data are added
to the training data set and a new DNN is trained on the updated training data
set. This process is then repeated with the training set increasing in size
over time. Uncertainty sampling is one most popular criteria used in
acquisition functions [293] where predictive uncertainty determines which
training samples have the highest uncertainty and should be labelled next.
Uncertainty based active learning strategies for deep learning applications
have been successfully used in several works [23, 24, 294, 25, 26].

Figure 10: The active learning framework: The acquisition function evaluates
the uncertainties on the network’s test predictions in order to select
unlabelled data. The selected data are labelled and added to the pool of
labelled data, which is used to train and improve the performance of the
predictor.

####  VII-2 Reinforcement Learning

The general framework of deep reinforcement learning is shown in Figure 11. In
the context of reinforcement learning, uncertainty estimates can be used to
solve the exploration-exploitation dilemma. It says that uncertainty estimates
can be used to effectively balance the exploration of unknown environments
with the exploitation of existing knowledge extracted from known environments.
For example, if a robot interacts with an unknown environment, the robot can
safely avoid catastrophic failures by reasoning about its uncertainty. To
estimate the uncertainty in this framework, Huang et al. [27] used an ensemble
of bootstrapped models (models trained on different data sets sampled with
replacement from the original data set), while Gal and Ghahramani [20]
approximated Bayesian inference via dropout sampling. Inspired by [20] and
[27], Kahn et al. [28] and Lötjens et al. [29] used a mixture of deep Bayesian
networks performing dropout sampling on an ensemble of bootstrapped models.
For further reading, Ghavamzadeh et al. [295] presented a survey of Bayesian
reinforcement learning.

Figure 11: The reinforcement learning framework: The agent interacts with the
environment by executing a specific action influencing the next state of the
agent. The agent observes a reward representing the cost associated with the
executed action. The agent chooses actions based on a policy learned by a deep
neural network. However, the predicted uncertainty associated with the action
predicted by the deep neural network can help the agent to decide weather to
execute the predicted action or not.

###  VII-A Uncertainty in Real-World Applications

With increasing usage of deep learning approaches within many different
fields, quantifying and handling uncertainties has become more and more
important. On one hand, uncertainty quantification plays an important role in
risk minimization, which is needed in many application fields. On the other
hand, many fields offer only challenging data sources, which are hard to
control and verify. This makes the generation of trust-worthy ground truth a
very challenging task. In the following, three different fields where
uncertainty plays an important role are presented, namely Autonomous Driving,
medical image analysis, and earth observation.

####  VII-A1 Medical Analysis

Since the size, shape, and location of many diseases vary largely across
patients, the estimation of the predictive uncertainty is crucial in analyzing
medical images in applications such as lesion detection [1, 3], lung node
segmentation [296], brain tumor segmentation [152, 248, 249, 2, 261], parasite
segmentation in images of liver stage malaria [262], recognition of
abnormalities on chest radiographs [297], and age estimation [6]. Here,
uncertainty estimates in particular improve the interpretability of decisions
of DNNs [298]. They are essential to understand the reliability of
segmentation results, to detect false segmented areas and to guide human
experts in the task of refinement [249]. Well-calibrated and reliable
uncertainty estimates allow clinical experts to properly judge whether an
automated diagnosis can be trusted [298]. Uncertainty was estimated in medical
image segmentation based on Monte Carlo dropout [152, 296, 1, 2, 3, 263, 4, 5,
6], spike-and-slab dropout [261], and spatial dropout [262]. Wang et al. [248,
249] used test time data augmentation to estimate the data-dependent
uncertainty in medical image segmentation.

####  VII-A2 Robotics

Robots are active agents that perceive, decide, plan, and act in the real-
world – all based on their incomplete knowledge about the world. As a result,
mistakes of the robots not only cause failures of their own mission, but can
endanger human lives, e.g. in case of surgical robotics, self-driving cars,
space robotics, etc. Hence, the robotics application of deep learning poses
unique research challenges that significantly differ from those often
addressed in computer vision and other off-line settings [299]. For example,
the assumption that the testing condition come from the same distribution as
training is often invalid in many settings of robotics, resulting in
deterioration of the performance of DNNs in uncontrolled and detrimental
conditions. This raises the questions how we can quantify the uncertainty in a
DNN’s predictions in order to avoid catastrophic failures. Answering such
questions are important in robotics, as it might be a lofty goal to expect
data-driven approaches (in many aspects from control to perception) to always
be accurate. Instead, reasoning about uncertainty can help in leveraging the
recent advances in deep learning for robotics.

Reasoning about uncertainties and the use of probabilistic representations, as
oppose to relying on a single, most-likely estimate, have been central to many
domains of robotics research, even before the advent of deep learning [300].
In robot perception, several uncertainty-aware methods have been proposed in
the past, starting from localization methods [301, 302, 303] to simultaneous
localization and mapping (SLAM) frameworks [304, 305, 306, 307]. As a result,
many probabilistic methods such as factor graphs [308, 309] are now the work-
horse of advanced consumer products such as robotic vacuum cleaners and
unmanned aerial vehicles. In case of planning and control, estimation problems
are widely treated as Bayesian sequential learning problems, and sequential
decision making frameworks such as POMDPs [310, 311] assume a probabilistic
treatment of the underlying planning problems. With probabilistic
representations, many reinforcement learning algorithms are backed up by
stability guarantees for safe interactions in the real-world [312, 313, 314].
Lastly, there have been also several advances starting from reasoning
(semantics [315] to joint reasoning with geometry), embodiment (e.g. active
perception [316]) to learning (e.g. active learning [317, 318, 319] and
identifying unknown objects [320, 321, 322]).

Similarly, with the advent of deep learning, many researchers proposed new
methods to quantify the uncertainty in deep learning as well as on how to
further exploit such information. As oppose to many generic approaches, we
summarize task-specific methods and their application in practice as
followings. Notably, [323] proposed to perform novelty detection using auto-
encoders, where the reconstructed outputs of auto-encoders was used to decide
how much one can trust the network’s predictions. Peretroukhin et al. [324]
developed a SO(3) representation and uncertainty estimation framework for the
problem of rotational learning problems with uncertainty. [325, 28, 326, 327]
demonstrated uncertainty-aware, real world application of a reinforcement
learning algorithm for robotics, while [328, 329] proposed to leverage spatial
information, on top of MC-dropout. [207, 330, 331] developed deep learning
based localization systems along with uncertainty estimates. Other approaches
also learn on the robots’ past experiences of failures or detect
inconsistencies of the predictors [332, 333]. In summary, the robotics
community has been both, the users and the developers of the uncertainty
estimation frameworks targeted to a specific problems.

Yet, robotics pose several unique challenges to uncertainty estimation methods
for DNNs. These are for example, (i) how to limit the computational burden and
build real-time capable methods that can be executed on the robots with
limited computational capacities (e.g. aerial, space robots, etc); (ii) how to
leverage spatial and temporal information, as robots sense sequentially
instead of having a batch of training data for uncertainty estimates; (iii)
whether robots can select the most uncertainty samples and update its learner
online; (iv) Whether robots can purposefully manipulate the scene when
uncertain. Most of these challenges arise due to the properties of robots that
they are physically situated systems.

####  VII-A3 Earth Observation(EO)

Earth Observation (EO) systems are increasingly used to make critical
decisions related to urban planning [334], resource management [335], disaster
response [336], and many more. Right now, there are hundreds of EO satellites
in space, owned by different space agencies and private companies. Figure 12
shows the satellites owned by the European Space Agency (ESA). Like in many
other domains, deep learning has shown great initial success in the field of
EO over the past few years [337]. These early successes consisted of taking
the latest developments of deep learning in computer vision and applying them
to small curated earth observation data sets [337]. At the same time, the
underlying data is very challenging. Even though the amount of data is huge,
so is the variability in the data. This variability is caused by different
sensor types, spatial changes (e.g. different regions and resolutions), and
temporal changes (e.g. changing light conditions, weather conditions,
seasons). Besides the challenge of efficient uncertainty quantification
methods for such large amounts of data, several other challenges that can be
tackled with uncertainty quantification exist in the field of EO. All in all,
the sensitivity of many EO applications together with the nature of EO systems
and the challenging EO data make the quantification of uncertainties very
important in this field. Despite hundreds of publications in the last years on
DL for EO, the range of literature on measuring uncertainties of these systems
is relatively small.

Furthermore, due to the large variation in the data, a data sample received at
test time is often not covered by the training data distribution. For example
while preparing training data for a local climate zone classification, the
human experts might be presented only with images where there is no
obstruction and structures are clearly visible. When a model which is trained
on this data set is deployed in real world, it might see the images with
clouds obstructing the structures or snow giving them a completely different
look. Also, the classes in EO data can have a very wide distribution. For
example, there are millions of types of houses in the world and no training
data can contain the examples for all of them. The question is where the OOD
detector will draw the line and declare the following houses as OOD. Hence,
OOD detection is important in earth observation and uncertainty measurements
play an important part in this [22].

Another common task in EO, where uncertainties can play an important role, is
the data fusion. Optical images normally contain only a few channels like RGB.
In contrast to this, EO data can contain optical images with up to hundreds of
channels, and a variety of different sensors with different spatial, temporal,
and semantic properties. Fusing the information from these different sources
and channels propagates the uncertainties from different sources onto the
prediction. The challenge lies in developing methods that do not only quantify
uncertainties but also the amount of contribution from different channels
individually and which learn to focus on the trustworthy data source for a
given sample [338].

Unlike normal computer vision scenarios where the image acquisition equipment
is quite near to the subject, the EO satellites are hundreds of kilometers
away from the subject. The sensitivity of sensors, the atmospheric absorption
properties, and surface reflectance properties all contribute to uncertainties
in the acquired data. Integrating the knowledge of physical EO systems, which
also contain information about uncertainty models in those systems, is another
major open issue. However, for several applications in EO, measuring
uncertainties is not only something good to have but rather an important
requirement of the field. E.g., the geo-variables derived from EO data may be
assimilated into process models (ocean, hydrological, weather, climate, etc)
and the assimilation requires the probability distribution of the estimated
variables.

Figure 12: European Space Agency (ESA) Developed Earth Observation Missions
[339].

##  VIII Conclusion and Outlook

###  VIII-A Conclusion - How well do the current uncertainty quantification
methods work for real world applications?

Even though many advances on uncertainty quantification in neural networks
have been made over the last years, their adoption in practical mission- and
safety-critical applications is still limited. There are several reasons for
this, which are discussed one-by-one as follows:

  * •

Missing Validation of Existing Methods over Real-World Problems
Although DNNs have become the defacto standard in solving numerous computer
vision and medical image processing tasks, the majority of existing models are
not able to appropriately quantify uncertainty that is inherent to their
inferences particularly in real world applications. This is primarily because
the baseline models are mostly developed using standard data sets such as
Cifar10/100, ImageNet, or well known regression data sets that are specific to
a particular use case and are therefore not readily applicable to complex
real-world environments, as for example low resolutional satellite data or
other data sources affected by noise. Although many researchers from other
fields apply uncertainty quantification in their field [21, 10, 8], a broad
and structured evaluation of existing methods based on different real world
applications is not available yet. Works like [56] already built first steps
towards a real life evaluation.

  * •

Lack of Standardized Evaluation Protocol
Existing methods for evaluating the estimated uncertainty are better suited to
compare uncertainty quantification methods based on measurable quantities such
as the calibration [340] or the performance on out-of-distribution detection
[32]. As described in Section VI, these tests are performed on standardized
sets within the machine learning community. Furthermore, the details of these
experiments might differ in the experimental setting from paper to paper
[214]. However, a clear standardized protocol of tests that should be
performed on uncertainty quantification methods is still not available. For
researchers from other domains it is difficult to directly find state of the
art methods for the field they are interested in, not to speak of the hard
decision on which sub-field of uncertainty quantification to focus. This makes
the direct comparison of the latest approaches difficult and also limits the
acceptance and adoption of current existing methods for uncertainty
quantification.

  * •

Inability to Evaluate Uncertainty Associated to a Single Decision
Existing measures for evaluating the estimated uncertainty (e.g., the expected
calibration error) are based on the whole testing data set. This means, that
equivalent to classification tasks on unbalanced data sets, the uncertainty
associated with single samples or small groups of samples may potentially get
biased towards the performance on the rest of the data set. But for practical
applications, assessing the reliability of a predicted confidence would give
much more possibilities than an aggregated reliability based on some testing
data, which are independent from the current situation [341]. Especially for
mission- and safety-critical applications, pointwise evaluation measures could
be of paramount importance and hence such evaluation approaches are very
desirable.

  * •

Lack of Ground Truth Uncertainties
Current methods are empirically evaluated and the performance is underlined by
reasonable and explainable values of uncertainty. A ground truth uncertainty
that could be used for validation is in general not available. Additionally,
even though existing methods are calibrated on given data sets, one cannot
simply transfer these results to any other data set since one has to be aware
of shifts in the data distribution and that many fields can only cover a tiny
portion of the actual data environment. In application fields as EO, the
preparation of a huge amount of training data is hard and expensive and hence
synthetic data can be used to train a model. For this artificial data,
artificial uncertainties in labels and data should be taken into account to
receive a better understanding of the uncertainty quantification performance.
The gap between the real and synthetic data, or estimated and real uncertainty
further limits the adoption of currently existing methods for uncertainty
quantification.

  * •

Explainability Issue:
Existing methods of neural network uncertainty quantification deliver
predictions of certainty without any clue about what causes possible
uncertainties. Even though those certainty values often look reasonable to a
human observer, one does not know whether the uncertainties are actually
predicted based on the same observations the human observer made. But without
being sure about the reasons and motivations of single uncertainty
estimations, a proper transfer from one data set to another, and even only a
domain shift, are much harder to realize with a guaranteed performance.
Regarding safety critical real life applications, the lack of explainability
makes the application of the available methods significantly harder. Besides
the explainability of neural networks decisions, existing methods for
uncertainty quantification are not well understood on a higher level. For
instance, explaining the behavior of single deterministic approaches,
ensembles or Bayesian methods is a current direction of research and remains
difficult to grasp in every detail [227]. It is, however, crucial to
understand how those methods operate and capture uncertainty to identify
pathways for refinement, detect and characterize uncertainty, failures and
important shortcomings [227].

###  VIII-B Outlook

  * •

Generic Evaluation Framework
As already discussed above, there are still problems regarding the evaluation
of uncertainty methods, as the lack of ’ground truth’ uncertainties, the
inability to test on single instances, and standardized benchmarking
protocols, etc. To cope with such issues, the provision of an evaluation
protocol containing various concrete baseline data sets and evaluation metrics
that cover all types of uncertainty would undoubtedly help to boost research
in uncertainty quantification. Also, the evaluation with regard to risk-averse
and worst case scenarios should be considered there. This means, that
uncertainty predictions with a very high predicted uncertainty should never
fail, as for example for a prediction of a red or green traffic light. Such a
general protocol would enable researchers to easily compare different types of
methods against an established benchmark as well as on real world data sets.
The adoption of such a standard evaluation protocol should be encouraged by
conferences and journals.

  * •

Expert & Systematic Comparison of Baselines
A broad and structured comparison of existing methods for uncertainty
estimation on real world applications is not available yet. An evaluation on
real world data is even not standard in current machine learning research
papers. As a result, given a specific application, it remains unclear which
method for uncertainty estimation performs best and whether the latest methods
outperform older methods also on real world examples. This is also partly
caused by the fact, that researchers from other domains that use uncertainty
quantification methods, in general present successful applications of single
approaches on a specific problem or a data set by hand. Considering this,
there are several points that could be adopted for a better comparison within
the different research domains. For instance, domain experts should also
compare different approaches against each other and present the weaknesses of
single approaches in this domain. Similarly, for a better comparison among
several domains, a collection of all the works in the different real world
domains could be collected and exchanged on a central platform. Such a
platform might also help machine learning researchers in providing an
additional source of challenges in the real world and would pave way to
broadly highlight weaknesses in the current state of the art approaches.
Google’s repository on baselines in uncertainties in neural networks
[340]666https://github.com/google/uncertainty-baselines could be such a
platform and a step towards achieving this goal.

  * •

Uncertainty Ground Truths
It remains difficult to validate existing methods due to the lack of
uncertainty ground truths. An actual uncertainty ground truth on which methods
can be compared in an ImageNet like manner would make the evaluation of
predictions on single samples possible. To reach this, the evaluation of the
data generation process and occurring sources of uncertainty, as for example
the labeling process, might be investigated in more detail.

  * •

Explainability and Physical Models
Knowing the actual reasons for a false high certainty or a low certainty makes
it much easier to engineer the methods for real life applications, which again
increases the trust of people into such methods. Recently, Antorán et al.
[342] claimed to have published the first work on explainable uncertainty
estimation. Uncertainty estimations, in general, form an important step
towards explainable artificial intelligence. Explainable uncertainty
estimations would give an even deeper understanding of the decision process of
a neural network, which, in practical deployment of DNNs, shall incorporate
the desired ability to be risk averse while staying applicable in real world
(especially safety critical applications). Also, the possibility of improving
explainability with physically based arguments offers great potential. While
DNNs are very flexible and efficient, they do not directly embed the domain
specific expert knowledge that is mostly available and can often be described
by mathematical or physical models, as for example earth system science
problems [343]. Such physic guided models offer a variety of possibilities to
include explicit knowledge as well as practical uncertainty representations
into a deep learning framework [344, 345].

## References

  * [1] T. Nair, D. Precup, D. L. Arnold, and T. Arbel, “Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation,” _Medical image analysis_ , vol. 59, p. 101557, 2020.
  * [2] A. G. Roy, S. Conjeti, N. Navab, C. Wachinger, A. D. N. Initiative _et al._ , “Bayesian quicknat: Model uncertainty in deep whole-brain segmentation for structure-wise quality control,” _NeuroImage_ , vol. 195, pp. 11–22, 2019.
  * [3] P. Seeböck, J. I. Orlando, T. Schlegl, S. M. Waldstein, H. Bogunović, S. Klimscha, G. Langs, and U. Schmidt-Erfurth, “Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct,” _IEEE transactions on medical imaging_ , vol. 39, no. 1, pp. 87–98, 2019\.
  * [4] T. LaBonte, C. Martinez, and S. A. Roberts, “We know where we don’t know: 3d bayesian cnns for credible geometric uncertainty,” _arXiv preprint arXiv:1910.10793_ , 2019.
  * [5] J. C. Reinhold, Y. He, S. Han, Y. Chen, D. Gao, J. Lee, J. L. Prince, and A. Carass, “Validating uncertainty in medical image translation,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 95–98.
  * [6] S. Eggenreich, C. Payer, M. Urschler, and D. Štern, “Variational inference and bayesian cnns for uncertainty estimation in multi-factorial bone age prediction,” _arXiv preprint arXiv:2002.10819_ , 2020.
  * [7] D. Feng, L. Rosenbaum, and K. Dietmayer, “Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 3266–3273.
  * [8] J. Choi, D. Chun, H. Kim, and H.-J. Lee, “Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 502–511.
  * [9] A. Amini, A. Soleimany, S. Karaman, and D. Rus, “Spatial uncertainty sampling for end-to-end control,” _arXiv preprint arXiv:1805.04829_ , 2018.
  * [10] A. Loquercio, M. Segu, and D. Scaramuzza, “A general framework for uncertainty estimation in deep learning,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3153–3160, 2020.
  * [11] K. Lee, H. Lee, K. Lee, and J. Shin, “Training confidence-calibrated classifiers for detecting out-of-distribution samples,” in _International Conference on Learning Representations_ , 2018.
  * [12] J. Mitros and B. Mac Namee, “On the validity of bayesian neural networks for uncertainty estimation,” _arXiv preprint arXiv:1912.01530_ , 2019.
  * [13] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 991–14 002.
  * [14] M. S. Ayhan and P. Berens, “Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks,” in _Medical Imaging with Deep Learning Conference_ , 2018.
  * [15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1321–1330.
  * [16] A. G. Wilson and P. Izmailov, “Bayesian deep learning and a probabilistic perspective of generalization,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 4697–4708.
  * [17] M. Rawat, M. Wistuba, and M.-I. Nicolae, “Harnessing model uncertainty for detecting adversarial examples,” in _NIPS Workshop on Bayesian Deep Learning_ , 2017.
  * [18] A. C. Serban, E. Poll, and J. Visser, “Adversarial examples-a complete characterisation of the phenomenon,” _arXiv preprint arXiv:1810.01185_ , 2018\.
  * [19] L. Smith and Y. Gal, “Understanding measures of uncertainty for adversarial example detection,” in _Proceesings of the Conference on Uncertainty in Artificial Intelligence_ , 2018, pp. 560–569.
  * [20] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_ , 2016, pp. 1050–1059.
  * [21] M. Rußwurm, S. M. Ali, X. X. Zhu, Y. Gal, and M. Körner, “Model and data uncertainty for satellite time series forecasting with deep recurrent models,” in _2020 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_ , 2020.
  * [22] J. Gawlikowski, S. Saha, A. Kruspe, and X. X. Zhu, “Out-of-distribution detection in satellite image classification,” in _RobustML workshop at ICLR 2021_. ICRL, 2021, pp. 1–5.
  * [23] Y. Gal, R. Islam, and Z. Ghahramani, “Deep bayesian active learning with image data,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1183–1192.
  * [24] K. Chitta, J. M. Alvarez, and A. Lesnikowski, “Large-scale visual active learning with deep probabilistic ensembles,” _arXiv preprint arXiv:1811.03575_ , 2018.
  * [25] J. Zeng, A. Lesnikowski, and J. M. Alvarez, “The relevance of bayesian layer positioning to model uncertainty in deep bayesian active learning,” _arXiv preprint arXiv:1811.12535_ , 2018.
  * [26] V.-L. Nguyen, S. Destercke, and E. Hüllermeier, “Epistemic uncertainty sampling,” in _International Conference on Discovery Science_. Springer, 2019, pp. 72–86.
  * [27] W. Huang, J. Zhang, and K. Huang, “Bootstrap estimated uncertainty of the environment model for model-based reinforcement learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3870–3877.
  * [28] G. Kahn, A. Villaflor, V. Pong, P. Abbeel, and S. Levine, “Uncertainty-aware reinforcement learning for collision avoidance,” _arXiv preprint arXiv:1702.01182_ , 2017.
  * [29] B. Lötjens, M. Everett, and J. P. How, “Safe reinforcement learning with model uncertainty estimates,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 8662–8668.
  * [30] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural networks,” in _Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37_ , 2015, pp. 1613–1622.
  * [31] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in neural information processing systems_ , 2017, pp. 6402–6413.
  * [32] A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7047–7058.
  * [33] X. Zhao, Y. Ou, L. Kaplan, F. Chen, and J.-H. Cho, “Quantifying classification uncertainty using regularized evidential neural networks,” _arXiv preprint arXiv:1910.06864_ , 2019.
  * [34] Q. Wu, H. Li, W. Su, L. Li, and Z. Yu, “Quantifying intrinsic uncertainty in classification via deep dirichlet mixture networks,” _arXiv preprint arXiv:1906.04450_ , 2019.
  * [35] J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal, “Uncertainty estimation using a single deep deterministic neural network,” in _Proceedings of the 37th International Conference on Machine Learning_. PMLR, 2020, pp. 9690–9700.
  * [36] T. Ramalho and M. Miranda, “Density estimation in representation space to predict model uncertainty,” in _Engineering Dependable and Secure Machine Learning Systems: Third International Workshop, EDSMLS 2020, New York City, NY, USA, February 7, 2020, Revised Selected Papers_ , vol. 1272. Springer Nature, 2020, p. 84.
  * [37] A. Mobiny, H. V. Nguyen, S. Moulik, N. Garg, and C. C. Wu, “Dropconnect is effective in modeling uncertainty of bayesian deep networks,” _arXiv preprint arXiv:1906.04569_ , 2019.
  * [38] D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. Lacoste, and A. Courville, “Bayesian hypernetworks,” _arXiv preprint arXiv:1710.04759_ , 2017.
  * [39] M. Valdenegro-Toro, “Deep sub-ensembles for fast uncertainty estimation in image classification,” in _Bayesian Deep Learning Workshop at Neural Information Processing Systems 2019_ , 2019.
  * [40] Y. Wen, D. Tran, and J. Ba, “Batchensemble: an alternative approach to efficient ensemble and lifelong learning,” in _8th International Conference on Learning Representations_ , 2020.
  * [41] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of Big Data_ , vol. 6, no. 1, pp. 1–48, 2019.
  * [42] Q. Wen, L. Sun, X. Song, J. Gao, X. Wang, and H. Xu, “Time series data augmentation for deep learning: A survey,” _arXiv preprint arXiv:2002.12478_ , 2020.
  * [43] T. Tsiligkaridis, “Information robust dirichlet networks for predictive uncertainty estimation,” _arXiv preprint arXiv:1910.04819_ , 2019.
  * [44] M. Sensoy, L. Kaplan, and M. Kandemir, “Evidential deep learning to quantify classification uncertainty,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 3179–3189.
  * [45] A. Malinin, B. Mlodozeniec, and M. Gales, “Ensemble distribution distillation,” in _8th International Conference on Learning Representations_ , 2020.
  * [46] M. Raghu, K. Blumer, R. Sayres, Z. Obermeyer, B. Kleinberg, S. Mullainathan, and J. Kleinberg, “Direct uncertainty prediction for medical second opinions,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5281–5290.
  * [47] J. Wenger, H. Kjellström, and R. Triebel, “Non-parametric calibration for classification,” in _International Conference on Artificial Intelligence and Statistics_ , 2020, pp. 178–190.
  * [48] J. Zhang, B. Kailkhura, and T. Y.-J. Han, “Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 11 117–11 128.
  * [49] R. Ghanem, D. Higdon, and H. Owhadi, _Handbook of uncertainty quantification_. Springer, 2017, vol. 6.
  * [50] Y. Gal, “Uncertainty in deep learning,” Ph.D. dissertation, University of Cambridge, 2016.
  * [51] A. G. Kendall, “Geometry and uncertainty in deep learning for computer vision,” Ph.D. dissertation, University of Cambridge, 2019.
  * [52] A. Malinin, “Uncertainty estimation in deep learning with application to spoken language assessment,” Ph.D. dissertation, University of Cambridge, 2019\.
  * [53] H. Wang and D.-Y. Yeung, “Towards bayesian deep learning: A framework and some existing methods,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 28, no. 12, pp. 3395–3408, 2016.
  * [54] ——, “A survey on bayesian deep learning,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 5, pp. 1–37, 2020.
  * [55] N. Ståhl, G. Falkman, A. Karlsson, and G. Mathiason, “Evaluation of uncertainty quantification in deep learning,” in _Information Processing and Management of Uncertainty in Knowledge-Based Systems_. Springer International Publishing, 2020, pp. 556–568.
  * [56] F. K. Gustafsson, M. Danelljan, and T. B. Schon, “Evaluating scalable bayesian deep learning methods for robust computer vision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 318–319.
  * [57] E. Hüllermeier and W. Waegeman, “Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods,” _Machine Learning_ , vol. 110, no. 3, pp. 457–506, 2021.
  * [58] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya _et al._ , “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” _Information Fusion_ , 2021.
  * [59] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner _et al._ , “Relational inductive biases, deep learning, and graph networks,” _arXiv preprint arXiv:1806.01261_ , 2018.
  * [60] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in _Advances in neural information processing systems_ , 2017, pp. 5574–5584.
  * [61] Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with bernoulli approximate variational inference,” _arXiv preprint arXiv:1506.02158_ , 2015.
  * [62] C. Bishop, _Pattern Recognition and Machine Learning_. Springer-Verlag New York, 2006.
  * [63] H. Ritter, A. Botev, and D. Barber, “A scalable laplace approximation for neural networks,” in _6th International Conference on Learning Representations_ , vol. 6. International Conference on Representation Learning, 2018.
  * [64] J. Nandy, W. Hsu, and M. L. Lee, “Towards maximizing the representation gap between in-domain & out-of-distribution examples,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., 2020, pp. 9239–9250.
  * [65] A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov, “Pitfalls of in-domain uncertainty estimation and ensembling in deep learning,” in _International Conference on Learning Representations_ , 2020.
  * [66] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [67] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in _5th International Conference on Learning Representations_ , 2017.
  * [68] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in _6th International Conference on Learning Representations_ , 2018.
  * [69] A. Shafaei, M. Schmidt, and J. J. Little, “A less biased evaluation of out-of-distribution sample detectors,” in _British Machine Vision Conference 2019_ , 2019.
  * [70] M. Mundt, I. Pliushch, S. Majumder, and V. Ramesh, “Open set recognition through deep neural network uncertainty: Does out-of-distribution detection require generative classifiers?” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2019.
  * [71] P. Oberdiek, M. Rottmann, and H. Gottschalk, “Classification uncertainty of deep neural networks based on gradient information,” in _IAPR Workshop on Artificial Neural Networks in Pattern Recognition_. Springer, 2018, pp. 113–125.
  * [72] J. Lee and G. AlRegib, “Gradients as a measure of uncertainty in neural networks,” in _2020 IEEE International Conference on Image Processing_. IEEE, 2020, pp. 2416–2420.
  * [73] G. E. Hinton and D. Van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in _Proceedings of the sixth annual conference on Computational learning theory_ , 1993, pp. 5–13.
  * [74] D. Barber and C. M. Bishop, “Ensemble learning in bayesian neural networks,” _Nato ASI Series F Computer and Systems Sciences_ , vol. 168, pp. 215–238, 1998.
  * [75] A. Graves, “Practical variational inference for neural networks,” in _Advances in neural information processing systems_ , 2011, pp. 2348–2356.
  * [76] C. Louizos, K. Ullrich, and M. Welling, “Bayesian compression for deep learning,” in _Advances in neural information processing systems_ , 2017, pp. 3288–3298.
  * [77] D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in _International Conference on Machine Learning_ , 2015, pp. 1530–1538.
  * [78] R. M. Neal, “Bayesian training of backpropagation networks by the hybrid monte carlo method,” Citeseer, Tech. Rep., 1992.
  * [79] ——, “An improved acceptance procedure for the hybrid monte carlo algorithm,” _Journal of Computational Physics_ , vol. 111, no. 1, pp. 194–203, 1994.
  * [80] ——, “Bayesian learning for neural networks,” Ph.D. dissertation, University of Toronto, 1995.
  * [81] M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in _Proceedings of the 28th international conference on machine learning_ , 2011, pp. 681–688.
  * [82] C. Nemeth and P. Fearnhead, “Stochastic gradient markov chain monte carlo,” _Journal of the American Statistical Association_ , pp. 1–18, 2020.
  * [83] T. Salimans and D. P. Kingma, “Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,” in _Advances in Neural Information Processing Systems 29_. Curran Associates, Inc., 2016, pp. 901–909.
  * [84] J. Lee, M. Humt, J. Feng, and R. Triebel, “Estimating model uncertainty of neural networks in sparse information form,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5702–5713.
  * [85] O. Achrack, O. Barzilay, and R. Kellerman, “Multi-loss sub-ensembles for accurate classification with uncertainty estimation,” _arXiv preprint arXiv:2010.01917_ , 2020.
  * [86] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” in _International conference on learning representations_ , 2017.
  * [87] G. D. Cavalcanti, L. S. Oliveira, T. J. Moura, and G. V. Carvalho, “Combining diversity measures for ensemble pruning,” _Pattern Recognition Letters_ , vol. 74, pp. 38–45, 2016.
  * [88] H. Guo, H. Liu, R. Li, C. Wu, Y. Guo, and M. Xu, “Margin & diversity based ordering ensemble pruning,” _Neurocomputing_ , vol. 275, pp. 237–246, 2018\.
  * [89] W. G. Martinez, “Ensemble pruning via quadratic margin maximization,” _IEEE Access_ , vol. 9, pp. 48 931–48 951, 2021.
  * [90] J. Lindqvist, A. Olmin, F. Lindsten, and L. Svensson, “A general framework for ensemble distribution distillation,” in _2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2020, pp. 1–6.
  * [91] D. Molchanov, A. Lyzhov, Y. Molchanova, A. Ashukha, and D. Vetrov, “Greedy policy search: A simple baseline for learnable test-time augmentation,” _arXiv preprint arXiv:2002.09103_ , vol. 2, no. 7, 2020.
  * [92] M. Możejko, M. Susik, and R. Karczewski, “Inhibited softmax for uncertainty estimation in neural networks,” _arXiv preprint arXiv:1810.01861_ , 2018.
  * [93] L. Oala, C. Heiß, J. Macdonald, M. März, W. Samek, and G. Kutyniok, “Interval neural networks: Uncertainty scores,” _arXiv preprint arXiv:2003.11566_ , 2020.
  * [94] A. Malinin and M. Gales, “Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buc, E. Fox, and R. Garnett, Eds., 2019, pp. 14 547–14 558.
  * [95] V. T. Vasudevan, A. Sethy, and A. R. Ghias, “Towards better confidence estimation for neural models,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 7335–7339.
  * [96] M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 41–50.
  * [97] T. Joo, U. Chung, and M.-G. Seo, “Being bayesian about categorical probability,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 4950–4961.
  * [98] T. Tsiligkaridis, “Failure prediction by confidence estimation of uncertainty-aware dirichlet networks,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2021, pp. 3525–3529.
  * [99] ——, “Information robust dirichlet networks for predictive uncertainty estimation,” _arXiv preprint arXiv:1910.04819_ , 2019.
  * [100] A. P. Dempster, “A generalization of bayesian inference,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 30, no. 2, pp. 205–232, 1968.
  * [101] A. Amini, W. Schwarting, A. Soleimany, and D. Rus, “Deep evidential regression,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 14 927–14 937.
  * [102] B. Charpentier, D. Zügner, and S. Günnemann, “Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 1356–1367.
  * [103] N. Tagasovska and D. Lopez-Paz, “Single-model uncertainties for deep learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 6417–6428.
  * [104] T. Kawashima, Q. Yu, A. Asai, D. Ikami, and K. Aizawa, “The aleatoric uncertainty estimation using a separate formulation with virtual residuals,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 1438–1445.
  * [105] Y.-C. Hsu, Y. Shen, H. Jin, and Z. Kira, “Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 951–10 960.
  * [106] J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield, “Large automatic learning, rule extraction, and generalization,” _Complex systems_ , vol. 1, no. 5, pp. 877–922, 1987.
  * [107] N. Tishby, E. Levin, and S. A. Solla, “Consistent inference of probabilities in layered networks: Predictions and generalization,” in _International Joint Conference on Neural Networks_ , vol. 2, 1989, pp. 403–409.
  * [108] W. L. Buntine and A. S. Weigend, “Bayesian back-propagation,” _Complex systems_ , vol. 5, no. 6, pp. 603–643, 1991.
  * [109] D. J. C. MacKay, “Bayesian model comparison and backprop nets,” in _Advances in neural information processing systems_ , 1992, pp. 839–846.
  * [110] M.-A. Sato, “Online model selection based on the variational bayes,” _Neural computation_ , vol. 13, no. 7, pp. 1649–1681, 2001.
  * [111] A. Corduneanu and C. M. Bishop, “Variational bayesian model selection for mixture distributions,” in _Artificial intelligence and Statistics_ , vol. 2001. Morgan Kaufmann Waltham, MA, 2001, pp. 27–34.
  * [112] S. Ghosh, J. Yao, and F. Doshi-Velez, “Model selection in bayesian neural networks via horseshoe priors,” _Journal of Machine Learning Research_ , vol. 20, no. 182, pp. 1–46, 2019.
  * [113] M. Federici, K. Ullrich, and M. Welling, “Improved bayesian compression,” _arXiv preprint arXiv:1711.06494_ , 2017.
  * [114] J. Achterhold, J. M. Koehler, A. Schmeink, and T. Genewein, “Variational network quantization,” in _International Conference on Learning Representations_ , 2018.
  * [115] D. J. MacKay, “Information-based objective functions for active data selection,” _Neural computation_ , vol. 4, no. 4, pp. 590–604, 1992.
  * [116] A. Kirsch, J. van Amersfoort, and Y. Gal, “Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 7026–7037.
  * [117] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” in _International Conference on Learning Representations_ , 2018\.
  * [118] S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach, “Uncertainty-guided continual learning with bayesian neural networks,” in _International Conference on Learning Representations_ , 2020.
  * [119] S. Farquhar and Y. Gal, “A unifying bayesian view of continual learning,” _arXiv preprint arXiv:1902.06494_ , 2019.
  * [120] H. Li, P. Barnaghi, S. Enshaeifar, and F. Ganz, “Continual learning using bayesian neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [121] M. E. E. Khan, A. Immer, E. Abedi, and M. Korzepa, “Approximate inference turns deep networks into gaussian processes,” in _Advances in neural information processing systems_ , 2019, pp. 3094–3104.
  * [122] J. S. Denker and Y. LeCun, “Transforming neural-net output levels to probability distributions,” in _Advances in neural information processing systems_ , 1991, pp. 853–859.
  * [123] D. J. MacKay, “A practical bayesian framework for backpropagation networks,” _Neural computation_ , vol. 4, no. 3, pp. 448–472, 1992.
  * [124] J. Hernandez-Lobato, Y. Li, M. Rowland, T. Bui, D. Hernández-Lobato, and R. Turner, “Black-box alpha divergence minimization,” in _International Conference on Machine Learning_ , 2016, pp. 1511–1520.
  * [125] Y. Li and Y. Gal, “Dropout inference in bayesian neural networks with alpha-divergences,” in _International Conference on Machine Learning_ , 2017, pp. 2052–2061.
  * [126] T. Minka _et al._ , “Divergence measures and message passing,” Technical report, Microsoft Research, Tech. Rep., 2005.
  * [127] T. P. Minka, “Expectation propagation for approximate bayesian inference,” in _Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence_ , 2001, pp. 362–369.
  * [128] J. Zhao, X. Liu, S. He, and S. Sun, “Probabilistic inference of bayesian neural networks with generalized expectation propagation,” _Neurocomputing_ , vol. 412, pp. 392–398, 2020.
  * [129] J. M. Hernández-Lobato and R. Adams, “Probabilistic backpropagation for scalable learning of bayesian neural networks,” in _International Conference on Machine Learning_ , 2015, pp. 1861–1869.
  * [130] D. Tran, A. Kucukelbir, A. B. Dieng, M. Rudolph, D. Liang, and D. M. Blei, “Edward: A library for probabilistic modeling, inference, and criticism,” _arXiv preprint arXiv:1610.09787_ , 2016.
  * [131] D. Tran, M. D. Hoffman, R. A. Saurous, E. Brevdo, K. Murphy, and D. M. Blei, “Deep probabilistic programming,” in _International Conference on Machine Learning_ , 2016.
  * [132] E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip, P. Horsfall, and N. D. Goodman, “Pyro: Deep universal probabilistic programming,” _The Journal of Machine Learning Research_ , vol. 20, no. 1, pp. 973–978, 2019.
  * [133] R. Cabañas, A. Salmerón, and A. R. Masegosa, “Inferpy: Probabilistic modeling with tensorflow made easy,” _Knowledge-Based Systems_ , vol. 168, pp. 25–27, 2019.
  * [134] Y. Ito, C. Srinivasan, and H. Izumi, “Bayesian learning of neural networks adapted to changes of prior probabilities,” in _International Conference on Artificial Neural Networks_. Springer, 2005, pp. 253–259.
  * [135] S. Sun, G. Zhang, J. Shi, and R. Grosse, “Functional variational bayesian neural networks,” in _International Conference on Learning Representations_ , 2018.
  * [136] S. Depeweg, J. M. Hernández-Lobato, S. Udluft, and T. Runkler, “Sensitivity analysis for predictive uncertainty in bayesian neural networks,” _arXiv preprint arXiv:1712.03605_ , 2017.
  * [137] S. Farquhar, L. Smith, and Y. Gal, “Try depth instead of weight correlations: Mean-field is a less restrictive assumption for deeper networks,” _arXiv preprint arXiv:2002.03704_ , 2020.
  * [138] J. Postels, F. Ferroni, H. Coskun, N. Navab, and F. Tombari, “Sampling-free epistemic uncertainty estimation using approximated variance propagation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2931–2940.
  * [139] J. Gast and S. Roth, “Lightweight probabilistic deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3369–3378.
  * [140] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft, “Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1184–1193.
  * [141] ——, “Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 1184–1193.
  * [142] D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,” in _Advances in neural information processing systems_ , 2015, pp. 2575–2583.
  * [143] A. Wu, S. Nowozin, E. Meeds, R. Turner, J. Hernández-Lobato, and A. Gaunt, “Deterministic variational inference for robust bayesian neural networks,” in _7th International Conference on Learning Representations, ICLR 2019_ , 2019.
  * [144] C. Louizos and M. Welling, “Structured and efficient variational deep learning with matrix gaussian posteriors,” in _International Conference on Machine Learning_ , 2016, pp. 1708–1716.
  * [145] G. Zhang, S. Sun, D. Duvenaud, and R. Grosse, “Noisy natural gradient as variational inference,” in _International Conference on Machine Learning_ , 2018, pp. 5852–5861.
  * [146] S. Sun, C. Chen, and L. Carin, “Learning structured weight uncertainty in bayesian neural networks,” in _Artificial Intelligence and Statistics_ , 2017, pp. 1283–1292.
  * [147] J. Bae, G. Zhang, and R. Grosse, “Eigenvalue corrected noisy natural gradient,” _arXiv preprint arXiv:1811.12565_ , 2018.
  * [148] A. Mishkin, F. Kunstner, D. Nielsen, M. Schmidt, and M. E. Khan, “Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 6245–6255.
  * [149] C. Louizos and M. Welling, “Multiplicative normalizing flows for variational bayesian neural networks,” in _International Conference on Machine Learning_ , 2017, pp. 2218–2227.
  * [150] K. Osawa, S. Swaroop, M. E. E. Khan, A. Jain, R. Eschenhagen, R. E. Turner, and R. Yokota, “Practical deep learning with bayesian principles,” in _Advances in neural information processing systems_ , 2019, pp. 4287–4299.
  * [151] Y. Gal, J. Hron, and A. Kendall, “Concrete dropout,” in _Advances in neural information processing systems_ , 2017, pp. 3581–3590.
  * [152] Z. Eaton-Rosen, F. Bragman, S. Bisdas, S. Ourselin, and M. J. Cardoso, “Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 691–699.
  * [153] C. R. N. Tassi, “Bayesian convolutional neural network: Robustly quantify uncertainty for misclassifications detection,” in _Mediterranean Conference on Pattern Recognition and Artificial Intelligence_. Springer, 2019, pp. 118–132.
  * [154] P. McClure and N. Kriegeskorte, “Robustly representing uncertainty through sampling in deep neural networks,” _arXiv preprint arXiv:1611.01639_ , 2016\.
  * [155] M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava, “Fast and scalable bayesian deep learning by weight-perturbation in adam,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2611–2620.
  * [156] M. E. Khan, Z. Liu, V. Tangkaratt, and Y. Gal, “Vprop: Variational inference using rmsprop,” in _Advances in neural information processing systems_ , 2017, pp. 3288–3298.
  * [157] A. Atanov, A. Ashukha, D. Molchanov, K. Neklyudov, and D. Vetrov, “Uncertainty estimation via stochastic batch normalization,” in _International Symposium on Neural Networks_. Springer, 2019, pp. 261–269.
  * [158] S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth, “Hybrid monte carlo,” _Physics letters B_ , vol. 195, no. 2, pp. 216–222, 1987.
  * [159] R. M. Neal _et al._ , “Mcmc using hamiltonian dynamics,” _Handbook of markov chain monte carlo_ , vol. 2, no. 11, p. 2, 2011.
  * [160] K. A. Dubey, S. J. Reddi, S. A. Williamson, B. Poczos, A. J. Smola, and E. P. Xing, “Variance reduction in stochastic gradient langevin dynamics,” in _Advances in neural information processing systems_ , 2016, pp. 1154–1162.
  * [161] B. Leimkuhler and S. Reich, _Simulating hamiltonian dynamics_. Cambridge university press, 2004, vol. 14.
  * [162] P. J. Rossky, J. Doll, and H. Friedman, “Brownian dynamics as smart monte carlo simulation,” _The Journal of Chemical Physics_ , vol. 69, no. 10, pp. 4628–4633, 1978.
  * [163] G. O. Roberts and O. Stramer, “Langevin diffusions and metropolis-hastings algorithms,” _Methodology and computing in applied probability_ , vol. 4, no. 4, pp. 337–357, 2002.
  * [164] H. Kushner and G. G. Yin, _Stochastic approximation and recursive algorithms and applications_. Springer Science & Business Media, 2003, vol. 35.
  * [165] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1, no. 2.
  * [166] Y.-A. Ma, T. Chen, and E. Fox, “A complete recipe for stochastic gradient mcmc,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 2917–2925.
  * [167] G. Marceau-Caron and Y. Ollivier, “Natural langevin dynamics for neural networks,” in _International Conference on Geometric Science of Information_. Springer, 2017, pp. 451–459.
  * [168] Z. Nado, J. Snoek, R. B. Grosse, D. Duvenaud, B. Xu, and J. Martens, “Stochastic gradient langevin dynamics that exploit neural network structure,” in _International Conference on Learning Representations (Workshop)_ , 2018.
  * [169] U. Simsekli, R. Badeau, A. T. Cemgil, and G. Richard, “Stochastic quasi-newton langevin monte carlo,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48_ , 2016, pp. 642–651.
  * [170] Y. Zhang and C. A. Sutton, “Quasi-newton methods for markov chain monte carlo,” in _Advances in Neural Information Processing Systems_ , 2011, pp. 2393–2401.
  * [171] T. Fu, L. Luo, and Z. Zhang, “Quasi-newton hamiltonian monte carlo.” in _Conference on Uncertainty in Artificial Intelligence_ , 2016.
  * [172] C. Li, C. Chen, D. Carlson, and L. Carin, “Preconditioned stochastic gradient langevin dynamics for deep neural networks,” in _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_. AAAI Press, 2016, pp. 1788–1794.
  * [173] S. Ahn, A. K. Balan, and M. Welling, “Bayesian posterior sampling via stochastic gradient fisher scoring,” in _International Conference on Learning Representations_ , 2012.
  * [174] S. Patterson and Y. W. Teh, “Stochastic gradient riemannian langevin dynamics on the probability simplex,” in _Advances in neural information processing systems_ , 2013, pp. 3102–3110.
  * [175] N. Ye and Z. Zhu, “Stochastic fractional hamiltonian monte carlo,” in _IJCAI_ , 2018, pp. 3019–3025.
  * [176] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven, “Bayesian sampling using stochastic gradient thermostats,” in _Advances in neural information processing systems_ , 2014, pp. 3203–3211.
  * [177] X. Shang, Z. Zhu, B. Leimkuhler, and A. J. Storkey, “Covariance-controlled adaptive langevin thermostat for large-scale bayesian sampling,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 37–45.
  * [178] B. Leimkuhler and X. Shang, “Adaptive thermostats for noisy gradient systems,” _SIAM Journal on Scientific Computing_ , vol. 38, no. 2, pp. A712–A736, 2016.
  * [179] S. Ahn, B. Shahbaba, and M. Welling, “Distributed stochastic gradient mcmc,” in _International conference on machine learning_ , 2014, pp. 1044–1052.
  * [180] K.-C. Wang, P. Vicol, J. Lucas, L. Gu, R. Grosse, and R. Zemel, “Adversarial distillation of bayesian neural network posteriors,” in _International Conference on Machine Learning_ , 2018, pp. 5190–5199.
  * [181] A. K. Balan, V. Rathod, K. P. Murphy, and M. Welling, “Bayesian dark knowledge,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 3438–3446.
  * [182] D. Zou, P. Xu, and Q. Gu, “Stochastic variance-reduced hamilton monte carlo methods,” in _International Conference on Machine Learning_ , 2018, pp. 6028–6037.
  * [183] A. Durmus, U. Simsekli, E. Moulines, R. Badeau, and G. Richard, “Stochastic gradient richardson-romberg markov chain monte carlo,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 2047–2055.
  * [184] A. Durmus, E. Moulines _et al._ , “High-dimensional bayesian inference via the unadjusted langevin algorithm,” _Bernoulli_ , vol. 25, no. 4A, pp. 2854–2882, 2019.
  * [185] I. Sato and H. Nakagawa, “Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process,” in _International Conference on Machine Learning_ , 2014, pp. 982–990.
  * [186] C. Chen, N. Ding, and L. Carin, “On the convergence of stochastic gradient mcmc algorithms with high-order integrators,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 2278–2286.
  * [187] Y. W. Teh, A. H. Thiery, and S. J. Vollmer, “Consistency and fluctuations for stochastic gradient langevin dynamics,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 193–225, 2016.
  * [188] C. Li, A. Stevens, C. Chen, Y. Pu, Z. Gan, and L. Carin, “Learning weight uncertainty with stochastic gradient mcmc for shape classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5666–5675.
  * [189] F. Wenzel, K. Roth, B. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin, “How good is the bayes posterior in deep neural networks really?” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 248–10 259.
  * [190] N. Ye, Z. Zhu, and R. K. Mantiuk, “Langevin dynamics with continuous tempering for training deep neural networks,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017, pp. 618–626.
  * [191] R. Chandra, K. Jain, R. V. Deo, and S. Cripps, “Langevin-gradient parallel tempering for bayesian neural learning,” _Neurocomputing_ , vol. 359, pp. 315–326, 2019.
  * [192] A. Botev, H. Ritter, and D. Barber, “Practical gauss-newton optimisation for deep learning,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , 2017, pp. 557–565.
  * [193] J. Martens and R. Grosse, “Optimizing neural networks with kronecker-factored approximate curvature,” in _Proceeding of the 32nd International conference on machine learning_ , 2015, pp. 2408–2417.
  * [194] S. Becker and Y. LeCun, “Improving the convergence of back-propagation learning with second-order methods,” in _Proceedings of the 1988 Connectionist Models Summer School, San Mateo_ , D. Touretzky, G. Hinton, and T. Sejnowski, Eds. Morgan Kaufmann, 1989, pp. 29–37.
  * [195] D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large scale optimization,” _Mathematical Programming_ , vol. 45, pp. 503–528, 08 1989\.
  * [196] P. Hennig, “Fast probabilistic optimization from noisy gradients,” in _Proceedings of the 30th International Conference on Machine Learning_ , vol. 28-1. PMLR, 2013, pp. 62–70.
  * [197] N. L. Roux and A. W. Fitzgibbon, “A fast natural newton method,” in _Proceedings of the International Conference on Machine Learning_ , 2010.
  * [198] R. B. Grosse and J. Martens, “A kronecker-factored approximate fisher matrix for convolution layers,” in _Proceedings of the 33nd International Conference on Machine Learning_ , 2016, pp. 573–582.
  * [199] S.-W. Chen, C.-N. Chou, and E. Chang, “Bda-pch: Block-diagonal approximation of positive-curvature hessian for training neural networks,” _CoRR, abs/1802.06502_ , 2018.
  * [200] J. Ba, R. Grosse, and J. Martens, “Distributed second-order optimization using kronecker-factored approximations,” in _International Conference on Learning Representations_ , 2017.
  * [201] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent, “Fast approximate natural gradient descent in a kronecker factored eigenbasis,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 9573–9583.
  * [202] Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in _Advances in Neural Information Processing Systems 2_ , 1990, pp. 598–605.
  * [203] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [204] A. Kristiadi, M. Hein, and P. Hennig, “Being bayesian, even just a bit, fixes overconfidence in relu networks,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5436–5446.
  * [205] M. Humt, J. Lee, and R. Triebel, “Bayesian optimization meets laplace approximation for robotic introspection,” _arXiv preprint arXiv:2010.16141_ , 2020.
  * [206] A. Kristiadi, M. Hein, and P. Hennig, “Learnable uncertainty under laplace approximations,” _arXiv preprint arXiv:2010.02720_ , 2020.
  * [207] K. Shinde, J. Lee, M. Humt, A. Sezgin, and R. Triebel, “Learning multiplicative interactions with bayesian neural networks for visual-inertial odometry,” in _Workshop on AI for Autonomous Driving at the 37th International Conference on Machine Learning_ , 2020.
  * [208] J. Feng, M. Durner, Z.-C. Marton, F. Balint-Benczedi, and R. Triebel, “Introspective robot perception using smoothed predictions from bayesian neural networks,” in _International Symposium on Robotics Research_ , 2019\.
  * [209] A. Y. Foong, Y. Li, J. M. Hernández-Lobato, and R. E. Turner, “’in-between’ uncertainty in bayesian neural networks,” _arXiv preprint arXiv:1906.11537_ , 2019.
  * [210] A. Immer, M. Korzepa, and M. Bauer, “Improving predictions of bayesian neural nets via local linearization,” in _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_. PMLR, 2021, pp. 703–711.
  * [211] M. Hobbhahn, A. Kristiadi, and P. Hennig, “Fast predictive uncertainty for classification with bayesian deep networks,” _arXiv preprint arXiv:2003.01227_ , 2020.
  * [212] E. Daxberger, E. Nalisnick, J. U. Allingham, J. Antorán, and J. M. Hernández-Lobato, “Expressive yet tractable bayesian deep learning via subnetwork inference,” _arXiv preprint arXiv:2010.14689_ , 2020.
  * [213] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson, “A simple baseline for bayesian uncertainty in deep learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 153–13 164.
  * [214] J. Mukhoti, P. Stenetorp, and Y. Gal, “On the importance of strong baselines in bayesian deep learning,” _arXiv preprint arXiv:1811.09385_ , 2018.
  * [215] A. Filos, S. Farquhar, A. N. Gomez, T. G. Rudner, Z. Kenton, L. Smith, M. Alizadeh, A. de Kroon, and Y. Gal, “A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks,” _arXiv preprint arXiv:1912.10481_ , 2019.
  * [216] J. Mukhoti and Y. Gal, “Evaluating bayesian deep learning methods for semantic segmentation,” _arXiv preprint arXiv:1811.12709_ , 2018.
  * [217] O. Sagi and L. Rokach, “Ensemble learning: A survey,” _WIREs Data Mining and Knowledge Discovery_ , vol. 8, no. 4, p. e1249, 2018.
  * [218] L. K. Hansen and P. Salamon, “Neural network ensembles,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 12, no. 10, pp. 993–1001, 1990.
  * [219] Y. Cao, T. A. Geddes, J. Y. H. Yang, and P. Yang, “Ensemble deep learning in bioinformatics,” _Nature Machine Intelligence_ , pp. 1–9, 2020.
  * [220] L. Nannia, S. Ghidoni, and S. Brahnam, “Ensemble of convolutional neural networks for bioimage classification,” _Applied Computing and Informatics_ , 2020.
  * [221] L. Wei, S. Wan, J. Guo, and K. K. Wong, “A novel hierarchical selective ensemble classifier with bioinformatics application,” _Artificial intelligence in medicine_ , vol. 83, pp. 82–90, 2017.
  * [222] F. Lv, M. Han, and T. Qiu, “Remote sensing image classification based on ensemble extreme learning machine with stacked autoencoder,” _IEEE Access_ , vol. 5, pp. 9021–9031, 2017.
  * [223] X. Dai, X. Wu, B. Wang, and L. Zhang, “Semisupervised scene classification for remote sensing images: A method based on convolutional neural networks and ensemble learning,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 16, no. 6, pp. 869–873, 2019.
  * [224] E. Marushko and A. Doudkin, “Methods of using ensembles of heterogeneous models to identify remote sensing objects,” _Pattern Recognition and Image Analysis_ , vol. 30, no. 2, pp. 211–216, 2020.
  * [225] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel, “Model-ensemble trust-region policy optimization,” in _International Conference on Learning Representations_ , 2018.
  * [226] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine, “Epopt: Learning robust neural network policies using model ensembles,” in _International Conference on Learning Representations_ , 2017.
  * [227] S. Fort, H. Hu, and B. Lakshminarayanan, “Deep ensembles: A loss landscape perspective,” _arXiv preprint arXiv:1912.02757_ , 2019.
  * [228] A. Renda, M. Barsacchi, A. Bechini, and F. Marcelloni, “Comparing ensemble strategies for deep learning: An application to facial expression recognition,” _Expert Systems with Applications_ , vol. 136, pp. 1–11, 2019\.
  * [229] E. J. Herron, S. R. Young, and T. E. Potok, “Ensembles of networks produced from neural architecture search,” in _International Conference on High Performance Computing_. Springer, 2020, pp. 223–234.
  * [230] S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra, “Why m heads are better than one: Training a diverse ensemble of deep networks,” _arXiv preprint arXiv:1511.06314_ , 2015.
  * [231] I. E. Livieris, L. Iliadis, and P. Pintelas, “On ensemble techniques of weight-constrained neural networks,” _Evolving Systems_ , pp. 1–13, 2020\.
  * [232] L. Nanni, S. Brahnam, and G. Maguolo, “Data augmentation for building an ensemble of convolutional neural networks,” in _Innovation in Medicine and Healthcare Systems, and Multimedia_. Singapore: Springer Singapore, 2019, pp. 61–69.
  * [233] J. Guo and S. Gould, “Deep cnn ensemble with data augmentation for object detection,” _arXiv preprint arXiv:1506.07224_ , 2015.
  * [234] R. Rahaman and A. H. Thiery, “Uncertainty quantification and deep ensembles,” _stat_ , vol. 1050, p. 20, 2020.
  * [235] Y. Wen, G. Jerfel, R. Muller, M. W. Dusenberry, J. Snoek, B. Lakshminarayanan, and D. Tran, “Combining ensembles and data augmentation can harm your calibration,” in _International Conference on Learning Representations_ , 2021.
  * [236] W. Kim, B. Goyal, K. Chawla, J. Lee, and K. Kwon, “Attention-based ensemble for deep metric learning,” in _Proceedings of the European Conference on Computer Vision_ , 2018.
  * [237] J. Yang and F. Wang, “Auto-ensemble: An adaptive learning rate scheduling based deep learning model ensembling,” _IEEE Access_ , vol. 8, pp. 217 499–217 509, 2020.
  * [238] M. Leutbecher and T. N. Palmer, “Ensemble forecasting,” _Journal of computational physics_ , vol. 227, no. 7, pp. 3515–3539, 2008.
  * [239] W. S. Parker, “Ensemble modeling, uncertainty and robust predictions,” _Wiley Interdisciplinary Reviews: Climate Change_ , vol. 4, no. 3, pp. 213–223, 2013.
  * [240] W. H. Beluch, T. Genewein, A. Nürnberger, and J. M. Köhler, “The power of ensembles for active learning in image classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9368–9377.
  * [241] A. Vyas, N. Jammalamadaka, X. Zhu, D. Das, B. Kaul, and T. L. Willke, “Out-of-distribution detection using an ensemble of self supervised leave-out classifiers,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 550–564.
  * [242] J. Kocić, N. Jovičić, and V. Drndarević, “An end-to-end deep neural network for autonomous driving designed for embedded automotive platforms,” _Sensors_ , vol. 19, no. 9, p. 2064, 2019.
  * [243] G. Martínez-Muñoz, D. Hernández-Lobato, and A. Suárez, “An analysis of ensemble pruning techniques based on ordered aggregation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 31, no. 2, pp. 245–259, 2008.
  * [244] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2006, pp. 535–541.
  * [245] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _stat_ , vol. 1050, p. 9, 2015.
  * [246] E. Englesson and H. Azizpour, “Efficient evaluation-time uncertainty estimation by improved distillation,” in _Workshop on Uncertainty and Robustness in Deep Learning at International Conference on Machine Learning_ , 2019\.
  * [247] S. Reich, D. Mueller, and N. Andrews, “Ensemble distillation for structured prediction: Calibrated, accurate, fast—choose three,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 5583–5595.
  * [248] G. Wang, W. Li, S. Ourselin, and T. Vercauteren, “Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation,” in _International MICCAI Brainlesion Workshop_. Springer, 2018, pp. 61–72.
  * [249] G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren, “Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks,” _Neurocomputing_ , vol. 338, pp. 34–45, 2019.
  * [250] N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath, “Test-time augmentation for deep learning-based cell segmentation on microscopy images,” _Scientific reports_ , vol. 10, no. 1, pp. 1–7, 2020\.
  * [251] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [252] D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag, “When and why test-time augmentation works,” _arXiv preprint arXiv:2011.11156_ , 2020.
  * [253] I. Kim, Y. Kim, and S. Kim, “Learning loss for test-time augmentation,” in _Advances in Neural Information Processing Systems_ , 2020, pp. 4163–4174.
  * [254] Q. Yu and K. Aizawa, “Unsupervised out-of-distribution detection by maximum classifier discrepancy,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9518–9526.
  * [255] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, “Likelihood ratios for out-of-distribution detection,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 14 707–14 718.
  * [256] J. Yao, W. Pan, S. Ghosh, and F. Doshi-Velez, “Quality of uncertainty quantification for bayesian neural network inference,” _arXiv preprint arXiv:1906.09686_ , 2019.
  * [257] X. Huang, J. Yang, L. Li, H. Deng, B. Ni, and Y. Xu, “Evaluating and boosting uncertainty quantification in classification,” _arXiv preprint arXiv:1909.06030_ , 2019.
  * [258] J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 233–240.
  * [259] T. Pearce, A. Brintrup, M. Zaki, and A. Neely, “High-quality prediction intervals for deep learning: A distribution-free, ensembled approach,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4075–4084.
  * [260] D. Su, Y. Y. Ting, and J. Ansel, “Tight prediction intervals using expanded interval minimization,” _arXiv preprint arXiv:1806.11222_ , 2018.
  * [261] P. McClure, N. Rho, J. A. Lee, J. R. Kaczmarzyk, C. Y. Zheng, S. S. Ghosh, D. M. Nielson, A. G. Thomas, P. Bandettini, and F. Pereira, “Knowing what you know in brain segmentation using bayesian deep neural networks,” _Frontiers in neuroinformatics_ , vol. 13, p. 67, 2019.
  * [262] A. P. Soleimany, H. Suresh, J. J. G. Ortiz, D. Shanmugam, N. Gural, J. Guttag, and S. N. Bhatia, “Image segmentation of liver stage malaria infection with spatial uncertainty sampling,” _arXiv preprint arXiv:1912.00262_ , 2019.
  * [263] R. D. Soberanis-Mukul, N. Navab, and S. Albarqouni, “Uncertainty-based graph convolutional networks for organ segmentation refinement,” in _Medical Imaging with Deep Learning_. PMLR, 2020, pp. 755–769.
  * [264] P. Seebock, J. I. Orlando, T. Schlegl, S. M. Waldstein, H. Bogunovic, S. Klimscha, G. Langs, and U. Schmidt-Erfurth, “Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct,” _IEEE Transactions on Medical Imaging_ , vol. 39, p. 87–98, 2020.
  * [265] V. Kuleshov, N. Fenner, and S. Ermon, “Accurate uncertainties for deep learning using calibrated regression,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2796–2804.
  * [266] S. Seo, P. H. Seo, and B. Han, “Learning for single-shot confidence calibration in deep neural networks through stochastic inferences,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9030–9038.
  * [267] Z. Li and D. Hoiem, “Improving confidence estimates for unfamiliar examples,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2686–2695.
  * [268] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [269] G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,” _arXiv preprint arXiv:1701.06548_ , 2017.
  * [270] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” in _Advances in Neural Information Processing Systems_ , 2019, pp. 4694–4703.
  * [271] B. Venkatesh and J. J. Thiagarajan, “Heteroscedastic calibration of uncertainty estimators in deep learning,” _arXiv preprint arXiv:1910.14179_ , 2019.
  * [272] P. Izmailov, W. J. Maddox, P. Kirichenko, T. Garipov, D. Vetrov, and A. G. Wilson, “Subspace inference for bayesian deep learning,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 1169–1179.
  * [273] Z. Zhang, A. V. Dalca, and M. R. Sabuncu, “Confidence calibration for convolutional neural networks using structured dropout,” _arXiv preprint arXiv:1906.09551_ , 2019.
  * [274] M.-H. Laves, S. Ihler, K.-P. Kortmann, and T. Ortmaier, “Well-calibrated model uncertainty with temperature scaling for dropout variational inference,” _arXiv preprint arXiv:1909.13550_ , 2019.
  * [275] A. Mehrtash, W. M. Wells, C. M. Tempany, P. Abolmaesumi, and T. Kapur, “Confidence calibration and predictive uncertainty estimation for deep medical image segmentation,” _IEEE Transactions on Medical Imaging_ , 2020\.
  * [276] B. Zadrozny and C. Elkan, “Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers,” in _International Conference on Machine Learning_ , vol. 1. Citeseer, 2001, pp. 609–616.
  * [277] D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” in _International Conference on Learning Representations_ , 2019.
  * [278] S. Thulasidasan, G. Chennupati, J. A. Bilmes, T. Bhattacharya, and S. Michalak, “On mixup training: Improved calibration and predictive uncertainty for deep neural networks,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 888–13 899.
  * [279] J. Maroñas, D. Ramos, and R. Paredes, “Improving calibration in mixup-trained deep neural networks through confidence-based loss functions,” _arXiv preprint arXiv:2003.09946_ , 2020.
  * [280] K. Patel, W. Beluch, D. Zhang, M. Pfeiffer, and B. Yang, “On-manifold adversarial data augmentation improves uncertainty calibration,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 8029–8036.
  * [281] D. Comaniciu, V. Ramesh, and P. Meer, “Real-time tracking of non-rigid objects using mean shift,” in _Proceedings IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 2. IEEE, 2000, pp. 142–149.
  * [282] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.
  * [283] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 2015, 2015, p. 2901.
  * [284] M. Kull, M. Perelló-Nieto, M. Kängsepp, T. de Menezes e Silva Filho, H. Song, and P. A. Flach, “Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 12 295–12 305.
  * [285] D. Levi, L. Gispan, N. Giladi, and E. Fetaya, “Evaluating and calibrating uncertainty prediction in regression tasks,” _arXiv preprint arXiv:1905.11659_ , 2019.
  * [286] J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, and T. Schön, “Evaluating model calibration in classification,” in _The 22nd International Conference on Artificial Intelligence and Statistics_. PMLR, 2019, pp. 3459–3467.
  * [287] M. H. DeGroot and S. E. Fienberg, “The comparison and evaluation of forecasters,” _Journal of the Royal Statistical Society: Series D (The Statistician)_ , vol. 32, no. 1-2, pp. 12–22, 1983.
  * [288] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning,” in _CVPR Workshops_ , 2019, pp. 38–41.
  * [289] A. Ghandeharioun, B. Eoff, B. Jou, and R. Picard, “Characterizing sources of uncertainty to proxy calibration and disambiguate annotator and data bias,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop_. IEEE, 2019, pp. 4202–4206.
  * [290] F. J. Pulgar, A. J. Rivera, F. Charte, and M. J. del Jesus, “On the impact of imbalanced data in convolutional neural networks performance,” in _International Conference on Hybrid Artificial Intelligence Systems_. Springer, 2017, pp. 220–232.
  * [291] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7167–7177.
  * [292] M. L. Iuzzolino, T. Umada, N. R. Ahmed, and D. A. Szafir, “In automation we trust: Investigating the role of uncertainty in active learning systems,” _arXiv preprint arXiv:2004.00762_ , 2020.
  * [293] B. Settles, “Active learning literature survey,” University of Wisconsin-Madison Department of Computer Sciences, Tech. Rep., 2009.
  * [294] R. Pop and P. Fulop, “Deep ensemble bayesian active learning: Addressing the mode collapse issue in monte carlo dropout via ensembles,” _arXiv preprint arXiv:1811.03897_ , 2018.
  * [295] M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar, “Bayesian reinforcement learning: A survey,” _Foundations and Trends® in Machine Learning_ , vol. 8, no. 5-6, pp. 359–483, 2015.
  * [296] S. Hu, D. Worrall, S. Knegt, B. Veeling, H. Huisman, and M. Welling, “Supervised uncertainty quantification for segmentation with multiple annotations,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 137–145.
  * [297] F. C. Ghesu, B. Georgescu, E. Gibson, S. Guendel, M. K. Kalra, R. Singh, S. R. Digumarthy, S. Grbic, and D. Comaniciu, “Quantifying and leveraging classification uncertainty for chest radiograph assessment,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 676–684.
  * [298] M. S. Ayhan, L. Kuehlewein, G. Aliyeva, W. Inhoffen, F. Ziemssen, and P. Berens, “Expert-validated estimation of diagnostic uncertainty for deep neural networks in diabetic retinopathy detection,” _Medical Image Analysis_ , p. 101724, 2020.
  * [299] N. Sunderhauf, O. Brock, W. Scheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford _et al._ , “The limits and potentials of deep learning for robotics,” _The International Journal of Robotics Research_ , vol. 37, no. 4-5, pp. 405–420, 2018.
  * [300] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * [301] D. Fox, “Markov localization-a probabilistic framework for mobile robot localization and navigation.” Ph.D. dissertation, Citeseer, 1998.
  * [302] D. Fox, W. Burgard, H. Kruppa, and S. Thrun, “A probabilistic approach to collaborative multi-robot localization,” _Autonomous robots_ , vol. 8, no. 3, pp. 325–344, 2000.
  * [303] S. Thrun, D. Fox, W. Burgard, and F. Dellaert, “Robust monte carlo localization for mobile robots,” _Artificial intelligence_ , vol. 128, no. 1-2, pp. 99–141, 2001.
  * [304] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: part i,” _IEEE robotics & automation magazine_, vol. 13, no. 2, pp. 99–110, 2006.
  * [305] T. Bailey and H. Durrant-Whyte, “Simultaneous localization and mapping (slam): Part ii,” _IEEE robotics & automation magazine_, vol. 13, no. 3, pp. 108–117, 2006.
  * [306] M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit, “Fastslam: A factored solution to the simultaneous localization and mapping problem,” _Aaai/iaai_ , vol. 593598, 2002.
  * [307] M. Kaess, V. Ila, R. Roberts, and F. Dellaert, “The bayes tree: An algorithmic foundation for probabilistic robot mapping,” in _Algorithmic Foundations of Robotics IX_. Springer, 2010, pp. 157–173.
  * [308] F. Dellaert and M. Kaess, “Factor graphs for robot perception,” _Foundations and Trends in Robotics_ , vol. 6, no. 1-2, pp. 1–139, 2017.
  * [309] H. A. Loeliger, “An introduction to factor graphs,” _IEEE Signal Processing Magazine_ , vol. 21, no. 1, pp. 28–41, 2004.
  * [310] D. Silver and J. Veness, “Monte-carlo planning in large pomdps,” in _Advances in Neural Information Processing Systems_ , 2010.
  * [311] S. Ross, J. Pineau, S. Paquet, and B. Chaib-Draa, “Online planning algorithms for pomdps,” _Journal of Artificial Intelligence Research_ , vol. 32, pp. 663–704, 2008.
  * [312] S. M. Richards, F. Berkenkamp, and A. Krause, “The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems,” in _Conference on Robot Learning_. PMLR, 2018, pp. 466–476.
  * [313] F. Berkenkamp, A. P. Schoellig, and A. Krause, “Safe controller optimization for quadrotors with gaussian processes,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 491–496.
  * [314] F. Berkenkamp, M. Turchetta, A. P. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [315] H. Grimmett, R. Triebel, R. Paul, and I. Posner, “Introspective classification for robot perception,” _The International Journal of Robotics Research_ , vol. 35, no. 7, pp. 743–762, 2016.
  * [316] R. Bajcsy, “Active perception,” _Proceedings of the IEEE_ , vol. 76, no. 8, pp. 966–1005, 1988.
  * [317] R. Triebel, H. Grimmett, R. Paul, and I. Posner, “Driven learning for driving: How introspection improves semantic mapping,” in _Robotics Research_. Springer, 2016, pp. 449–465.
  * [318] A. Narr, R. Triebel, and D. Cremers, “Stream-based active learning for efficient and adaptive classification of 3d objects,” in _2016 IEEE International Conference on Robotics and Automation_. IEEE, 2016, pp. 227–233.
  * [319] D. A. Cohn, Z. Ghahramani, and M. I. Jordan, “Active learning with statistical models,” _Journal of artificial intelligence research_ , vol. 4, pp. 129–145, 1996.
  * [320] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 427–436.
  * [321] K. Wong, S. Wang, M. Ren, M. Liang, and R. Urtasun, “Identifying unknown instances for autonomous driving,” in _Conference on Robot Learning_. PMLR, 2020, pp. 384–393.
  * [322] W. Boerdijk, M. Sundermeyer, M. Durner, and R. Triebel, “”What’s this?”–learning to segment unknown objects from manipulation sequences,” in _Intern. Conf. on Robotics and Automation_ , 2021.
  * [323] C. Richter and N. Roy, “Safe visual navigation via deep learning and novelty detection,” _Robotics: Science and Systems Foundation_ , 2017.
  * [324] V. Peretroukhin, M. Giamou, D. M. Rosen, W. N. Greene, N. Roy, and J. Kelly, “A smooth representation of belief over so (3) for deep rotation learning with uncertainty,” _arXiv preprint arXiv:2006.01031_ , 2020.
  * [325] B. Lütjens, M. Everett, and J. P. How, “Safe reinforcement learning with model uncertainty estimates,” in _2019 International Conference on Robotics and Automation_. IEEE, 2019, pp. 8662–8668.
  * [326] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine, “Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5129–5136.
  * [327] F. Stulp, E. Theodorou, J. Buchli, and S. Schaal, “Learning to grasp under uncertainty,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, 2011, pp. 5703–5708.
  * [328] V. Tchuiev and V. Indelman, “Inference over distribution of posterior class probabilities for reliable bayesian classification and object-level perception,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 4329–4336, 2018.
  * [329] Y. Feldman and V. Indelman, “Bayesian viewpoint-dependent robust classification under model and localization uncertainty,” in _2018 IEEE International Conference on Robotics and Automation_. IEEE, 2018, pp. 3221–3228.
  * [330] N. Yang, L. von Stumberg, R. Wang, and D. Cremers, “D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1281–1292.
  * [331] S. Wang, R. Clark, H. Wen, and N. Trigoni, “Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2043–2050.
  * [332] C. Gurău, C. H. Tong, and I. Posner, “Fit for purpose? predicting perception performance based on past experience,” in _International Symposium on Experimental Robotics_. Springer, 2016, pp. 454–464.
  * [333] S. Daftry, S. Zeng, J. A. Bagnell, and M. Hebert, “Introspective perception: Learning to predict failures in vision systems,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 1743–1750.
  * [334] M. Netzband, W. L. Stefanov, and C. Redman, _Applied remote sensing for urban planning, governance and sustainability_. Springer Science & Business Media, 2007.
  * [335] C. Giardino, M. Bresciani, P. Villa, and A. Martinelli, “Application of remote sensing in water resource management: the case study of lake trasimeno, italy,” _Water resources management_ , vol. 24, no. 14, pp. 3885–3899, 2010\.
  * [336] C. J. Van Westen, “Remote sensing for natural disaster management,” _International archives of photogrammetry and remote sensing_ , vol. 33, no. B7/4; PART 7, pp. 1609–1617, 2000.
  * [337] X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 5, no. 4, pp. 8–36, 2017.
  * [338] J. Gawlikowski, M. Schmitt, A. Kruspe, and X. X. Zhu, “On the fusion strategies of sentinel-1 and sentinel-2 data for local climate zone classification,” in _IEEE International Geoscience and Remote Sensing Symposium_ , 2020, pp. 2081–2084.
  * [339] ESA, “European space agency (esa) developed earth observation satellites,” 2019\. [Online]. Available: http://www.esa.int:8080/ESA_Multimedia/Images/2019/05/ESA-developed_Earth_observation_missions
  * [340] Z. Nado, N. Band, M. Collier, J. Djolonga, M. W. Dusenberry, S. Farquhar, A. Filos, M. Havasi, R. Jenatton, G. Jerfel _et al._ , “Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning,” _arXiv preprint arXiv:2106.04015_ , 2021.
  * [341] M. Kull and P. A. Flach, “Reliability maps: a tool to enhance probability estimates and improve classification accuracy,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2014, pp. 18–33.
  * [342] J. Antorán, U. Bhatt, T. Adel, A. Weller, and J. M. Hernández-Lobato, “Getting a clue: A method for explaining uncertainty estimates,” in _International Conference on Learning Representations_ , 2021.
  * [343] M. Reichstein, G. Camps-Valls, B. Stevens, M. Jung, J. Denzler, and N. Carvalhais, “Deep learning and process understanding for data-driven earth system science,” _Nature_ , vol. 566, no. 7743, pp. 195–204, 2019\.
  * [344] J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar, “Integrating physics-based modeling with machine learning: A survey,” _arXiv preprint arXiv:2003.04919_ , 2020.
  * [345] E. De Bézenac, A. Pajot, and P. Gallinari, “Deep learning for physical processes: Incorporating prior scientific knowledge,” _Journal of Statistical Mechanics: Theory and Experiment_ , vol. 2019, no. 12, p. 124009, 2019\.