2021

[1]Franko Schmähling

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Physikal.-Techn. Bundesanstalt Braunschweig & Berlin, Germany

# A framework for benchmarking uncertainty in deep regression

<EMAIL_ADDRESS>Jörg Martin<EMAIL_ADDRESS>Clemens Elster
<EMAIL_ADDRESS>*

###### Abstract

We propose a framework for the assessment of uncertainty quantification in
deep regression. The framework is based on regression problems where the
regression function is a linear combination of nonlinear functions. Basically,
any level of complexity can be realized through the choice of the nonlinear
functions and the dimensionality of their domain. Results of an uncertainty
quantification for deep regression are compared against those obtained by a
statistical reference method. The reference method utilizes knowledge of the
underlying nonlinear functions and is based on a Bayesian linear regression
using a reference prior. Reliability of uncertainty quantification is assessed
in terms of coverage probabilities, and accuracy through the size of
calculated uncertainties. We illustrate the proposed framework by applying it
to current approaches for uncertainty quantification in deep regression. The
flexibility, together with the availability of a reference solution, makes the
framework suitable for defining benchmark sets for uncertainty quantification.

###### keywords:

Deep Learning, Bayesian Networks, Deep Regression, Reference Prior

##  1 Introduction

Methods from deep learning have made tremendous progress over recent years and
are meanwhile routinely applied in many different areas. Examples comprise the
diagnosis of cancer KOOI2017303 , autonomous driving huang2020autonomous  or
language processing Otter_NLP_DL ; Deng_DL_NLP . Physical and engineering
sciences also benefit increasingly from deep learning and current applications
range from optical form measurements HoffmannL2020  over image quality
assessment in mammography kretz2020mammography ; Kretz2020  or medical imaging
in general LUNDERVOLD2019102  to applications in weather forecasts and climate
modeling Schultz_DL_weather ; Watson-Parris_DL_weatherclimate .

Especially for safety relevant applications, such as medical diagnosis, it is
crucial to be able to quantify the uncertainty associated with a prediction.
Uncertainty quantification can help to understand the behavior of a trained
neural net and, in particular, foster confidence in its predictions. This is
especially true for deep regression, where a single point estimate of a sought
function without any information regarding its accuracy can be largely
meaningless.

Bayesian networks Bishop2006 ; Gal2016 ; Depeweg2018  lend themselves to an
uncertainty quantification in a natural way. All network parameters are
modeled as random variables, and their posterior distribution can be used to
calculate the epistemic part of the uncertainty. However, the shear number of
network parameters prohibits the application of established computation
techniques such as Markov chain Monte Carlo RobertCasella2005 , which have
been successfully applied in numerous Bayesian inferences on low-dimensional
models.

Approximate Bayesian techniques, and in particular Variational Bayesian
Inference Bishop2006 , have therefore been suggested for deep learning. At the
same time, these methods are closely related to classical network
regularization techniques Hinton2012 ; Srivastava2014 ; pmlr-v28-wang13a .

Approaches based on ensemble learning Lakshminarayanan2017  comprise another
branch of current uncertainty quantification methods. These techniques scale
well and are easy to implement. Furthermore, they have been shown to perform
very successfully compared to other methods for uncertainty quantification
Caldeira2020 ; Gustafsson2020 ; Ovadia2019 ; Scalia2020 .

In accordance with the growing efforts on the development of methods for
uncertainty quantification in deep learning, the amount of work on comparing
different uncertainty quantification approaches increases as well Yao2019 ;
Levi2019 ; Gustafsson2020 ; Kendall2017 ; Ovadia2019 ; Caldeira2020 ;
Scalia2020 . Different ways of assessing the performance of uncertainty
quantification methods have been proposed, ranging from explorative
comparisons of the performance on test data for simple regression problems
with known ground truth to various metrics Gustafsson2020 ; Levi2019 .
We propose a novel framework for the assessment of uncertainty quantification
in deep regression. The framework is based on regression problems of the kind

$y_{i}\sim\mathcal{N}\left(G(x_{i})^{T}\gamma,\sigma^{2}\right)\,,$ (1)

that are linear in the parameter $\gamma=(\gamma_{1},\gamma_{2},\ldots)^{T}$,
and where $G(x)=(G_{1}(x),G_{2}(x),\ldots)^{T}$ is a vector built of nonlinear
functions $G_{1},\,G_{2},\,\ldots~{}$. Deep regression intends to infer the
regression function $G(x)^{T}\gamma$, and it does not make use of the specific
structure of (1). The level of complexity is controlled through the choice of
the nonlinearity in the function $G(x)$ as well as by the dimensionality of
its input $x$. Results of an uncertainty quantification for deep regression of
problems of the kind in (1) can be compared to those obtained by a statistical
reference method. The reference method utilizes knowledge of $G$ and the
linearity with respect to the unknown parameters $\gamma$. It consists of a
Bayesian inference using the improper prior $\pi(\gamma)\propto 1$, which
coincides with the reference prior for the statistical model (1)
berger1992development . This prior is probability matching Kass&Wasserman1996
so that the proposed reference uncertainty quantification can also be
recommended from the perspective of classical statistics. The statistical
reference method provides an anchor solution against which different methods
of uncertainty quantification for deep regression can be compared.

Performance of uncertainty quantification is here assessed in terms of
reliability and accuracy. The former is quantified through coverage
probabilities of interval estimates for the regression function at individual
inputs $x$, and the latter by the size of the associated uncertainty. We
concentrate on the assessment of the epistemic uncertainty which is relevant
when inference of the regression function is the goal of the task. The
performance metrics have been chosen on grounds of their simplicity and
typical use in statistics. Nevertheless, other performance metrics, cf. for
instance Gustafsson2020 ; Levi2019 , could be used as well in conjunction with
the proposed framework.

The assessment of methods for uncertainty quantification in deep regression is
illustrated by applying it to several current approaches and for several
choices of the function $G(x)$.

The paper is structured as follows. In section 2 we present the proposed
structure of test problems along with their treatment by a Bayesian reference
inference. In section 3 we illustrate the usage of the framework from section
2 for various choices of the underlying nonlinearity and study the performance
of several deep learning methods for uncertainty quantification. Finally, some
conclusions are given in section 4.

##  2 Creating reference models for uncertainty quantification

###  2.1 A primer on (deep) regression and uncertainty

A typical, normal regression for training data
$\mathcal{D}=\\{(x_{1},y_{y}),\ldots,(x_{N},y_{N})\\}$ assumes a model of the
sort

$\displaystyle y_{i}|x_{i}\sim\mathcal{N}(f_{\theta}(x_{i}),\sigma^{2}),$ (2)

where $\mathcal{N}$ denotes a normal distribution, $f_{\theta}$ is some
function, parametrized by $\theta$, and $\sigma^{2}$ denotes the variance of
underlying noise. For simplicity we will, in this work, only consider one-
dimensional outputs $y_{i}$, and we will also assume knowledge about
$\sigma^{2}$. The input $x_{i}$ and the parameter $\theta$ will be of
arbitrary dimensions. Assuming independent observations, (2) gives rise to the
statistical model
$p(\mathcal{D}|\theta)=\prod_{i=1}^{N}\mathcal{N}(y_{i}|f_{\theta}(x_{i}),\sigma^{2})$
and inferring $\theta$ could be done, for instance, by maximizing
$p(\mathcal{D}|\theta)$ with respect to $\theta$ (maximum likelihood estimate
- MLE) or by a Bayesian approach, that is by computing the posterior

$\displaystyle\pi(\theta|\mathcal{D})\propto\pi(\theta)\cdot
p(\mathcal{D}|\theta)\,,$ (3)

where $\pi(\theta)$ denotes the prior for $\theta$. The uncertainty associated
with $\theta$ might, in the Bayesian case, be taken as the variation arising
from the distribution (3) while, in the frequentist case, often the variation
of the MLE under the data distribution is used.

In the case of deep regression, $f_{\theta}$ is given by a neural network and
the parameter $\theta$ contains all parameters of the network, such as weights
and biases, and is therefore typically of a pretty high dimension ($10^{6}$ or
more). In such cases there is usually no hope of sampling from
$\pi(\theta|\mathcal{D})$ or computing its normalization constant. Instead,
one has to rely on approximations. In deep learning, many methods Graves2011 ;
Kingma2015 ; Gal2017 ; Blundell2015 ; martin2021errorsinvariables  rely on the
statistical tool of variational inference, that is to approximate
$\pi(\theta|\mathcal{D})$ via a feasible distribution $q_{\phi}(\theta)$,
parametrized by a hyperparameter $\phi$, by minimizing the Kullback-Leibler
divergence with respect to $\phi$. Depending on the class of variational
distributions $q_{\phi}$ that is chosen, a variety of approximations arise
without a clear gold standard. Besides approaches based on variational
inference, there are other approaches to quantify the uncertainty about
$\theta$ Pearce2018 ; Lakshminarayanan2017 ; Barber98ensemblelearning ;
Bishop94mixturedensity . The approach of deep ensembles Lakshminarayanan2017 ,
introduced as a sort of frequentist approach, can also be viewed as a Bayesian
approach he2020bayesian ; Hoffmann2021 .

###  2.2 A generic model for benchmarking uncertainty

While for neural networks it is non-trivial to obtain an uncertainty for
$\theta$ in (2), for problems that are linear in a lower dimensional parameter
this is standard. In this section we will exploit this fact to propose a
generic scheme to create benchmarking test problems of arbitrary complexity.
Given a generally nonlinear function $G(x)$ with $p$-dimensional output and a
$p$-dimensional parameter $\gamma$, we can define the following model

$\displaystyle y_{i}\sim\mathcal{N}(G(x_{i})^{T}\gamma,\sigma^{2})\,,$ (4)

where we assume $\sigma^{2}$ to be known. We will use (4) in three different
ways:

  1. 1.

To generate training data $\mathcal{D}$: fix input data
$\mathbf{x}=(x_{1},\ldots,x_{N})$ and generate (independent)
$\mathbf{y}=(y_{1},\ldots,y_{N})$ according to (4). Store both, $\mathbf{x}$
and $\mathbf{y}$, in $\mathcal{D}=\\{(x_{i},y_{i})\,|\,i=1,\ldots,N\\}$.

  2. 2.

To generate test data with a known ground truth: we fix test inputs
$\textbf{x}^{\ast}=(x_{1}^{\ast},\ldots,x_{N^{\ast}}^{\ast})$. The values
$G(x_{i}^{\ast})\gamma$ will be taken as the (unnoisy) ground truth to judge
the quality of predictions and uncertainties of the studied models.

  3. 3.

To create an anchor model: the test input $\textbf{x}^{\ast}$ from 2. is
evaluated under a Bayesian inference, based on $\mathcal{D}$ from 1., and the
“true model” (4). This will be described in the following and is summarized by
the distribution (6) below. The anchor model serves as a reference point to
judge the performance of uncertainty methods for other (neural network based)
models of the more general shape (2).

The map $G(x)$ in (4) allows to tune the nonlinearity and complexity
underlying the training data $\mathcal{D}$. The linearity of (4) with respect
to $\gamma$, on the other hand, allows for an explicit Bayesian inference in
the anchor model (assuming knowledge about $G(x)$). We here propose to use the
(improper) reference prior $\pi(\gamma)\propto 1$ for the statistical model
(4), see e.g. berger1992development . Taking this prior has, in our case, the
convenient effect of exact probability matching, and hence the coverage
probability under repeated sampling of $\mathbf{y}$ of credible intervals
derived from the Bayesian posterior equals their credibility level. This
ensures that credible and confidence intervals coincide. The according
posterior distribution can be specified explicitly gelman2013bayesian  as

$\displaystyle\pi(\gamma|\mathcal{D})\propto\prod_{i=1}^{n}e^{-\frac{1}{2\sigma^{2}}(y_{i}-G(x_{i})^{T}\gamma)^{2}}\propto
e^{-\frac{1}{2\sigma^{2}}(\gamma-\hat{\gamma})^{T}V^{-1}(\gamma-\hat{\gamma})}\propto\mathcal{N}(\gamma|\hat{\gamma},\sigma^{2}V)\,,$
(5)

where
$\hat{\gamma}=VG(\mathbf{x})\mathbf{y},\,V=(G(\mathbf{x})G(\mathbf{x})^{T})^{-1}$
and where the evaluation $G(\mathbf{x})=$ $(G(x_{1})$ $,\ldots,G(x_{N}))$
$\in\mathbb{R}^{p\times N}$ should be read element-wise. For the test points
$\textbf{x}^{\ast}$, the distribution, conditional on $\mathcal{D}$, for the
prediction $G(\textbf{x}^{\ast})^{T}\gamma$ is then given by

$\displaystyle\pi(G(\textbf{x}^{\ast})^{T}\gamma|\mathcal{D})=\mathcal{N}(G(\textbf{x}^{\ast})^{T}\hat{\gamma},\sigma^{2}G(\textbf{x}^{\ast})^{T}VG(\textbf{x}^{\ast}))\,.$
(6)

We will refer to (6) as the anchor model in the remainder of this work and use
it for benchmarking the performance of other methods on $\mathcal{D}$. For the
evaluation we will treat, for the anchor model and all other methods,
$\sigma^{2}$ as known and use the same value that was taken to generate
$\mathcal{D}$. The distribution (6) models the epistemic uncertainty for the
test data points $\mathbf{x}^{\ast}$, which is, as we explained in the
introduction, the only uncertainty we study in this work. To consider the
aleatoric uncertainty, the posterior predictive distribution
$\pi(\mathbf{y}^{\ast}|\mathcal{D},\mathbf{x}^{\ast})=\mathcal{N}(\mathbf{y}^{\ast}|G(\mathbf{x}^{\ast})^{T}\hat{\gamma},\sigma^{2}G(\mathbf{x}^{\ast})^{T}VG(\mathbf{x}^{\ast})+\sigma^{2}\mathbf{I})$
could be used instead.

Due to the employed probability matching prior the anchor model expresses
both, the Bayesian and the frequentist view, and can thus be taken as
reference point for approaches from both worlds. From a frequentist point of
view, the anchor model (6) is unbiased and achieves the Cramer-Rao bound. This
makes the model “optimal” in the sense of the following lemma.

######  Lemma 1.

Consider an $x^{\ast}$ from the domain of $G$ and an estimator
$T(\mathbf{y}\mathchar 24635\relax\;x^{\ast})$ for $G(x^{\ast})^{T}\gamma$,
based on
$\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})\gamma,\sigma^{2}\mathbf{I})$.
Consider further some non-negative function $u(\mathbf{y}\mathchar
24635\relax\;x^{\ast})$. Then, at least one of the following two statements is
true

  1. 1.

The estimator $T(\cdot\,\mathchar 24635\relax\;x^{\ast})$ is biased.

  2. 2.

Whenever, for some $\mathbf{y}^{\prime}$, $u(\mathbf{y}^{\prime}\mathchar
24635\relax\;x^{\ast})$ is smaller than the uncertainty of the anchor model
(6), that is

$\displaystyle u(\mathbf{y}^{\prime}\mathchar
24635\relax\;x^{\ast})^{2}<\sigma^{2}G(x^{\ast})^{T}VG(x^{\ast})\,,$ (7)

then $u(\mathbf{y}^{\prime}\mathchar 24635\relax\;x^{\ast})$ is also smaller
than the root mean squared error of $T(\cdot\,\mathchar
24635\relax\;x^{\ast})$ under repeated sampling, i.e.
$u(\mathbf{y}^{\prime}\mathchar
24635\relax\;x^{\ast})^{2}<\mathbb{E}_{\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})^{T}\gamma,\sigma^{2}\mathbf{I})}\left[(T(\mathbf{y}\mathchar
24635\relax\;x^{\ast})-G(x^{\ast})^{T}\gamma)^{2}\right]$.

Proof: Assume that 1 is not true, so that $T(\cdot\,\mathchar
24635\relax\;x^{\ast})$ is unbiased. Note first, that, in this case, for any
$\mathbf{y}^{\prime}$ that satisfies (7) we do have

$\displaystyle u(\mathbf{y}^{\prime}\mathchar
24635\relax\;x^{\ast})^{2}<\mathrm{Var}_{\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})^{T}\gamma,\sigma^{2}\mathbf{I})}(T(\mathbf{y}\mathchar
24635\relax\;x^{\ast}))\,.$ (8)

Indeed, otherwise we could apply the Cramer-Rao inequality (in the form of
(Shao2003, , Theorem 3.3)) to the unbiased estimator $T(\cdot\,\mathchar
24635\relax\;x^{\ast})$ and would get a contradiction to (7):

$\displaystyle u(\mathbf{y}^{\prime}\mathchar
24635\relax\;x^{\ast})^{2}\geq\mathrm{Var}_{\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})^{T}\gamma,\sigma^{2}\mathbf{I})}(T(\mathbf{y}\mathchar
24635\relax\;x^{\ast}))\geq\sigma^{2}G(x^{\ast})^{T}VG(x^{\ast})\,.$

From (8) we arrive at 2, since the mean squared error is bounded from below by
the variance under repeated sampling

$\displaystyle u(\mathbf{y}^{\prime}\mathchar 24635\relax\;x^{\ast})^{2}$
$\displaystyle<\mathrm{Var}_{\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})^{T}\gamma,\sigma^{2}\mathbf{I})}(T(\mathbf{y}\mathchar
24635\relax\;x^{\ast}))$
$\displaystyle\leq\mathbb{E}_{\mathbf{y}\sim\mathcal{N}(G(\mathbf{x})^{T}\gamma,\sigma^{2}\mathbf{I})}\left[(T(\mathbf{y}\mathchar
24635\relax\;x^{\ast})-G(x^{\ast})^{T}\gamma)^{2}\right]\,.$

For our purposes, the statement of lemma 1 can be summarized as follows:
suppose for some test data point $x^{\ast}$ and training data $\mathbf{y}$ we
have a prediction $T(\mathbf{y}\mathchar 24635\relax\;x^{\ast})$ of a neural
network, for instance the average of the outputs under an approximation to the
posterior (3), and an uncertainty $u(\mathbf{y}\mathchar
24635\relax\;x^{\ast})$, for instance the corresponding standard deviation. If
the uncertainty is smaller than the one from the anchor model, lemma 1 tells
us that the prediction is biased or/and the uncertainty is “too small”, at
least compared to the root mean squared error.

##  3 Experiments

In this section we study various choices for the benchmarking framework from
section 2. Namely, we analyze the performance of a couple of state of the art
methods for uncertainty quantification in deep learning under test scenarios
created with (4) and compare them with each other and with the anchor model
from (6). Methods under test are variational Bayes methods such as Bernoulli-
Dropout Gal2016  (abbreviated ”BD” in the following), Concrete Dropout (CD)
Gal2017 , Variational Dropout (VD) Kingma2015  as well as methods based on
Deep Ensembles Lakshminarayanan2017 . For the latter we used three modes of
training: “standard” (Ens), with adversarial examples included in training
(EnsAdvA) and trained via bootstrapping the training data (EnsBS). In both,
Concrete and Variational Dropout, the dropout rate is inherently optimized
during the training process. As described in section 2 the anchor model (6)
will be included as a reference model. As the anchor model is based on
Bayesian linear regression we will use the abbreviation BLR.

Based on the framework from section 2 we can test these methods on various
numerical experiments, arising from different choices of the nonlinear
functions $G(x)$ in (4). In this work we will consider the following three
experiments.

  1. E.1

As a first choice we will consider a linear combination of $p=4$ sinusoidal
functions, with different phases and varying frequencies:

$G(x_{i})=(\sin(2\pi f_{1}\cdot x_{i}+\rho_{1}),\ldots,\sin(2\pi f_{4}\cdot
x_{i}+\rho_{4}))^{T}\,,$ (9)

where the $x_{i}$ are univariate and where the $\gamma\in\mathbb{R}^{4}$ in
(9) are randomly chosen from $[0,1]$ (uniform with support in $[0,1]$). All
four frequencies $f_{1},\ldots,f_{4}$ are equally spaced between
$[0.9,1.1]\cdot f_{\mathrm{main}}$ and phases $\rho$ are equidistantly chosen
between $[0,2\pi]$, where we will consider various choices for
$f_{\mathrm{main}}=1,\,2,\,3,\,\ldots$. The used $\sigma$ for this experiment
is $0.75$.

  2. E.2

As a second example, we will use the Styblinsky-Tang-function to scale the
dimensionality in an arbitrary manner, i.e. for $d=1,2\ldots,$ we define

$G(x_{i})=(x_{i,1},\,x_{i,1}^{2},\,x_{i,1}^{4},\,x_{i,2},\,x_{i,2}^{2},\,x_{i,2}^{4},\,\ldots,\,x_{i,d},\,x_{i,d}^{2},\,x_{i,d}^{4})^{T},$
(10)

with $x_{i,j}$ denoting the $j$-th component of $x_{i}\in\mathbb{R}^{d}$ and
with

$\gamma=(2.5,\,-8,\,0.5,\,2.5,\,-8,\,0.5,\,\ldots,\,2.5,\,-8,\,0.5)^{T}\in\mathbb{R}^{3d}.$

The used $\sigma$ for this experiment is $3$.

  3. E.3

As a third example, we will choose a simple polynomial mapping, so that
$G^{T}\gamma$ represents a linear combination of linear, quadratic and mixed
terms

$G(x_{i})=(1,\,x_{i,1},\,x_{i,2},\,x_{i,1}\cdot
x_{i,2},\,x_{i,1}^{2},\,x_{i,2}^{2})^{T}$ (11)

with randomly chosen coefficients $\gamma\in\mathbb{R}^{7}$ (uniform with
support in $[0,1]$) in (11) and $x_{i}\in\mathbb{R}^{2}$. The used $\sigma$
for this experiment is $0.5$.

As stated in the introduction of this work we evaluate the performance of the
different approaches for uncertainty quantification using common statistical
metrics. Namely, we will base the evaluation of the experiments on the
following three objectives.

  1. 1.

Prediction for an input: for the anchor model (BLR) and Bayesian deep learning
methods (BD, VD, CD) this is the mean of the output of the regression function
or the neural network, given the input, when the according parameter is drawn
from the posterior distribution (or its approximation). For ensemble based
methods (Ens, EnsAdvA, EnsBS) this is the average of the outputs of the single
ensemble members for the given input.

We will also compute the deviation as the absolute value of the difference
between the prediction and the unnoisy ground truth, as defined in section
2.2.

  2. 2.

Uncertainty for an input: for the anchor model (BLR) and Bayesian deep
learning methods (BD, VD, CD) this is the standard deviation of the output of
the regression function or the neural network, given the input, when the
according parameter is drawn from the posterior distribution (or its
approximation). For ensemble based methods (Ens, EnsAdvA, EnsBS) this is the
standard deviation of the outputs of the single ensemble members for the given
input.

  3. 3.

Coverage for an input: draw repeatedly training data under the corresponding
model (4), for neural network based methods retrain the involved networks, and
check whether the deviation, as in point 1, is smaller than 1.96 times the
computed uncertainty, as in point 2 (where the 1.96 is motivated from a normal
approximation). The coverage denotes the percentage of cases where this
succeeded. For the anchor model we expect a coverage of $95\%$.

Details on the implementation: For all experiments below, we used the same
network architecture of three hidden, fully connected layers with $128$, $64$,
$32$ neurons, a single output neuron and leaky ReLU activations (with negative
slope 0.01) between the layers. For the methods based on dropout, dropout
layers were added to the next-to-last layer. All methods and networks,
together with the benchmarking framework, are implemented by using MATLAB® and
the MATLAB® “Deep Learning Toolbox” MATLAB:2020b . All networks were
initialized via the Xavier initialisation. For ensemble based methods, we used
$120$ ensemble members. In experiments with the sinusoidal generic function
E.1 we have –train
set˝=50$elementsinthetrainingset(randomdistributedintrainingrange)and$–test
set˝=10^3$elementsinthetestset(chosenequidistantlyoverthetestrange).Fortheexperiment\ref{E2}$–train
set˝$scaledwiththeinputdimensionality$d$as$–train set˝=100⋅9^d-1,
d=1,2,3,…$.Forthesimple2Dpolynomialmapping(\ref{E3})wehave$450$inputsinthetrainset.Therangesforthetestsetwere$[-6,6]$fortheexperiment\ref{E1},and$[-5,5]$ineverydimensionoftheinputdomainforexperiment\ref{E2}and\ref{E3}.Therange(s)forthetrainsetare$[-4,4]$(ineveryinputdomain)forallabovelistedexperiments.Forallmethods,wescaledthenumberofepochsusedfortrainingas$300
⋅c$,where$c$waschosenequalto$f_main$for\ref{E1}andequaltothedimensionoftheinputdomainfor\ref{E2}and\ref{E3}.Theminibatchsizewaschosenequalto$–train
set˝$,exceptforensemblelearningwithbootstrap(EnsBS),wherewetook$–train
set˝/2$torunthebootstrappinginapullwithbackscenarioduringoneepoch.WeusedanAdamoptimizer\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{Adam_Kingma2015}{\@@citephrase{(}}{\@@citephrase{)}}}withlearningrate$10^-2$,an$L^2$regularizationof$1/–train
set˝$andmomentumparameters$β_1 =0.9, β_2=0.999$forallexperiments.\par$

###  3.1 Results

As a first application of the benchmarking framework, we perform the
experiment E.1 with different frequencies $f_{\mathrm{main}}$. For every
choice of $f_{\mathrm{main}}$ we define a train and a test set as described in
section 2. As stated above, the input range of the test set is broader, so
that we are able to observe the behavior of the methods under test for out-of-
distribution data. For the evaluation, we will distinguish between the out-of-
distribution range and the range used in the train set (in-distribution). Two
examples are illustrated in fig. 1 and 2. Both figs. show the predictions
(solid line) and associated uncertainties times 1.96 (shaded) provided by the
anchor model (BLR), Bernoulli Dropout (BD), Variational Dropout (VD) and
ensemble with bootstrap (EnsBS), for a “complexity level” of
$f_{\mathrm{main}}=2$ (see fig. 1) and $f_{\mathrm{main}}=5$ (see fig. 2). The
ground truth is illustrated as a dashed red line. The ground truth is
completely covered by the predictions and associated uncertainties (times
1.96) of the BLR (6). In the in-distribution range nearly all methods cover
the ground truth by their associated uncertainties. In the out-of-distribution
range the uncertainty grows for the deep learning methods, but is often to
small to cover the ground truth.

(a) BLR

(b) BD

(c) VD

(d) EnsBS

Figure 1: Experiment E.1 with $f_{\mathrm{main}}=1$. Predictions (solid lines)
and calculated uncertainties times 1.96 (shaded areas) of the anchor model BLR
(subplot (a)) and different methods in deep learning (subplots (b)-(d)),
together with the used train set (black dots) and ground truth (red dashed
line). The abbreviations for the methods are as in the beginning of section
3.1.

(a) BLR

(b) BD

(c) VD

(d) EnsBS

Figure 2: Experiment E.1 with $f_{\mathrm{main}}=5$. Predictions (solid lines)
and calculated uncertainties times 1.96 (shaded areas) of the anchor model BLR
(subplot (a)) and different methods in deep learning (subplots (b)-(d)),
together with the used train set (black dots) and ground truth (red dashed
line). The abbreviations for the methods are as in the beginning of section
3.1. Fig. 3 summarizes the deviation between the ground truth and the
prediction, the uncertainty and the coverage, as specified above, for two in-
distribution inputs ($x=-2.38$ and $x=1.2$) and one out-of-distribution input
($x=-5.11$) over varying complexity, i.e increasing frequency
$f_{\mathrm{main}}$. We repeated the experiment, for every
$f_{\mathrm{main}}$, $k=50$ times with the same test set but with different
realization of the added noise of the train set and re-trained the neural
networks in each repetition. The two left columns show the average of the
deviation from the ground truth and the uncertainty together with their
associated standard error for the different choices of input $x$. For the
coverage (third column from the right) the size of the error bars were
generated by using the fact that a sum of $k$ Bernoulli distributed numbers
(ground truth in or outside the tube spanned by the $1.96$ times the standard
deviation around the estimate) are Binomial distributed with variance
$p(1-p)/k$, where p is the success probability, for which the calculated
coverage over $k$ repetitions was taken.

(a) $x=-2.38$

(b) $x=-2.38$

(c) $x=-2.38$

(d) $x=1.2$

(e) $x=1.2$

(f) $x=1.2$

(g) $x=-5.11$

(h) $x=-5.11$

(i) $x=-5.11$

Figure 3: Experiment E.1 for various choices of the frequency
$f_{\mathrm{main}}$. The three columns show for the anchor model and different
methods from deep learning the deviation from the ground truth (left column),
the uncertainty (middle column) and the coverage of the ground truth by the
uncertainty times 1.96 (right column) for three different inputs $x=-2.38$
(first row), $x=1.2$ (second row) and $x=-5.11$ (third row, out-of-
distribution) and different choices of the frequency $f_{\mathrm{main}}$ (on
the abscissa). The plots were generated with 50 runs. For the deviation and
uncertainty the mean together with its standard errors are plotted. The BLR-
solution (red in fig. 3) provides a complexity independent coverage
probability of $0.95$ , i.e. $95\%$ of the ground truth is covered by the
estimate $\pm 1.96$ times the standard deviation, as expected from the theory.
At the same time, the deviation of the prediction of the BLR from the ground
truth is quite low. The uncertainty is also comparably low (compare fig. 3),
but large enough to ensure $95\%$ coverage. As motivated by lemma 1 we can, in
a certain sense, consider a deep learning method as “optimal” if it reaches
the BLR-solution in all three characteristics: deviation, uncertainty and
coverage. Every method under test in the benchmarking framework should thus
strive for the goal to reach the corner of a cube, defined by these three
quantities, where the BLR-solution is located. Howevever, since the number of
parameters in a deep regression drastically exceeds that of the BLR model, we
cannot expect a deep regression approach to actually match the BLR-solution.
In fig. 3, row 1 and 2, we observe an increasing deviation by increasing
complexity for all methods, except for BLR, for inputs of the in-distribution
(row 1 $\rightarrow$ $x=-2.38$, row 2 $\rightarrow$ $x=1.20$). At the same
time we can observe, in the subplots of the second column, that the size of
the associated uncertainty over increasing frequency is only slightly
increasing or constant for the approaches based on dropout (BD, VD and CD). As
predictable from subplot (a) and (b) the coverage, depicted in (c), is poor
for the Dropout-based methods and substantially lower as the $0.95$ provided
by the anchor model. The non-standard ensemble methods (EnsAdvA and EnsBS), on
the other hand, deliver a coverage near $0.95$ but overestimate for larger
complexities their uncertainty, which be can be seen by the coverage of almost
1.0. Note, that for the considered scenario the uncertainty of the anchor
model is below the uncertainties of all other models. Row 3 of fig. 3 shows
the results for the case of one out-of-distribution input test data
($x=-5.11$) in the evaluation of the method. We encounter a slightly different
behavior. The deviation from the ground truth is not increasing with
increasing complexity for all methods, except for VD. The size of the
associated uncertainty is not increasing and too low to deliver a sufficient
coverage, except for the ensemble methods. The reason for this behavior seems
to lie in the poor or not sufficient quality of the prediction in the out-of-
distribution range together with an underestimated uncertainty.
In regard to the figs. above, all ensemble methods absorb the increasing
deviation over increasing complexity with an increasing uncertainty,
especially EnsAdvA and EnsBS. This results in a good, but partially
overestimated, coverage near the value provided by the anchor method (BLR).
All methods based on the dropout technique (BD, CD and VD) underestimate the
uncertainty with simultaneously increasing deviations over increasing
complexity, whereby a good coverage (ideal $0.95$) is not possible.

(a) BLR

(b) BD

(c) VD

(d) EnsBS

Figure 4:  Experiment E.2 in dimension $d=2$. Predictions (solid lines) and
according uncertainties times 1.96 (shaded areas) of the anchor model BLR
(subplot (a) - uncertainties are very small, barely visible) and different
methods in deep learning (subplots (b)-(d)) along a diagonal of the
$d$-dimensional hypercube (cf. section 3), together with the ground truth (red
dashed line). The abbreviations for the methods in the beginning of section
3.1.

(a) BLR

(b) BD

(c) VD

(d) EnsBS

Figure 5:  Experiment E.2 in dimension $d=4$. Predictions (solid lines) and
according expanded uncertainties (shaded areas) of the anchor model BLR
(subplot (a) - uncertainties are very small, barely visible) and different
methods in deep learning (subplots (b)-(d)) along a diagonal of the
$d$-dimensional hypercube (cf. section 3), together with the ground truth (red
dashed line). The abbreviations for the methods are as in the beginning of
section 3.1.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: Experiment E.2 for various choices of the dimensionality $d$. The
three columns show for the anchor model and different methods from deep
learning the deviation from the ground truth (left column), the uncertainty
(middle column) and the coverage of the ground truth by the uncertainty times
1.96 (right column) for two different inputs $\lambda=0.5996$ (first row, ind-
distribution, cf. fig. 4) and $\lambda=0.074$ (second row, out-of-
distribution, cf. fig. 5)  Fig. 4 and 5 show the according results for the
experiment E.2. The elements of the test set were chosen to contain only
inputs that are located at the diagonal of the $d$-dimensional hypercube from
$(-5,\ldots,-5)^{T}\in\mathbb{R}^{d}$ to $(5,\ldots,5)^{T}\in\mathbb{R}^{d}$,
i.e. $x_{\lambda}=(1-\lambda)(-5,-5,\ldots,-5)^{T}+\lambda(5,5,\ldots,5)^{T}$
with $0\leq\lambda\leq 1$. In that way, the diagonal contains both, out-of-
distribution and in-distribution inputs. Fig. 6 depicts the deviation from the
ground truth, uncertainty and coverage for two different choices of $\lambda$,
namely $\lambda=0.6$ for in-distribution and $\lambda=0.07$ for out-of-
distribution. For both inputs $x_{\lambda}$ we observe an overestimation of
the uncertainty, compared with the anchor model, for the ensemble methods with
adversarial attacks (EnsAdvA) and bootstrapping (EnsBS). Both ensemble methods
cover for larger dimensions the increasing deviation from the ground truth
with an increasing uncertainty, but the increase of uncertainty in dependence
in the increasing dimensionality seems a bit too large, especially for
EnsAdvA. For CD and VD we observe a decreasing coverage over increasing
dimensionality. At the same time the deviation from the ground truth is
strongly increasing for CD. The increasing uncertainty over increasing
dimensionality is too low, so that the uncertainty does not compensate for the
increasing error for this method. For BD we observe an increasing uncertainty
with increasing dimensionality and a moderately increasing deviation, which
leads to a coverage of round about $0.8$ over increasing dimensionality. For
one representative input of the out-of-distribution range (see subplots fig. 4
and 5) we observe for all DL-methods a poor coverage, i.e. we observe a poor
behavior in the out-of-distribution range and at the same time an
underestimated uncertainty.
For experiment E.3, fig. 7 shows the deviation from the ground truth, the
uncertainty and the coverage as a heat map for every input of the test set
(regular grid in $x_{1}$ and $x_{2}$). The in-distribution region is located
at $1\leq x_{1}\leq 4$ and $1\leq x_{2}\leq 4$. Every row represents a method,
the first row contains results provided by BLR, row 2 and 3 contain results
from BD and EnsBS. The deviation and uncertainty is shown as a decadic
logarithm. The BLR method shows on every position of the input grid a coverage
around $0.95$. BD shows in some regions of the input space a poor coverage,
i.e. the associated uncertainty of the BD does not cover the deviation in this
regions. The ensemble method using bootstrapping (EnsBS) only exhibits in the
out-of-distribution range a coverage well below 0.95. deviation uncertainty
coverage BLR BD VD EnsBS Figure 7: Results for the experiment E.3 of the
anchor model and different methods from deep learning for regular gridded test
data. Every plot of the 9 visible plots arranged in a tabular, contains one
heat map and shows the objectives introduced at the beginning of section 3
averaged over 30 repeated samplings: the deviation from the ground truth
(first column, in $\log_{10}$ scale), uncertainty (second column, in
$\log_{10}$ scale) and the coverage (third column). The different rows
correspond to different methods with abbreviations are as in the beginning of
section 3.1.

##  4 Conclusion and discussion

In this article we described a method to create benchmarking test problems for
uncertainty quantification in deep learning. We demonstrated how the linearity
of a generating model in its parameters can be exploited to create an anchor
model that helps to judge the quality of the prediction and the quantified
uncertainty of deep learning approaches. Within this set-up it is possible to
design generic data sets with explicit control over problem complexity and
input dimensionality. For various test problems we studied the performance of
several approaches for uncertainty quantification in deep regression. Common
statistical metrics were used to compare the performance of these methods with
each other and with the anchor model. In particular we assessed the behavior
of these methods under increasing problem complexity and input dimensionality.
The flexibility to design test problems with specified complexity, along with
the availability of a reference solution, qualifies the proposed framework to
form the basis for designing set of benchmark problems for uncertainty
quantification in deep regression.

## References

  *   * (1) Kooi, T., Litjens, G., van Ginneken, B., Gubern-Mérida, A., Sánchez, C.I., Mann, R., den Heeten, A., Karssemeijer, N.: Large scale deep learning for computer aided detection of mammographic lesions. Medical Image Analysis 35, 303–312 (2017). https://doi.org/10.1016/j.media.2016.07.007
  * (2) Huang, Y., Chen, Y.: Autonomous driving with deep learning: a survey of state-of-art technologies (2020)
  * (3) Otter, D.W., Medina, J.R., Kalita, J.K.: A survey of the usages of deep learning for natural language processing. IEEE Transactions on Neural Networks and Learning Systems 32(2), 604–624 (2021). https://doi.org/10.1109/TNNLS.2020.2979670
  * (4) Deng, L., Liu, Y.: Deep Learning in Natural Language Processing, 1st edn. Springer, ??? (2018)
  * (5) Hoffmann, L., Elster, C.: Deep neural networks for computational optical form measurements. Journal of Sensors and Sensor Systems 9(2), 301–307 (2020). https://doi.org/10.5194/jsss-9-301-2020
  * (6) Kretz, T., Müller, K.-R., Schaeffter, T., Elster, C.: Mammography image quality assurance using deep learning. IEEE Transactions on Biomedical Engineering 67(12), 3317–3326 (2020)
  * (7) Kretz, T.: Development of model observers for quantitative assessment of mammography image quality. Doctoral thesis, Technische Universität Berlin, Berlin (2020). https://doi.org/10.14279/depositonce-10552. http://dx.doi.org/10.14279/depositonce-10552
  * (8) Lundervold, A.S., Lundervold, A.: An overview of deep learning in medical imaging focusing on mri. Zeitschrift für Medizinische Physik 29(2), 102–127 (2019). https://doi.org/10.1016/j.zemedi.2018.11.002. Special Issue: Deep Learning in Medical Physics
  * (9) Schultz, M., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L.H., Mozaffari, A., Stadtler, S.: Can deep learning beat numerical weather prediction? Philosophical transactions of the Royal Society of London / A 379(2194), 20200097 (2021). https://doi.org/10.1098/rsta.2020.0097
  * (10) Watson-Parris, D.: Machine learning for weather and climate are worlds apart. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences 379(2194), 20200098 (2021). https://doi.org/10.1098/rsta.2020.0098
  * (11) Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Berlin, Heidelberg (2006)
  * (12) Gal, Y.: Uncertainty in deep learning. PhD Thesis, 174 (2016)
  * (13) Depeweg, S., Hernandez-Lobato, J.M., Doshi-Velez, F., Udluft, S.: Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. 35th International Conference on Machine Learning, ICML 2018 3, 1920–1934 (2018) arXiv:1710.07283
  * (14) Robert, C.P., Casella, G.: Monte Carlo Statistical Methods (Springer Texts in Statistics). Springer, Berlin, Heidelberg (2005)
  * (15) Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors. CoRR abs/1207.0580 (2012) arXiv:1207.0580
  * (16) Srivastava, N.: Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(6), 1929–1958 (2014)
  * (17) Wang, S., Manning, C.: Fast dropout training. Proceedings of the 30th International Conference on Machine Learning 28(2), 118–126 (2013)
  * (18) Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 2017-Decem(Nips), 6403–6414 (2017) arXiv:arXiv:1612.01474v3
  * (19) Caldeira, J., Nord, B.: Deeply uncertain: Comparing methods of uncertainty quantification in deep learning algorithms. arXiv (2020) arXiv:2004.10710. https://doi.org/10.1088/2632-2153/aba6f3
  * (20) Gustafsson, F.K., Danelljan, M., Schön, T.B.: Evaluating scalable Bayesian deep learning methods for robust computer vision (2020)
  * (21) Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.V., Lakshminarayanan, B., Snoek, J.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In: NeurIPS (2019)
  * (22) Scalia, G., Grambow, C.A., Pernici, B., Li, Y.-P., Green, W.H.: Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. Journal of Chemical Information and Modeling 60(6), 2697–2717 (2020) https://doi.org/10.1021/acs.jcim.9b00975. https://doi.org/10.1021/acs.jcim.9b00975. PMID: 32243154
  * (23) Yao, J., Pan, W., Ghosh, S., Doshi-Velez, F.: Quality of uncertainty quantification for Bayesian neural network inference. arXiv (2019) arXiv:1906.09686
  * (24) Levi, D., Gispan, L., Giladi, N., Fetaya, E.: Evaluating and calibrating uncertainty prediction in regression tasks. arXiv (2019) arXiv:1905.11659
  * (25) Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems 2017-Decem(Nips), 5575–5585 (2017) arXiv:1703.04977
  * (26) Berger, J.O., Bernardo, J.M.: On the development of the reference prior method. Bayesian statistics 4(4), 35–60 (1992)
  * (27) Kass, R.E., Wasserman, L.: The selection of prior distributions by formal rules. Journal of the American Statistical Association 91(435), 1343–1370 (1996) https://www.tandfonline.com/doi/pdf/10.1080/01621459.1996.10477003. https://doi.org/10.1080/01621459.1996.10477003
  * (28) Graves, A.: Practical variational inference for neural networks. Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 2011, 1–9 (2011)
  * (29) Kingma, D.P., Salimans, T., Welling, M.: Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems 2015-Janua(Mcmc), 2575–2583 (2015) arXiv:arXiv:1506.02557v2
  * (30) Gal, Y., Hron, J., Kendall, A.: Concrete dropout. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 3584–3593. Curran Associates Inc., Red Hook, NY, USA (2017)
  * (31) Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural networks. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. ICML’15, pp. 1613–1622. JMLR.org, ??? (2015)
  * (32) Martin, J., Elster, C.: Errors-in-variables for deep learning: rethinking aleatoric uncertainty. arXiv preprint arXiv:2105.09095 (2021)
  * (33) Pearce, T., Zaki, M., Brintrup, A., Anastassacos, N., Neely, A.: Uncertainty in neural networks: Bayesian ensembling. arXiv 108 (2018)
  * (34) Barber, D., Bishop, C.M.: Ensemble learning in Bayesian neural networks. In: Neural Networks and Machine Learning, pp. 215–237. Springer, ??? (1998)
  * (35) Bishop, C.: Mixture density networks. Workingpaper, Aston University (1994)
  * (36) He, B., Lakshminarayanan, B., Teh, Y.W.: Bayesian deep ensembles via the neural tangent kernel. arXiv preprint arXiv:2007.05864 (2020)
  * (37) Hoffmann, L., Elster, C.: Deep ensembles from a Bayesian perspective. arXiv preprint arXiv:2105.13283 (2021)
  * (38) Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian Data Analysis. CRC press, ??? (2013)
  * (39) Shao, J.: Mathematical Statistics. Springer, New York, NY (2003)
  * (40) The Mathworks, Inc.: MATLAB Version 9.9.0.1467703 (R2020b). Natick, Massachusetts (2020). The Mathworks, Inc.
  * (41) Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980