# DEUP: Direct Epistemic Uncertainty Estimation

Salem Lahlou
Mila, Université de Montréal
&Moksh Jain††footnotemark:
Mila, Université de Montréal
&Hadi Nekoei
Mila, Université de Montréal
&Victor I Butoi
Cornell University
&Paul Bertin
Mila, Université de Montréal
&Jarrid Rector-Brooks
Mila, Université de Montréal
&Maksym Korablyov
Mila, Université de Montréal
&Yoshua Bengio
Mila, Université de Montréal, CIFAR Fellow
Equal Contribution, Correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Epistemic Uncertainty (EU) is the part of out-of-sample prediction error due
to the lack of knowledge of the learner. While existing work focuses on model
variance as a proxy of EU, we propose a principled framework for directly
estimating it by learning to predict the generalization error and subtracting
an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability, using
a secondary learner. We discuss the merits of this novel interpretation of EU,
and highlight how it differs from variance-based proxies of EU and addresses
their shortcomings. Our framework, Direct Epistemic Uncertainty Prediction
(DEUP) is particularly interesting in interactive learning environments, where
the learner is allowed to acquire novel examples in each round. Through a wide
set of experiments, we illustrate how existing methods in sequential model
optimization can be improved with EU estimates from DEUP, and how DEUP can be
used to drive exploration in reinforcement learning. We also evaluate the
quality of uncertainty estimates from DEUP for probabilistic image
classification and predicting synergies of drug combinations.

A remaining great challenge for machine learning research is purposeful
knowledge-seeking by learning agents, which can benefit from estimation of
epistemic uncertainty, i.e., a measure of lack of knowledge. The epistemic
uncertainty can potentially be eliminated with enough data if the learner
converges to a Bayes-optimal predictor, i.e. one with the lowest possible
expected loss at every input point. EU estimation is already a key ingredient
in active learning and Sequential Model Optimization (SMO) [2, 21] as well as
exploration in Reinforcement Learning (RL) [36, 71]. EU estimators can tell us
how much could be gained from learning around a particular area of state-space
or input data space. But how should we even quantify it? Much previous work
has focused on model variance [22, 40] as a proxy thereof, i.e., how different
from each other are functions compatible with the data and with the learner’s
preferences (for instance the choice of the parametric family or learning
algorithms). In interactive learning settings like active learning or RL, the
model variance estimated only using the currently available data is used as a
proxy of EU for acquiring new data. This, as we posit below, has two
fundamental limitations:

  * •

At each step of interactive learning, the learner observes a gap between the
observed values at the acquired points and its prediction at those points.
Yet, this gap is typically not used to calibrate its uncertainty estimates at
similar points in the next acquisition steps. As an illustration for this
issue, consider the function shown in Fig. 1 (top, in black), and an
interactive learner that seeks to improve its estimate of the ground-truth
function by acquiring more points. If the acquisition relies on a variance-
based proxy of EU (as in Fig. 1 with a Gaussian Process, or GP), then the EU
estimate might even become worse in regions where the learner (post
acquisition) knows it underestimated the uncertainty. In other words, there is
no reason for such a learner to extrapolate the mismatch between the predicted
error and the true error (which we show below to be related to EU) to similar
points in subsequent rounds. In this example, where we consider one round of
acquisition, there is no mechanism that allows the second GP to look at the
first GP’s variance and the ground-truth value of the acquired points in order
to adjust its own variance estimates in that region, which should ideally be
larger considering the poor calibration in the round before.

  * •

In most cases, the learner has some bias (misspecification) associated with
it, due to finite capacity or preferences, self-imposed to avoid overfitting.
Particularly in the case of neural network learners, the capacity can be
increased with more data and more training: for example, early stopping will
choose a higher effective capacity with more data. This bias, which is not
incorporated in the model variance, is important to account for in the EU.
Recent work [50, 13] has also shown that parametric Bayesian methods are
suboptimal for learning predictive models when the model is misspecified. In
Section 1, we expand on the brittle relationship between model variance and
EU, which can get murky in the presence of bias or the very coarse
approximations of the posterior distribution over neural networks that are
typically used, e.g., with MC-Dropout [22] or Deep Ensembles [40].

Motivated by interactive learning settings arising for example in SMO and RL,
we study —as an alternative or complement—the quantification of EU in terms of
out-of-sample loss, which does include the bias term discussed above, and
allows for recalibration as more data is acquired. However, the total expected
out-of-sample loss at a point includes not just EU (reducible loss) but also
an aleatoric (irreducible) part. We thus propose a definition of EU as the
difference between the expected generalization error and aleatoric
uncertainty. Estimating it is useful to an interactive learner because it
could be reduced with enough data, especially in regions of the input space we
care about, i.e., where EU is large. We thus propose DEUP, for Direct
Epistemic Uncertainty Prediction, where we train a side learner (the
uncertainty predictor) with an appropriate objective and appropriate data in
order to predict this generalization error, and then subtract an estimate of
aleatoric uncertainty which can be estimated using multiple samples from the
oracle as discussed in Sec. 2. Note that our method is agnostic to the
particular search method still needed to select points with high EU in order
to propose new candidates for active learning [2, 57, 7] or SMO [39, 30, 67],
or exploration in RL [36, 59, 29].

Figure 1: Top.A GP is fit on (dark blue) points in $[0,0.5]\cup[1.5,2]$. The
shaded area represents the GP standard deviation, often used as a proxy for
epistemic uncertainty. Bottom left. A second GP fit on the same points, with 5
extra points (green). This second GP predicts almost 0 standard deviation
everywhere, even though the first GP significantly underestimated the
uncertainty in $[0.5,1.5]$, because no signal is given to the learner that the
region $[0.5,1.5]$ should be explored more. Bottom right. DEUP learns to map
the underestimated variances of the first GP to the L2 errors made by that GP,
and yields more reasonable uncertainty estimates that should inform the
learner of what area to explore.

A unique advantage of DEUP, compared with methods relying on model variance,
is that it can be explicitly trained to care about, and calibrate for
estimating the uncertainty for examples which may come from a distribution
slightly different from the distribution of most of the training examples.
Such distribution shifts arise naturally in contexts like active learning and
RL because the learner explores new areas of the input space. In these non-
stationary settings, we typically want to retrain the main predictor as we
acquire new training data, not just because more training data is generally
better but also to better track the changing distribution and generalize to
yet unseen but upcoming OOD inputs. This setting makes it challenging to train
the main predictor but it also entails a non-stationarity for the training
data seen by the error predictor: a large error initially made at a point $x$
before $x$ is incorporated in the training set (along with an outcome $y$)
will typically be greatly reduced after updating the main predictor with
($x,y$). To cope with this non-stationarity in the targets of the error
predictor, we propose in Sec. 2.2 to use additional features as input to the
uncertainty estimator, that are informative of both the input point and the
dataset used to obtain the current predictor. We mainly use density estimates
and model variance estimates, that sometimes come at no additional cost.

In summary, the main contributions of this paper are the following:

  * •

In Sec. 1, we propose a definition of epistemic uncertainty as the reducible
part of generalization error, and discuss how this novel interpretation
addresses the shortcomings of variance-based proxies of EU.

  * •

In Sec. 2, we present DEUP, a principled framework for estimating epistemic
uncertainty that is particularly interesting in interactive settings, and
propose a means to mitigate the issue of non-stationarity arising in such
settings.

  * •

In Sec. 4, we experimentally show how EU estimates from DEUP can improve upon
existing SMO methods and drive exploration in RL. We then evaluate the quality
of these uncertainty estimates in probabilistic image classification and in a
regression task predicting synergies of drug combinations.

##  1 Aleatoric and Epistemic Uncertainty

Consider a supervised learning algorithm (or learner) $\cal L$ mapping a
dataset ${\cal D}$ to a predictive function $\hat{f}={\cal L}({\cal D})$.
$\cal L$ tries to minimize the expected value of a supervised learning loss
$l(\hat{f}(x),y)\in\mathbb{R}$ under unknown probability measure $P(Y|x)$. In
this section, we focus on regression tasks with the squared loss
$l(\hat{y},y)=(\hat{y}-y)^{2}$, with $y\in{\mathbb{R}}$. In Appendix B, we
provide the corresponding results for a general loss function, with discrete
or continuous outputs.

###### Definition 1.1

The expected loss of a predictor $\hat{f}$ at $x$ is defined as:

$\displaystyle{\cal U}(\hat{f},x)=\int(\hat{f}(x)-y)^{2}dP(y|x).$ (1)

The expected loss is an unknown scalar, as we generally do not have access to
the true data distribution $P(Y|x)$. The errors made by any predictor
$\hat{f}$ at $x$ are due to both the inherent randomness of $P(Y|x)$
(aleatoric uncertainty) and the lack of knowledge of the predictor that can be
tackled by acquiring more information around $x$ (epistemic uncertainty).
Because of this natural decomposition of the expected loss, we will also refer
to it as total uncertainty, and we will use the two terms interchangeably.

Bayes-optimal predictors $f^{*}$ satisfy the following equation at every $x$:

$\displaystyle\forall\tilde{y}\in{\mathbb{R}}\quad\int(f^{*}(x)-y)^{2}dP(y|x)\leq\int(\tilde{y}-y)^{2}dP(y|x).$

They depend on the underlying data distribution only and not on learner $\cal
L$ or trained predictor $\hat{f}$. Additionally, for any $x$, all Bayes-
optimal predictors have the same total uncertainty at that $x$. The error made
by a Bayes-optimal predictor is irreducible and we define it as the aleatoric
uncertainty:

###### Definition 1.2

The aleatoric uncertainty at $x$ is the total uncertainty of any Bayes-optimal
predictor $f^{*}$ at $x$:

$\displaystyle{\cal A}(x)={\cal U}(f^{*},x).$ (2)

and we note that by definition ${\cal A}(x)\leq{\cal U}(f,x),\;\forall
f\;\forall x$.

We now define the epistemic uncertainty of a predictor $\hat{f}$ as the gap
between the error of $\hat{f}$ at $x$ and the lowest possible error at $x$,
i.e., the reducible part of the loss, given more knowledge.

###### Definition 1.3

The epistemic uncertainty ${\cal E}(\hat{f},x)$ of a predictor $\hat{f}$ at
$x$ is given by

$\displaystyle{\cal E}(\hat{f},x)={\cal U}(\hat{f},x)-{\cal A}(x)={\cal
U}(\hat{f},x)-{\cal U}(f^{*},x).$ (3)

Using these definitions, we can present our main result for the regression
setting.

###### Proposition 1

In a regression task with Gaussian ground truth $P(y|x)={\mathcal{N}}(y;\
f^{*}(x),\ \sigma^{2}(x))$,

$\displaystyle{\cal E}(\hat{f},x)=(\hat{f}(x)-f^{*}(x))^{2}$
$\displaystyle{\cal
U}(\hat{f},x)=\mathbb{E}_{P(y|x)}[(\hat{f}(x)-y)^{2}]=(\hat{f}(x)-f^{*}(x))^{2}+\sigma^{2}(x)$
$\displaystyle{\cal A}(x)=\mathbb{E}_{P(y|x)}[(f^{*}(x)-y)^{2}]=\sigma^{2}(x)$

The proof is provided in Appendix C.

$\mathcal{S}$$\hat{f}$$\tilde{f}$$f^{*}$$y$bias$b$$\Delta f$$\approx$std.
dev.noise$\varepsilon(\hat{f})$

Figure 2: Illustrating noise and bias. Observed $y$ is independent noise plus
true $\mathbb{E}[Y|x]=f^{*}(x)$, itself best approximated by unknown
$\tilde{f}(x)$ in parametric set $S$ (orange), e.g., using Bayesian posterior
distribution $p(f|\mathcal{D})$ (grey) over parameters. $\tilde{f}$ is the
closest function in $S$ to $f^{*}$, leading to a bias
$b(x)=f^{*}(x)-\tilde{f}(x)$. $\varepsilon(\hat{f})(x)=f^{*}(x)-\hat{f}(x)$ is
the reducible error of the main predictor $\hat{f}$ (e.g. posterior mean),
whose square corresponds to EU or lack of knowledge, that DEUP aims at
estimating. With $\tilde{f}$ the unknown ideal predictor and $\hat{f}$ the
actual (e.g. mean) predictor, the square of $\Delta
f(x)=\tilde{f}(x)-\hat{f}(x)$ induces variance in the posterior.
$\varepsilon(\hat{f})(x)=b(x)+\Delta f(x)$ indicates that using the variance
as a proxy for EU misses out a non-negligible quantity: the bias $b(x)$.

Relation to existing notions of EU: Consider a parametric model
$p(Y|x,\theta)$ and a learner maintaining a distribution over parameters
$\theta\in\Theta$, each corresponding to a predictor $f$ in a parametric set
of functions $S$, possibly starting from a prior $p(\theta)$ that would lead
to a posterior distribution $p(\theta|\mathcal{D})$. Clearly, the fact that
multiple $\theta$’s and corresponding values of $f$ are compatible with the
data and the prior indicates lack of knowledge. Because the lack of knowledge
indicates where an interactive learner should acquire more information, this
justifies the usage of dispersion measures, such as the variance or the
entropy of the posterior predictive, as proxies for EU.

However, the limited capacity of $S$ or the prior $p(\theta)$ may keep the
optimal $\tilde{f}$ in $S$ at a distance from the Bayes-optimal predictor
$f^{*}$. We can refer to these self-imposed constraints as a form of bias, in
the sense that the learner is usually biased towards the prior preferences,
e.g., towards smoother predictors (typically more so when the dataset is
smaller). This arises when training neural networks with limited data, where
the network may not use all of its capacity due to implicit (and not fully
understood) regularization properties of SGD [31], explicit regularization,
early stopping or a combination of these, which can induce a preference on the
functions it learns. In Fig. 2, we illustrate this gap with the bias function
$b(x)=f^{*}(x)-\tilde{f}(x)$. Because of this bias, model variance cannot be
an accurate measure of EU $\mathcal{E}(\hat{f},x)$ in the general case.

In Deep Ensembles [40], for example, if all the networks in the ensemble tend
to fail in systematic (i.e. potentially reducible) ways, this aspect of
prediction failure will not be captured by variance. Whereas Deep Ensembles
variance provides us uncertainty regarding which of the networks we could draw
is best, this does not tell us how poor that network is even in a noise-free
setting. On the other hand, with flexible models like neural networks, adding
examples around $x$ where $b(x)^{2}$ is large may allow to increase capacity
around $x$ and reduce $b(x)^{2}$.

##  2 Direct Epistemic Uncertainty Prediction

DEUP (Direct Epistemic Uncertainty Prediction) uses observed out-of-sample
errors in order to train an error predictor which can be used to estimate
epistemic uncertainty elsewhere, as suggested directly by Def. 1.1-1.3. These
may be in-distribution or out-of-distribution errors, depending on what we
care about and the kind of data that is available. In Algo. 1, we provide the
pseudo-code for DEUP in interactive settings. The pseudo-code for the simpler
version with a fixed training set is given in Appendix D. We consider three
separate cases for the aleatoric uncertainty:

  1. 1.

If we know that ${\cal A}(x)=0$ then $u(x)$ is an estimate of ${\cal
E}(\hat{f},x)$ as well as of ${\cal U}(\hat{f},x)$. In this case, we choose
$a$ to be the zero function in Algo. 1.Example: Sec. 4.1. Also, in cases where
it is not possible to estimate the aleatoric uncertainty, we can rely on the
total uncertainty estimates as a proxy for EU. Example: Sec. 4.3.

  2. 2.

If an estimator $x\rightarrow a(x)$ of aleatoric uncertainty is available,
then $u(x)-a(x)$ becomes an estimate of the epistemic uncertainty of $\hat{f}$
at $x$. As an example for this scenario, consider an active learning setting,
where the evaluation of a new candidate requires wet lab experiments, where we
know the margins of errors of the instruments used.

  3. 3.

When we have access to an oracle that samples $Y$ given query $x$ from the
environment $P(Y|x)$ (e.g., in active learning or SMO), then ${\cal A}(x)$ can
be estimated using the empirical variance of different outcomes of the oracle
at the same input $x$; see Appendix E. It is common practice to perform
replicate experiments in biological assays [42, 64]. Variation across
replicates, for a given input x, provides information about the amount of
noise associated with x, i.e. the magnitude of the aleatoric uncertainty
${\cal A}(x)$.

###  2.1 Fixed Training Set

Consider the scenario with a fixed training set
$\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N_{train}}$, where $x_{i}\sim P(X)$
and $y_{i}\sim P(Y|X=x_{i})=\mathcal{N}(Y;f^{*}(x),\sigma^{2}(x))$. The goal
is to obtain an estimator of the total uncertainty $\mathcal{U}(\hat{f},.)$
for $\hat{f}=\mathcal{L}(\mathcal{D})$, that can be readily applied for input
points $x$ sampled from the same distribution $P(X)$.

According to Prop.1, we can train a secondary predictor $u$ on (input, target)
pairs $(x,e)$, where $e=(y-\hat{f}(x))^{2}$ in order to obtain such an
estimator. To see this, note that at the limit of infinite data, if the
learning algorithm ensures asymptotic convergence to a Bayes-optimal predictor
(such as $k-$nearest neighbors with $k$ increasing at a proper rate, or neural
networks whose size and regularization are hyperparameters optimized on a
growing validation set), the resulting estimates $u(x)$ converge to
$\mathbb{E}_{y\sim P(y|x)}[(\hat{f}(x)-y)^{2}]=\mathcal{U}(\hat{f},x)$. These
(input, target) pairs can be obtained from a hold-out dataset
$\mathcal{D}^{OOS}=\\{(x_{i},y_{i})\\}_{i=N_{train}+1}^{N_{train}+N_{OOS}}$,
sampled from the same distribution as $\mathcal{D}$.

###  2.2 Interactive Settings

Data: $\mathcal{D}_{init}$ initial dataset with pairs $(x,y)\in{\cal
X}\times\mathbb{R}$
$a:{\cal X}\mapsto\mathbb{R}$, estimator of aleatoric uncertainty
$\hat{f}:{\cal X}\mapsto\mathbb{R}$, main predictor (of $y$ given $x$)
$u:{\cal X}\mapsto\mathbb{R}$, total uncertainty estimator
$\phi:{\cal X}\mapsto\mathbb{R}^{k}$, chosen stationarizing features
$\pi$: acquisition machinery that proposes new input points from $\cal X$,
using the current $\hat{f}$ and EU estimates.

$\mathcal{D}_{u}\leftarrow\emptyset$, training dataset for $u$
$\mathcal{D}\leftarrow\mathcal{D}_{init}$, dataset of training points for
$\hat{f}$ seen so far
$x_{acq}\leftarrow\emptyset,y_{acq}\leftarrow\emptyset$
Optional: Pre-fill $\mathcal{D}_{u}$ using Algo. 2 and fit $u$ on
$\mathcal{D}_{u}$
while  _stopping criterion not reached_ do

Optional: Fit $a$ on $\mathcal{D}$ if necessary (e.g. see Appendix E)
Fit predictor $\hat{f}$ and features $\phi$ on $\mathcal{D}$
$\mathcal{D}_{u}\leftarrow\mathcal{D}_{u}\cup\\{(\phi(x_{acq}),(y_{acq}-\hat{f}(x_{acq}))^{2})\\}$
Fit $u$ on $\mathcal{D}_{u}$
$x_{acq}\sim\pi(.|\hat{f},u-a)$ (can be either a single point, or a batch of
points)
Sample outcomes from the ground truth distribution: $y_{acq}\sim P(\ .\
|x_{acq})$
$\mathcal{D}_{u}\leftarrow\mathcal{D}_{u}\cup\\{(\phi(x_{acq}),(y_{acq}-\hat{f}(x_{acq}))^{2})\\}$
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(x_{acq},y_{acq})\\}$

end while

Algorithm 1 Training procedure for DEUP in an Active Learning setting

Interactive settings, in which EU estimates are used to guide the acquisition
of new examples, provide a more interesting use case for DEUP. They introduce
however their own challenges, as the main predictor is retrained multiple
times with the newly acquired points, which we address below.

We consider an initial training dataset
$\mathcal{D}^{0}=\\{(x^{0}_{i},y^{0}_{i})\\}_{i=1}^{N_{0}}$. At each round
$t>0$, we have access to a trained predictor $\hat{f}^{t-1}$, estimating the
unknown ground-truth $f^{*}$, and an estimator $\hat{e}^{t-1}$ of its EU.
Suppose there is an acquisition function, that given any predictor $\hat{f}$
and an estimator $\hat{e}$ of its EU, defines a distribution on the input
space $\mathcal{X}$: $\pi(X|\hat{f},\hat{e})$, from which we can sample points
to add to the training set. Assume that at round $t>0$, $N_{t}$ points
$\\{(x^{t}_{i},y^{t}_{i})\\}_{i=1}^{N_{t}}$ are acquired, where
$x^{t}_{i}\sim\pi(X|\hat{f}^{t-1},\hat{e}^{t-1})$, and $y^{t}_{i}\sim
P(Y|X=x^{t}_{i})=\mathcal{N}(Y;f^{*}(x),\sigma^{2}(x))$ added to the training
dataset:
$\mathcal{D}^{t}=\mathcal{D}^{t-1}\cup\\{(x^{t}_{i},y^{t}_{i})\\}_{i=1}^{N_{t}}$.
From $\mathcal{D}^{t}$, a predictor $\hat{f}^{t}=\mathcal{L}(\mathcal{D}^{t})$
is obtained, which in turn will be used to decide which $N_{t+1}$ points are
going to be acquired next, through the acquisition policy $\pi$ and an
estimator $\hat{e}^{t}$ of its EU. How can we train such an EU estimator,
without wasting $N_{OOS}$ (input, target) pairs that could be used to obtain a
better estimator of $f^{*}$?

The problem with the trivial dataset:  Inspired by the fixed training set
setting, it might be tempting to train an error predictor $\hat{e}^{t}$ on
$\mathcal{D}^{t}_{u,1}=\cup_{\tau=0}^{t}\\{(x^{\tau}_{i},(y^{\tau}_{i}-\hat{f}^{t}(x^{\tau}_{i}))^{2})\\}_{i=1}^{N_{\tau}}$.
However, $x^{\tau}_{i}$ was used to train $\hat{f}^{t}$ and so we have an in-
sample rather than OOS error. There would thus be no reason for $\hat{e}^{t}$
to provide accurate estimates in unexplored regions of the search space
$\mathcal{X}$.

An alternative is to learn from the errors made by previous versions of the
predictor, i.e. $\\{f^{\tau},\ \tau\in\\{0,\dots,t\\}\\}$, on points that were
not used to train each of them, which we can simply choose to be the
subsequent acquired points. More formally, consider the dataset
$\mathcal{D}^{t}_{u,2}=\cup_{\tau=0}^{t}\cup_{\tau^{\prime}=\tau}^{t}\\{(x^{\tau^{\prime}}_{i},(y^{\tau^{\prime}}_{i}-\hat{f}^{\tau}(x^{\tau^{\prime}}_{i}))^{2})\\}_{i=1}^{N_{\tau^{\prime}}}$.
A learning algorithm with good generalization properties (such as modern
neural networks), would in principle be able to extrapolate from this dataset,
and estimate the errors (or total uncertainty) made by $\hat{f}^{t}$ on points
$x\in\mathcal{X}$ not seen so far, i.e. belonging to what we call the frontier
of knowledge. However, as each $x_{i}^{\tau^{\prime}}$ appears
$t-\tau^{\prime}+1$ times in $\mathcal{D}^{t}_{u,2}$, with different targets
at each time (each corresponding to error made by $\hat{f}^{\tau}$ on
$x_{i}^{\tau^{\prime}}$, for $\tau\in\\{\tau^{\prime},\dots,t\\}$), there is
no reason to assume that the resulting predictor would estimate the errors of
$\hat{f}^{t}$. The reason is that the targets are actually functions of not
only the input $x$, but also of the dataset used to train the different
predictors $\hat{f}^{\tau}$, i.e. $\mathcal{D}^{\tau}$. This is actually
obvious when we write
$(y^{\tau^{\prime}}_{i}-\hat{f}^{\tau}(x^{\tau^{\prime}}_{i}))^{2}=\mathcal{U}(\mathcal{L}(\mathcal{D}^{\tau}),x^{\tau^{\prime}}_{i})$.
This means that in order to obtain an estimator $\hat{e}^{t}$, we should train
a general error estimator $\hat{e}$, with inputs of the form $(\mathcal{D},x)$
and targets of the form $(y-\mathcal{L}(\mathcal{D})(x))^{2}$, using the
historical dataset
$\mathcal{D}^{t}_{u,2}=\cup_{\tau=0}^{t}\cup_{\tau^{\prime}=\tau}^{t}\\{((\mathcal{D}^{\tau},x^{\tau^{\prime}}_{i}),(y^{\tau^{\prime}}_{i}-\hat{f}^{\tau}(x^{\tau^{\prime}}_{i}))^{2})\\}_{i=1}^{N_{\tau^{\prime}}}$,
and then define $\hat{e}^{t}(x)$ to be $\hat{e}(\mathcal{D}^{t},x)$ for any
$x\in\mathcal{X}$. In summary, it is the acquired points, before they are
added to the training set, that play the role of out-of-sample (they are
actually OOD) points to train the EU estimator.

However, $\mathcal{D}$ being a very high-dimensional object, with size growing
with the number of acquired points, we may face severe overfitting issues when
using $(\mathcal{D},x)$ as inputs. We thus propose using stationarizing
features of the dataset $\mathcal{D}$ at $x$, that we denote by
$\phi_{\mathcal{D}}(x)$, as inputs to the error predictor instead of
$(\mathcal{D},x)$ (we typically include $x$ in $\phi_{\mathcal{D}}(x)$). These
stationarizing features also address concerns with directly estimating
uncertainty using a second order predictor trained simply using L2 error,
raised in [8].

In this paper, we explored
$\phi_{\mathcal{D}}(x)=\left(x,s,\hat{q}(x|\mathcal{D}),\hat{V}(\tilde{\cal
L},\mathcal{D},x)\right)$, where $\hat{q}(x|\mathcal{D})$ is a density
estimate from data $\mathcal{D}$ at $x$, $s=1$ if $x\in\mathcal{D}$ otherwise
$0$, $\tilde{\cal L}$ a learner that produces a distribution over predictors,
e.g. a GP or a Deep Ensemble [40]), and $\hat{V}(\tilde{\cal
L},\mathcal{D},x)$ is an estimate of the model variance of $\tilde{\cal L}$ at
$x$. Note that $\tilde{\mathcal{L}}$ can be chosen to be the same as
$\mathcal{L}$ For numerical reasons, we found it preferable to use
$\log\hat{q}$ and $\log\hat{V}$ instead of $\hat{q}$ or $\hat{V}$ as input
features. $\hat{q}$ can be obtained by training a density estimator (such as a
Kernel Density Estimator or a flow-based deep network [63], in our case). Like
the other predictors, the density estimator also needs to be fine-tuned when
new data is added to the training set. While these features are not required
per se to train DEUP, they provide clues to help training the uncertainty
estimator, and one can play with the trade-off of computational cost versus
usefulness of each clue. They sometimes come at no greater cost, if our main
predictor is the mean prediction of the learner’s output distribution, and if
we use the corresponding variance as the only extra feature, as is the case in
the experiments of Sec. 4.1 with GPs.

In our experiments, we found that using inputs $\phi_{\cal D}(x)$, or even a
subset of the $4$ possible features, is sufficient to train an uncertainty
estimator with targets $(\hat{f}(x)-y)^{2}$.

For computational reasons, it is unfeasible to store all previous datasets and
all previous predictors $\hat{f}^{\tau}$, and we propose a subset of
$\mathcal{D}^{t}_{u,2}$, that makes it possible to have an online algorithm,
without memorization:

$\mathcal{D}^{t}_{u}=\cup_{\tau=0}^{t}\cup_{\tau^{\prime}=\tau}^{\tau+1}\\{(\phi_{\mathcal{D}^{\tau}}(x^{\tau^{\prime}}_{i})),(y^{\tau^{\prime}}_{i}-\hat{f}^{\tau}(x^{\tau^{\prime}}_{i}))^{2})\\}_{i=1}^{N_{\tau^{\prime}}}.$

The corresponding pseudo-code is provided in Algo. 1. Note that ${\cal D}_{u}$
is incremented twice with $x_{acq}$ but $\phi_{\cal D}(x_{acq})$ is different
each time because $x_{acq}$ first is and then is not yet in ${\cal D}$.

####  2.2.1 Pretraining the error predictor

Consider an interactive setting, and suppose that the oracle is so costly that
we cannot afford to wait for a few rounds of acquisition in order to build a
training dataset large enough for the uncertainty estimator to provide
reasonable EU estimates. To this end, we propose a cross-validation strategy
to pretrain the secondary learner before any acquisition step, using pairs
$(\phi_{\tilde{D}}(x),(f_{\tilde{D}}(x)-y)^{2})$,

Input: $\mathcal{D}_{u}$
while  _$|\mathcal{D}_{u}| <N_{pretrain}$_ do

Split $\mathcal{D}_{init}$ into $K$ random subsets
$\mathcal{D}_{1},\dots,\mathcal{D}_{K}$ of equal size. Define $\tilde{\cal
D}=\bigcup_{k=1}^{K-1}\mathcal{D}_{k}$
Fit a new predictor $\hat{f}$ on $\tilde{\mathcal{D}}$, and fit the features
$\phi$ on $\tilde{\mathcal{D}}$
$\mathcal{D}_{u}\leftarrow\mathcal{D}_{u}\cup\bigcup_{(x,y)\in\mathcal{D}}\\{(\phi(x),(y-\hat{f}(x))^{2})\\}$

end while

Algorithm 2 Pre-filling the uncertainy estimator training dataset
$\mathcal{D}_{u}$

where $\tilde{D}$ is any subset of the initially available data, that we use
to train a predictor $f_{\tilde{D}}=\mathcal{L}(\tilde{D})$, and $(x,y)$ any
pair in $D$, whether in $\tilde{D}$ or not. With this pretraining phase, we
obtain better uncertainty estimates as soon as the first batch of acquired
points come in. There are several ways to build the pretraining dataset. We
present one of them in Alg. 2. The procedure stops when the training dataset
for the secondary learner, $\mathcal{D}_{u}$, contains at least $N_{pretrain}$
elements. In our experiments, we choose $N_{pretrain}$ to be a small multiple
of the number of initial training points.

##  3 Related Work

In [35], the authors characterize the sources of uncertainty as aleatoric
(inherent noise) and epistemic (incomplete knowledge). Using Gaussian
Processes [79] or GP is a popular EU estimation because the variance among the
functions in the posterior (given the data) can be computed analytically.

In a deep learning context, [9, 32, 16] use the posterior distribution of
network weights [44] in Bayesian Neural Networks (BNNs) to capture EU. Other
techniques that rely on measuring the discrepancy between different predictors
as a proxy for EU include MC-Dropout [22], that interprets Dropout [26] as a
variational inference technique in BNNs. These approaches, because they rely
on sampling multiple sets of weights or dropout masks at inference time, share
some similarities with ensemble-based methods, that include bagging [10] and
boosting [18], in which multiple predictors are trained and their outputs are
averaged to make a prediction, although the latter measure variability due to
the training set instead of the spread of functions compatible with the given
training set, as in Bayesian approaches. For example, [66] use Random Forests
[11] to estimate EU. Deep Ensembles [40] are closer to the Bayesian approach,
using an ensemble of neural networks that differ because of randomness in
initialization and training (as you would have with MCMC, albeit in a more
heuristic way). In addition to this, several other variants of this central
idea of measuring discrepancy between different predictors have been proposed
recently [47, 70, 3, 43, 76, 78, 4, 34, 46, 75, 55]. We discuss some of these
methods in more detail in Appendix A.

More closely related to DEUP, [81] propose a loss prediction module for
learning to predict the value of the loss function. [27] also propose using a
separate network that learns to predict the variance of an ensemble. These
methods, however, are trained only to capture the in-sample error, and do not
capture the out-of-sample error which is more relevant for scenarios like
active learning where we want to pick $x$ where the reducible generalization
error (see Def. 1.3) is large. EpiOut [72, 23] propose learning a binary
output that simply distinguishes between low or high EU.

##  4 Experiments

In our experiments, we focus on interactive settings, where having good
uncertainty estimates is essential for efficient acquisition. As explained in
Sec. 2.2, it is the acquired points, before they are used to retrain the main
predictor, that act as the out-of-sample examples to train DEUP. In RL,
because the targets (e.g. of Q-Learning) are themselves estimates and moving,
data seen at any particular point is normally out-of-sample and can inform the
uncertainty estimator, when the inputs are used with the stationarizing
features. We emphasize that in order to make fair comparisons, DEUP does not
have access to any additional OOD data during training, in any of the
experiments that follow. Instead, we use Algo. 2 to generate the OOD data used
for training the error predictor. Finally, note that in terms of computational
cost, training DEUP with density and model variance as stationarizing features
is on-par with training a ensemble of $5$ networks.

###  4.1 Sequential Model Optimization

Sequential model optimization is a form of active learning, where at each
stage, the learner chooses query examples to label, looking for examples with
a high value of the unknown oracle function. Such examples are selected so
they have a high predicted value (to maximize the unknown oracle function) and
a large predicted uncertainty (offering the opportunity of discovering yet
higher values). Acquisition functions, such as Upper Confidence Bound (UCB,
[68]) and Expected Improvement (EI, [52]) trade-off exploration and
exploitation, and one can select the next candidate by looking for $x$’s
maximizing the acquisition function. We combine UCB and EI with DEUP (DEUP-UCB
and DEUP-EI) to perform active learning, treating the main predictor and DEUP
EU predictions at $x$ respectively as mean and variance of a Gaussian
distribution for the learner’s guess of the value of the oracle at $x$. We
showcase how using DEUP to calibrate GP variances (used as the only extra
input for DEUP) allows for better performances in higher-dimensional
optimization tasks. Specifically, we compare DEUP-EI to TuRBO-EI [19], a
state-of-the-art method for sequential optimization, that fits a collection of
local GP models instead of a global one in order to perform efficient high-
dimensional optimization, on the Ackley function [1] as oracle, a common
benchmark for optimization algorithms. The oracle function can be defined for
arbitrary dimensions, and has many local minima.

In Fig. 3, we compare different methods on the Ackley-10 function, in addition
to the optimum reached in budget-constrained optimization problems for
different oracle input dimensions, and we find that adapting DEUP to TuRBO
consistently outperforms regular TuRBO, especially in higher dimensions. See
also Appendix F for 1D and 2D SMO tasks where DEUP-EI outperforms GP-EI [12],
as well as neural networks with MC-Dropout or Ensembles. We find GP-EI getting
stuck in local optima whereas DEUP-EI was able to reach the global maximum
consistently. Experimental details are provided in Appendix F.3.

Figure 3: Left. Max. value reached by the different optimization methods, for
the 10 dimensional Ackley function. In each run, all the methods start with
the same initial 20 points. Shaded areas represent the standard error across 3
runs. Right. Max. value reached in the budget-constrained setting, on the
Ackley functions of different dimensions. Error bars represent the standard
error across 3 different runs, with different initial sets of 20 pairs. The
budget is $120$ function calls in total. Higher is better and TuRBO-DEUP-EI is
less hurt by dimensionality.

###  4.2 Reinforcement Learning

Figure 4: Average regret on CartPole task. Error bars represent standard error
across 5 runs.

Similar to SMO, a key challenge in RL is efficient exploration of the input
state space. To investigate the effectiveness of DEUP’s uncertainty estimates
in the context of RL, we incorporate epistemic uncertainties predicted by DEUP
to DQN [51], which we refer to as DEUP-DQN. Specifically, we train the
uncertainty predictor with the objective to predict the TD-error, using log-
density estimates as a stationarizing feature. The predicted uncertainties are
then used as an exploration bonus in the Q-values. Details of the experimental
setup are in Appendix G. We evaluate DEUP-DQN on CartPole, a classic RL task
from bsuite [60], against DQN + $\epsilon$-greedy, DQN + MC-Dropout [22] and
Bootstrapped DQN [59]. Fig. 4 shows that DEUP achieves lower regret faster,
compared to all the baselines, which demonstrates the advantage of DEUP’s
uncertainty estimates for efficient exploration. Future work should
investigate ways to scale this method to more complex environments.

###  4.3 Uncertainty Estimation

####  4.3.1 Epistemic Uncertainty Predictions for Rejecting Difficult Examples

Epistemic uncertainty estimates can be used to reject difficult examples where
the predictor might fail, such as OOD inputs111e.g. rare but challenging
inputs can be directed to a human, avoiding a costly mistake. We thus consider
a standard OOD Detection task [76, 75], where we train a ResNet-18 [24] for
CIFAR-10 classification [37] and reject OOD examples using estimated
uncertainty in the prediction. To facilitate rejection of classes other than
those in the training set,

Model | SRCC | AUROC
---|---|---
MC-Dropout | $0.287\pm 0.002$ | $0.894\pm 0.008$
Deep Ensemble | $0.381\pm 0.004$ | $\bm{0.933\pm 0.008}$
DUQ | $0.376\pm 0.003$ | $0.927\pm 0.013$
DUE | $0.378\pm 0.004$ | $0.929\pm 0.005$
DEUP (D+V) | $\bm{0.426\pm 0.009}$ | $\bm{0.933\pm 0.010}$
Table 1: Spearman Rank Correlation Coefficient (SRCC) between predicted
uncertainty and OOD generalization error (SVHN); Area under ROC Curve (AUROC)
for OOD Detection (SVHN) with CIFAR-10 ResNet-18 models (3 seeds). DEUP
significantly outperforms baselines in terms of SRCC and is equivalent to Deep
Ensembles but scoring better than the other methods in terms of the coarser
AUROC metric.

we use a Bernoulli Cross-Entropy Loss for each class [76]:
$l(\hat{f}(x),y)=-\sum_{i}y_{i}\log\hat{f}_{i}(x)+(1-y_{i})\log(1-\hat{f}_{i}(x))$,
where $y$ is a one-hot vector ($y_{i}=1$ if $i$ is the correct class, and $0$
otherwise), and $\hat{f}_{i}(x)=$ predicted probability for class $i$, so the
target for out-of-distribution data (from other classes) is
$y=\\{0,\dots,0\\}$. To ascertain how well an epistemic error estimate sorts
non-training examples by the above NLL loss, we consider the rank correlation
between the predicted uncertainty and the observed OOD generalization error on
SVHN examples [56]. This metric focuses on the quality of the uncertainty
estimates rather than just their ability to simply classify in- vs out-of-
distribution. We also report the AUC for the OOD detection task; more details
in Appendix H. Table 1 shows that with the variance from DUE [75] and the
density from MAF [62] as stationarizing features, we obtain uncertainty
estimates that have high rank correlation with the underlying generalization
errors and competitive AUROC, compared with the baselines. In addition, since
the error predictor is trained separately from the main predictor, there is no
explicit trade-off between the accuracy of the main predictor and the quality
of uncertainty estimates. We achieve competitive accuracy of $93.89\%$ for the
main predictor. We ignore the effect of aleatoric uncertainty (due to
inconsistent human labelling), which would require a human study to ascertain.
We note that we choose the DUE baseline as it is representative of related
methods such as SNGP [43] and DDU [55], and performs best in our experiments.
We present additional results in a distribution shift setting in Appendix H.
Note that in the pretraining phase of the uncertainty estimator (Alg. 2), we
obtain the subsets by splitting the data based on classes, with each split
containing $\lfloor n/K\rfloor$ classes. So when we train on $K-1$ subsets,
the $\lfloor n/K\rfloor$ classes from the remaining subset become _out-of-
distribution_.

####  4.3.2 Epistemic Uncertainty Estimation for Drug Combinations

(a) Ensemble

(b) DEUP

Model Corr. w. res. U. Bound Ratio Log Likelihood MC-Dropout $0.14\pm 0.07$
$0.56\pm 0.05$ $0.25\pm 0.12$ $-20.1\pm 6.8$ Deep Ensemble $0.30\pm 0.09$
$0.59\pm 0.04$ $0.50\pm 0.13$ $-14.3\pm 4.7$ DUE $0.12\pm 0.12$ $0.15\pm 0.03$
$\bm{0.80\pm 0.79}$ $-13.0\pm 0.52$ DEUP $\bm{0.47\pm 0.03}$ $0.63\pm 0.05$
$\bm{0.75\pm 0.07}$ $\bm{-3.5\pm 0.25}$

(a) Quality of uncertainty estimates from different methods.

Figure 5: Drug Combinations. Predicted mean and uncertainty (error bars) on 50
test examples ordered by increasing value of true synergy score (orange).
Model predictions and uncertainties in blue. Ensemble (a) (and MC-dropout, not
shown) consistently underestimate uncertainty while DEUP (b) captures the
right order of magnitude. (c) Corr. w. res. shows correlation between model
residuals and predicted uncertainties $\hat{\sigma}$. A best-case Upper Bound
on Corr. w. res. is obtained from the correlation between $\hat{\sigma}$ and
true samples from $\mathcal{N}(0,\hat{\sigma})$. Ratio is the ratio between
col. 1 and 2 (larger is better). Log-likelihood: average over 3 seeds of per
sample predictive log-likelihood.

We validate DEUP in a real-world regression task predicting the synergy of
drug combinations. While much effort in drug discovery is spent on finding
novel small molecules, a potentially cheaper method is identifying
combinations of pre-existing drugs which are synergistic (i.e., work well
together). However, every possible combination cannot be tested due to the
high monetary cost and time required to run experiments. Therefore, developing
good estimates of EU can help practitioners select experiments that are both
informative and promising. As shown in Table 2(a), the out-of-sample error
predicted by DEUP correlates better with residuals of the model on the test
set in comparison to several other uncertainty estimation methods. Moreover,
DEUP better captured the order of magnitude of the residuals as shown in Fig.
5. Details on experiments and metrics are in Appendix I.

##  5 Conclusion and Future Work

Whereas standard measures of epistemic uncertainty focus on variance, we argue
that bias (misspecification) can also be reduced with predictors like neural
networks. In a regression setup, the expected out-of-sample squared error
minus aleatoric uncertainty thus captures all the uncertainty about the
ground-truth function $\mathbb{E}[Y|x]$, that more data can reduce. This
motivates the DEUP framework, where we train a second network to predict the
errors of the first. In interactive settings, this nonetheless raises non-
stationarity challenges for this estimator, and we propose extra input
features to tackle this issue, and show experimentally their advantages.
Future work should investigate ways to improve the computational efficiency of
DEUP, and ways to estimate aleatoric uncertainty when no estimator thereof is
readily available and when one cannot simply query an oracle several times on
the same input $x$.

## Acknowledgements

The authors would like to thank Tristan Deleu, Anirudh Goyal, Tom Bosc, Léna
Néhale Ezzine, Doina Precup, Pierre Luc-Bacon, John Bradshaw, José Miguel
Hernández-Lobato as well as anonymous reviewers for useful comments and
feedback. This research was enabled in part by support provided by Compute
Canada, the Bill & Melinda Gates Foundation, IVADO and a CIFAR AI Chair. The
authors also acknowledge funding from Samsung, IBM, Microsoft.

## References

  * [1] D. Ackley.  A connectionist machine for genetic hillclimbing, volume 28.  Springer Science & Business Media, 2012.
  * [2] C. C. Aggarwal, X. Kong, Q. Gu, J. Han, and S. Y. Philip.  Active learning: A survey.  In Data Classification: Algorithms and Applications, pages 571–605. CRC Press, 2014.
  * [3] A. Amini, W. Schwarting, A. Soleimany, and D. Rus.  Deep evidential regression.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [4] J. Antoran, J. Allingham, and J. M. Hernández-Lobato.  Depth uncertainty in neural networks.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [5] M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy.  BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization.  In Advances in Neural Information Processing Systems 33, 2020.
  * [6] P. J. Ballester and J. B. Mitchell.  A machine learning approach to predicting protein–ligand binding affinity with applications to molecular docking.  Bioinformatics, 26(9):1169–1175, 2010.
  * [7] E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio.  Flow network based generative models for non-iterative diverse candidate generation.  NeurIPS’2021, arXiv:2106.04399, 2021.
  * [8] V. Bengs, E. Hüllermeier, and W. Waegeman.  On the difficulty of epistemic uncertainty quantification in machine learning: The case of direct uncertainty estimation through loss minimisation.  arXiv preprint arXiv:2203.06102, 2022.
  * [9] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra.  Weight uncertainty in neural network.  In International Conference on Machine Learning, pages 1613–1622. PMLR, 2015.
  * [10] L. Breiman.  Bagging predictors.  Machine learning, 24(2):123–140, 1996.
  * [11] L. Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * [12] A. D. Bull.  Convergence rates of efficient global optimization algorithms.  Journal of Machine Learning Research, 12(10), 2011.
  * [13] M. R. Cervera, R. Dätwyler, F. D’Angelo, H. Keurti, B. F. Grewe, and C. Henning.  Uncertainty estimation under model misspecification in neural network regression, 2021.
  * [14] Y. Chung, W. Neiswanger, I. Char, and J. Schneider.  Beyond pinball loss: Quantile methods for calibrated uncertainty quantification.  arXiv preprint arXiv:2011.09588, 2020.
  * [15] T. Cihlar and M. Fordyce.  Current status and prospects of hiv treatment.  Current opinion in virology, 18:50–56, 2016.
  * [16] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft.  Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning.  In International Conference on Machine Learning, pages 1184–1193. PMLR, 2018.
  * [17] M. W. Dusenberry, G. Jerfel, Y. Wen, Y. Ma, J. Snoek, K. Heller, B. Lakshminarayanan, and D. Tran.  Efficient and scalable bayesian neural nets with rank-1 factors.  In International Conference on Machine Learning (ICML), 2020.
  * [18] B. Efron and R. J. Tibshirani.  An introduction to the bootstrap.  CRC press, 1994.
  * [19] D. Eriksson, M. Pearce, J. R. Gardner, R. Turner, and M. Poloczek.  Scalable global optimization via local bayesian optimization.  arXiv preprint arXiv:1910.01739, 2019.
  * [20] S. Farquhar, Y. Gal, and T. Rainforth.  On statistical bias in active learning: How and when to fix it.  ICLR’2021, arXiv:2101.11665, 2021.
  * [21] P. Frazier.  A tutorial on bayesian optimization.  ArXiv, abs/1807.02811, 2018.
  * [22] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning (ICML), pages 1050–1059. PMLR, 2016.
  * [23] D. Hafner, D. Tran, T. Lillicrap, A. Irpan, and J. Davidson.  Noise contrastive priors for functional uncertainty.  In Conference on Uncertainty in Artificial Intelligence (UAI), 2019\.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] D. Hendrycks and T. Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [26] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv preprint arXiv:1207.0580, 2012.
  * [27] S. Hu, N. Pezzotti, and M. Welling.  A new perspective on uncertainty quantification of deep ensembles.  arXiv, 2020.
  * [28] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson.  Averaging weights leads to wider optima and better generalization.  In Conference on Uncertainty in Artificial Intelligence (UAI), 2018\.
  * [29] D. Janz, J. Hron, P. Mazur, K. Hofmann, J. M. Hernández-Lobato, and S. Tschiatschek.  Successor uncertainties: Exploration and uncertainty in temporal difference learning.  In Neural Information Processing Systems (NeurIPS), 2019.
  * [30] D. R. Jones, M. Schonlau, and W. J. Welch.  Efficient global optimization of expensive black-box functions.  Journal of Global optimization, 13(4):455–492, 1998.
  * [31] S. Kale, A. Sekhari, and K. Sridharan.  Sgd: The role of implicit regularization, batch-size and multiple-epochs.  In NeurIPS, 2021.
  * [32] A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In Neural Information Processing Systems (NeurIPS), 2017.
  * [33] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In International Conference for Learning Representations, 2015.
  * [34] P. Kirichenko, P. Izmailov, and A. G. Wilson.  Why normalizing flows fail to detect out-of-distribution data.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [35] A. D. Kiureghian and O. Ditlevsen.  Aleatory or epistemic? does it matter?  Structural Safety, 31(2):105–112, 2009.  Risk Acceptance and Risk Communication.
  * [36] L. Kocsis and C. Szepesvári.  Bandit based monte-carlo planning.  In In: ECML-06. Number 4212 in LNCS, pages 282–293. Springer, 2006\.
  * [37] A. Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [38] M. Kull and P. Flach.  Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration.  In Machine Learning and Knowledge Discovery in Databases, 2015.
  * [39] H. J. Kushner.  A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise.  Journal of Basic Engineering, 86:97–106, 1964.
  * [40] B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [41] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [42] M.-L. T. Lee, F. C. Kuo, G. Whitmore, and J. Sklar.  Importance of replication in microarray gene expression studies: statistical methods and evidence from repetitive cdna hybridizations.  Proceedings of the National Academy of Sciences, 97(18):9834–9839, 2000.
  * [43] J. Z. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax-Weiss, and B. Lakshminarayanan.  Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [44] D. J. MacKay.  A practical bayesian framework for backpropagation networks.  Neural computation, 4(3):448–472, 1992.
  * [45] W. Maddox, T. Garipov, P. Izmailov, D. Vetrov, and A. G. Wilson.  A simple baseline for bayesian uncertainty in deep learning.  In Neural Information Processing Systems (NeurIPS), 2019.
  * [46] A. Malinin, S. Chervontsev, I. Provilkov, and M. Gales.  Regression prior networks, 2020.
  * [47] A. Malinin and M. Gales.  Predictive uncertainty estimation via prior networks.  In NIPS’18: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
  * [48] A. Malinin, B. Mlodozeniec, and M. Gales.  Ensemble distribution distillation.  In International Conference on Learning Representations, 2020.
  * [49] A. Malyutina, M. M. Majumder, W. Wang, A. Pessia, C. A. Heckman, and J. Tang.  Drug combination sensitivity scoring facilitates the discovery of synergistic and efficacious drug combinations in cancer.  PLoS computational biology, 15(5):e1006752, 2019.
  * [50] A. Masegosa.  Learning under model misspecification: Applications to variational and ensemble methods.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, 2020.
  * [51] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [52] J. Močkus.  On bayesian methods for seeking the extremum.  In Optimization techniques IFIP technical conference, pages 400–404. Springer, 1975.
  * [53] R. B. Mokhtari, T. S. Homayouni, N. Baluch, E. Morgatskaya, S. Kumar, B. Das, and H. Yeger.  Combination therapy in combating cancer.  Oncotarget, 8(23):38022, 2017.
  * [54] H. L. Morgan.  The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service.  Journal of Chemical Documentation, 5(2):107–113, 1965.
  * [55] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. S. Torr, and Y. Gal.  Deep deterministic uncertainty: A simple baseline, 2021.
  * [56] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In Advances in Neural Information Processing Systems (NIPS), 2011\.
  * [57] V.-L. Nguyen, S. Destercke, and E. Hüllermeier.  Epistemic uncertainty sampling.  In International Conference on Discovery Science, pages 72–86. Springer, 2019.
  * [58] W. H. Organization and S. T. Initiative.  Treatment of tuberculosis: guidelines.  World Health Organization, 2010.
  * [59] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy.  Deep exploration via bootstrapped dqn.  arXiv preprint arXiv:1602.04621, 2016.
  * [60] I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvári, S. Singh, B. Van Roy, R. Sutton, D. Silver, and H. van Hasselt.  Behaviour suite for reinforcement learning.  In International Conference on Learning Representations, 2020.
  * [61] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek.  Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift.  In Advances in Neural Information Processing Systems, 2019.
  * [62] G. Papamakarios, T. Pavlakou, and I. Murray.  Masked autoregressive flow for density estimation.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [63] D. Rezende and S. Mohamed.  Variational inference with normalizing flows.  In International Conference on Machine Learning (ICML), pages 1530–1538. PMLR, 2015.
  * [64] N. J. Schurch, P. Schofield, M. Gierliński, C. Cole, A. Sherstnev, V. Singh, N. Wrobel, K. Gharbi, G. G. Simpson, T. Owen-Hughes, et al.  How many biological replicates are needed in an rna-seq experiment and which differential expression tool should you use?  Rna, 22(6):839–851, 2016.
  * [65] M. Sensoy, L. Kaplan, and M. Kandemir.  Evidential deep learning to quantify classification uncertainty.  In Neural Information Processing Systems (NeurIPS), page 3183–3193, 2018.
  * [66] M. H. Shaker and E. Hüllermeier.  Aleatoric and epistemic uncertainty with random forests.  In International Symposium on Intelligent Data Analysis, pages 444–456. Springer, 2020.
  * [67] J. Snoek, H. Larochelle, and R. P. Adams.  Practical bayesian optimization of machine learning algorithms.  In Neural Information Processing Systems (NeurIPS), 2012.
  * [68] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger.  Gaussian process optimization in the bandit setting: No regret and experimental design.  In International Conference on Machine Learning (ICML), 2010.
  * [69] A. Subramanian, R. Narayan, S. M. Corsello, D. D. Peck, T. E. Natoli, X. Lu, J. Gould, J. F. Davis, A. A. Tubelli, J. K. Asiedu, et al.  A next generation connectivity map: L1000 platform and the first 1,000,000 profiles.  Cell, 171(6):1437–1452, 2017.
  * [70] N. Tagasovska and D. Lopez-Paz.  Single-model uncertainties for deep learning.  In Neural Information Processing Systems (NeurIPS), 2019.
  * [71] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel.  # exploration: A study of count-based exploration for deep reinforcement learning.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [72] J. Umlauft, A. Lederer, T. Beckers, and S. Hirche.  Real-time uncertainty decomposition for online learning control.  ArXiv, abs/2010.02613, 2020.
  * [73] M. P. Vadera, A. D. Cobb, B. Jalaeian, and B. M. Marlin.  Ursabench: Comprehensive benchmarking of approximate bayesian inference methods for deep neural networks.  ArXiv, abs/2007.04466, 2020.
  * [74] M. P. Vadera, B. Jalaeian, and B. M. Marlin.  Generalized bayesian posterior expectation distillation for deep neural networks.  In UAI, 2020.
  * [75] J. van Amersfoort, L. Smith, A. Jesson, O. Key, and Y. Gal.  Improving deterministic uncertainty estimation in deep learning for classification and regression.  CoRR, abs/2102.11409, 2021.
  * [76] J. R. van Amersfoort, L. Smith, Y. Teh, and Y. Gal.  Simple and scalable epistemic uncertainty estimation using a single deep deterministic neural network.  In International Conference on Machine Learning (ICML), 2020.
  * [77] M. Welling and Y. W. Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In Proceedings of the 28th International Conference on International Conference on Machine Learning, 2011.
  * [78] Y. Wen, D. Tran, and J. Ba.  Batchensemble: An alternative approach to efficient ensemble and lifelong learning.  In International Conference on Learning Representations, 2020.
  * [79] C. K. Williams and C. Rasmussen.  Gaussian processes for regression.  In Neural Information Processing Systems (NeurIPS), 1995.
  * [80] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing.  Deep kernel learning.  In A. Gretton and C. C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 51 of Proceedings of Machine Learning Research, pages 370–378, Cadiz, Spain, 09–11 May 2016. PMLR.
  * [81] D. Yoo and I. Kweon.  Learning loss for active learning.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 93–102, 2019.
  * [82] B. Zagidullin, J. Aldahdooh, S. Zheng, W. Wang, Y. Wang, J. Saad, A. Malyutina, M. Jafari, Z. Tanoli, A. Pessia, et al.  Drugcomb: an integrative cancer drug combination data portal.  Nucleic acids research, 47(W1):W43–W51, 2019.
  * [83] R. Zhang, C. Li, J. Zhang, C. Chen, and A. G. Wilson.  Cyclical stochastic gradient mcmc for bayesian deep learning.  In ICLR, 2020.
  * [84] S. Zhang, A. Golbraikh, S. Oloff, H. Kohn, and A. Tropsha.  A novel automated lazy learning qsar (all-qsar) approach: method development, applications, and virtual screening of chemical databases using validated all-qsar models.  Journal of chemical information and modeling, 46(5):1984–1995, 2006\.

## Appendices

##  Appendix A Related Work

Recently, several novel deep learning-based techniques to estimate uncertainty
with a single model have been proposed. For example, Deep Evidential
Regression [3] is a method for estimating epistemic uncertainty that is based
on a parametric estimate of model variance. This is in line with previous work
using evidential uncertainty estimation [47, 65]. Orthonormal Certificates, a
set of learned features with a suitable loss function, are used in [70]. These
certificates capture the distance to the training set to learn an estimate of
the epistemic uncertainty. This is further studied in [43] who formalize
_distance awareness_ , which captures the model’s ability to quantify the
distance of a test sample from the training data manifold, as a necessary
condition for uncertainty estimation. This distance awareness can be captured
with a weight normalization step in training, in addition to using a GP as the
output layer. DUE [75] is an instance of Deep Kernel Learning [80], which is
defined as a GP with a deep feature extractor inside the kernel. DUE improves
upon SNGP by using an inducing point GP, and bi-lipschitz constraints on the
feature extractor, giving better test set accuracies as well as improved
training efficiency. DDU [55] further extends this line of work by fitting
Gaussian Discrminant Analysis (GDA) on the feature space of a regularized
neural network. DUN [4] uses the disagreement between the outputs from
intermediate layers as a measure of uncertainty. DUQ [76] on the other hand
uses two-sided Jacobian regularization on RBF networks [41] for reliable
uncertainty estimates.

[78] present an efficient way of implementing ensembles of neural networks, by
using one shared matrix and a rank-1 matrix per member. The weights for each
member are then computed as the Hadamard product of the shared matrix and the
rank-1 matrix of the member. There has also been extensive work in scaling up
Bayesian Neural Networks for high-dimensional data to capture epistemic
uncertainty. SWAG [45] fits a Gaussian distribution capturing the SWA [28]
mean and a covariance matrix representing the first two moments of SGD
iterated. This distribution is then used as a posterior over the neural
network weights. [17] parametrize the BNN with a distribution on a rank-1
subspace for each weight matrix, inspired by BatchEnsembles [78]. [74, 48]
propose approaches to improve the efficiency of ensembles by distilling the
distribution of predictions rather than the average, thus preserving the
information about the uncertainty captured by the ensemble.

There is also a large body of work on methods for approximating samples from
the Bayesian posterior on large datasets with efficient MCMC based approaches.
[77, 83, 73]. The variance of this posterior distribution can then be computed
and used as an uncertainty estimate. However, as we have discussed in the
sections above, this does not account for model misspecification and thus is
not an accurate estimate for the lack of knowledge of the predictor.

[38] present several decompositions of the total expected loss, including the
decomposition into the epistemic and irreducible (aleatoric) loss. They
present additive adjustments that reduce the scoring rules like the log-loss
and Brier score, but do not tackle the general problem of uncertainty
estimation.

There are also interesting connections between the problem of out-of-
distribution generalization arising in sequential model optimization and
Bayesian optimization, discussed here, and the possibility of reweighing
examples, see [20].

##  Appendix B Epistemic Uncertainty in a general loss function setting

We consider the setting presented in section 1, but with a general loss
function $l$. We use the same notations as in section 1.

###### Definition B.1

The total uncertainty of $f$ at $x$ is defined as:

$\displaystyle{\cal U}(f,x)=\int l(f(x),y)dP(y|x).$ (4)

###### Definition B.2

For a learning algorithm $\cal L$ which produces a distribution $P_{\cal
L}(f(x)|{\cal D}_{n})$ over possible solutions $f(x)$ at $x$, the model
variance at $x$ is defined as

$\displaystyle V({\cal L},{\cal D}_{n},x)$ $\displaystyle=\int
l(f(x),\hat{f}(x))dP_{\cal L}(f(x)|{\cal D}_{n}))$ (5)

with $\hat{f}(x)=\arg\min_{\bar{f}(x)}\int l(f(x),\bar{f}(x))dP_{\cal
L}(f(x)|{\cal D}_{n})).$ Note that for a loss function that is different from
the square loss, the semantics of variance (such as its non-negativity) might
be lost.

Let us consider the special cases of the negative log-likelihood loss in
general (for outputs which may be discrete or continuous) and that of the
squared error loss (which ends up being a special case of the former for
normally distributed outputs). Below we see $Q(Y|x)$ as a probability mass or
density function (over $y$), which is also a function of $x$.

###### Definition B.3

The negative log-likelihood (NLL) loss takes as first argument $Q(Y|x)$ a
probability mass or density function and returns

$\displaystyle l_{NLL}(Q(Y|x),y)=-\log Q(Y=y|x).$ (6)

###### Proposition 2

For the NLL loss with ground truth $P(Y|x)$ and predictor $Q(Y|x)$, the total
uncertainty ${\cal U}(Q(Y|x),x)$ is a cross-entropy, i.e.,

$\displaystyle{\cal U}(Q(Y|\ .\ ),x)$ $\displaystyle=CE(P(Y|x)||Q(Y|x))$
$\displaystyle=-\int dP(y|x)\log Q(y|x)$ (7)

The proposition is shown by applying the definitions.

###### Proposition 3

For the NLL loss with ground truth $P(Y|x)$, the aleatoric uncertainty ${\cal
A}(x)$ in this setting is the entropy $H[P(Y|x)]$ of the ground truth
conditional:

$\displaystyle{\cal A}(x)=-\int dP(y|x)\log P(y|x)=H[P(Y|x)],$ (8)

The proposition is shown from the cross-entropy $CE(P(Y|x)||Q(Y|x))$ being
minimized when $Q=P$.

###### Proposition 4

For the NLL loss with ground truth $P(Y|x)$ and predictor $Q(Y|x)$, the
epistemic uncertainty ${\cal E}(Q(Y|x),x)$ is the Kullback-Liebler divergence
between $P$ and $Q$ (with $P$ as the reference):

$\displaystyle{\cal E}(Q(Y|\ .\ ),x)$ $\displaystyle=KL(P(Y|x)||Q(Y|x))$
$\displaystyle=\int dP(y|x)\log\frac{P(y|x)}{Q(y|x)}$ (9)

The proposition is shown by combining the above two propositions and the
definition of epistemic uncertainty.

To move towards the MSE loss, consider the special case of NLL with a
conditionally Normal output density for both $P$ and $Q$.

###### Proposition 5

For the NLL loss with a conditionally Normal output density for both $P$ and
$Q$, with respective means $f^{*}(x)$ and $\hat{f}(x)$ and respective
variances $\sigma^{2}_{P}(x)$ and $\sigma^{2}_{Q}(x)$, the epistemic
uncertainty is

$\displaystyle{\cal E}(Q(Y|\ .\
),x)=\frac{1}{2\sigma^{2}_{Q}(x)}l_{MSE}(\hat{f}(x),f^{*}(x))+KL(P(Y|x)||\tilde{Q}(Y|x)),$
(10)

where $\tilde{Q}(\ .\ |x)$ is obtained by shifting $Q(\ .\ |x)$ towards $P(\
.\ |x)$ (i.e., $\tilde{Q}(\ .\ |x)$ is Gaussian with mean $f^{*}(x)$ and
variance $\sigma^{2}_{Q}(x)$ ), and the Bayes-optimal mean predictor is
$f^{*}(x)=E_{P}[Y|x]$. Note that if $\sigma_{P}=\sigma_{Q}$, then the KL term
is zero.

The proof is presented in Appendix C. We can compare with the MSE loss (which
assumes a constant variance $\sigma=\sigma_{P}=\sigma_{Q}$) and obtain the
same result up to variance scaling.

##  Appendix C Proofs

###  C.1 Proposition 1

It is a well known result that, because $f^{*}(x)$ is the mean of $P(.|x)$, it
is also the minimizer of $\hat{y}\mapsto\int(\hat{y}-y)^{2}dP(y|x)$. $f^{*}$
is thus a Bayes Optimal predictor.

By definition of the total uncertainty:

$\displaystyle{\cal U}(f,x)=\int(f(x)-y)^{2}dP(y|x)=E[(f(x)-Y)^{2}].$

Hence, by definition of aleatoric uncertainty:

$\displaystyle{\cal A}(x)={\cal U}(f^{*},x)=E[(f^{*}(x)-Y)^{2}].$

and by definition of epistemic uncertainty

$\displaystyle{\cal E}(f,x)$
$\displaystyle=E\left[(f(x)-y)^{2}-(f^{*}(x)-y)^{2}\right]$
$\displaystyle=f(x)^{2}-f^{*}(x)^{2}-2(f(x)-f^{*}(x))f^{*}(x)$
$\displaystyle=(f(x)-f^{*}(x))^{2}.$

Which concludes the proof.

###  C.2 Proposition 5

From Equation 9, we get:

$\displaystyle{\cal E}(Q(Y|\ .\ ),x)$ $\displaystyle=KL(P(Y|x)||Q(Y|x))$
$\displaystyle=\log\frac{\sigma_{Q}(x)}{\sigma_{P}(x)}+\frac{\sigma_{P}^{2}(x)+(f(x)-f^{*}(x))^{2}}{2\sigma^{2}_{Q}(x)}-\frac{1}{2}$
$\displaystyle=\frac{1}{2\sigma^{2}_{Q}(x)}l_{MSE}(f(x),f^{*}(x))+\log\frac{\sigma_{Q}(x)}{\sigma_{P}(x)}+\frac{\sigma_{P}^{2}(x)+(f^{*}(x)-f^{*}(x))^{2}}{2\sigma^{2}_{Q}(x)}-\frac{1}{2}$
$\displaystyle=\frac{1}{2\sigma^{2}_{Q}(x)}l_{MSE}(f(x),f^{*}(x))+KL(P(Y|x)|\tilde{Q}(Y|x))$

Which concludes the proof

##  Appendix D Pseudo Code for the fixed training set setting

Algorithm 3 illustrates the training procedure when a held-out validation set
is available. We focus on $y\in\mathbb{R}$ in this paper because it makes
sense for active learning applied to black-box optimization (where we want to
maximize it) but the algorithms can trivially be applied to the case where
$y\in\mathbb{R}^{d}$ for any $d$. Similarly, the algorithms can be generalized
to other losses besides the square loss, following the generalized theory
presented above in Appendix B.

Data: $\mathcal{D}$ the training dataset with pairs $(x,y)$ with $x\in\cal X$,
$y\in\mathbb{R}$; $\mathcal{D}_{out}$ the out-of-sample dataset with pairs
$(x,y)$, to train the uncertainty estimator
$\mathcal{X}$, the input/search space
$a:{\cal X}\mapsto\mathbb{R}$, trained estimator of aleatoric uncertainty (
$f:{\cal X}\mapsto\mathbb{R}$, main predictor, trained on $\mathcal{D}$
$u:{\cal X}\mapsto\mathbb{R}$, total uncertainty estimator (estimates error of
$f$)

Training:

Initialize empty dataset of errors $\mathcal{D}_{u}$

Optional: Pre-fill $\mathcal{D}_{u}$ using Algorithm 2 and fit $u$ on
$\mathcal{D}_{u}$

$x_{acq}\leftarrow\emptyset,y_{acq}\leftarrow\emptyset$

for  _every pair $(x,y)$ in $\mathcal{D}\cup\mathcal{D}_{out}$_ do

$\mathcal{D}_{u}\leftarrow\mathcal{D}_{u}\cup\\{(x,(y-f(x))^{2})\\}$

end for

Fit $u$ on $\mathcal{D}_{u}$ Evaluation: For every input $x$, return
$u(x)-a(x)$ as an estimator of epistemic uncertainty at $x$

Algorithm 3 DEUP with a fixed training set: Training procedure to obtain
estimates of epistemic uncertainty

##  Appendix E Estimating Aleatoric Uncertainty with access to an oracle

In scenarios like active learning, one has access to an oracle from which we
can obtain samples of $Y\sim P(Y|x)$ at any given point $x$. In that case, one
can train an estimator $a(x)$ of aleatoric uncertainty by obtaining $K>1$
samples $y_{1},\dots y_{K}$ at the same $x$, for a set of representative
$x$’s.

More formally, if we have multiple independent outcomes $y_{1},\dots,y_{K}\sim
P(Y|x)$ for each input point $x$, then training a predictor $a$ with the
squared loss on (input, target) examples
$\left(x,\frac{K}{K-1}\overline{Var}(y_{1},\dots,y_{K})\right)$, where
$\overline{Var}$ denotes the empirical variance, yields an estimator of the
aleatoric uncertainty.

Naturally, this estimator is asymptotically unbiased, if the learning
algorithm ensures asymptotic convergence to a Bayes-Optimal predictor.

This is due to the fact that
$E_{y_{1},\dots,y_{K}}\left[\frac{K}{K-1}\overline{Var}(y_{1},\dots,y_{K})\right]=Var_{P(y|x)}[y|x]$,
which according to Proposition 1 is equal to $\mathcal{A}(x)$.

##  Appendix F Sequential Model Optimization Experiments

We use BoTorch222https://botorch.org/ [5] as the base framework for our
experiments.

For all our Sequential Optimization algorithms, we use Algorithm 1 to train
DEUP uncertainty estimators. We found that the optional step of pre-filling
the uncertainty estimator dataset $\mathcal{D}_{e}$ was important given the
low number of available training points. We used half the initial training set
(randomly chosen) as in-sample examples (used to train the main predictor and
an extra-feature generator) and the other half as out-of-sample examples to
provide instances of high epistemic uncertainty to train an uncertainty
predictor; we repeated the procedure by alternating the roles of the two
halves of the dataset. We repeated the whole procedure twice using a new
random split of the dataset, thus ending up with $4$ training points in
$\mathcal{D}_{e}$ for every initial training point in $\mathcal{D}_{init}$.

The error predictor is trained with the $\log$ targets (i.e. log MSE between
predicted and observed error). This helps since the scale of the errors varies
over multiple orders of magnitude.

Computationally, the training time of DEUP-EI depends on various choices (e.g.
the features used to train the epistemic uncertainty predictor, the dimension
of the input, the learning algorithms, etc..). Additionally, the training time
for the uncertainty predictor varies at each step of the optimization. In
total, the sequential optimization experiments took about 1 CPU day.

###  F.1 One-dimensional function toy example

In Figure 6, we show the results of DEUP-EI, compared to GP-EI, MCDropout-EI
and Ensembles-EI, in the task of optimizing a synthetic one-dimensional
function. Because MCDropout and Ensembles are trained on in-sample data only,
they are unable to generalize their uncertainty estimates, which makes them
bad candidates for Sequential Model Optimization, because they are easily
stuck in local minima, and require many iterations before the acquisition
function gives more weight to the predicted uncertainties than the current
maximum.

For Random acquisition, we sampled for different seeds $56$ points, and used
the (average across the seeds of the) maximum of the first $6$ values as the
first value in the plots (Figures 3 and 6). Note that because the function is
specifically designed to have multiple local maxima, GP-EI also required more
optimization steps, and actually performed worse than random acquistion

As a stationarizing input feature, we used the variance of a GP fit on the
available data at every step. We found that the binary (in-sample/out-of-
sample) feature and density estimates were redundant with the variance feature
and didn’t improve the performance as captured by the number of additional
function calls. We used a GP for the DEUP uncertainty estimator. Using a
neural net provided similar results, but was computationally more expensive in
this 1-D case with few datapoints. We used a 3-hidden layer neural network,
with 128 neurons per layer and a ReLU activation function, with Adam [33] and
a learning rate of $10^{-3}$ (and default values for the other
hyperparameters) to train the main predictor for DEUP-EI (in order to fit the
available data). The same network architecture and learning rate were used for
the Dropout and Ensemble baselines. We used 3 networks for the Ensemble
baseline, and a dropout probability of 0.3 for the Dropout baseline, with 100
test-time forward passes to compute uncertainty estimates.

Figure 6: Left. Synthetic function to optimize. Right. Maximum value reached
by the different methods on the synthetic function. The shaded areas represent
the standard error across 5 different runs, with different initial sets of 6
pairs. For clarity, the shaded areas are omitted for the two worst performing
methods. In each run, all the methods start with the same initial set of 6
points. GP-EI tends to get stuck in local optima and requires more than 50
steps, on average, to reach the global maximum.

###  F.2 Two-dimensional function

To showcase DEUP’s usefulness for Sequential Model Optimization in with a
number of dimensions greater than 1, we consider the optimization of the Levi
N.13 function, a known benchmark for optimization. The function $f$ takes a
point $(x,y)$ in 2D space and returns:

$\displaystyle f(x,y)=-\left(\sin^{2}(3\pi x)+(x-1)^{2}(1+\sin^{2}(3\pi
y))+(y-1)^{2}(1+\sin^{2}(2\pi y))\right)$

We use the box $[-10,10]^{2}$ as the optimization domain. In this domain, the
maximum of the function is $0$, and it is reached at $(1,1)$. The function has
multiple local maxima, as shown in Figure 7(a)333Plot of the function copied
from https://www.sfu.ca/ ssurjano/levy13.html.

Similar to the previous one-dimensional function, MCDropout and Ensemble
provided bad performances and are omitted from the plot in 7(b). We used the
same setting and hyperparameters for DEUP as for the previous function. DEUP-
EI is again the only method that reaches the global maximum consistently in
under $56$ function evaluations.

(a) Visualization of $(x,y)\mapsto-f(x,y)$

(b) Comparisons with GP-EI and Random acquisition

Figure 7: Sequential Model Optimization on the Levi N.13 function

###  F.3 Additional details for the Ackley function experiment, for synthetic
data in higher dimensions

The Ackley function of dimension $d$ is defined as:

$\displaystyle Ackley_{d}:{\mathcal{B}}$
$\displaystyle\rightarrow{\mathbb{R}}$ $\displaystyle x$ $\displaystyle\mapsto
A\exp\left(-B\sqrt{\frac{1}{d}\sum_{i=1}^{d}x_{i}^{2}}\right)+\exp\left(\frac{1}{d}\sum_{i=1}^{d}cos(cx_{i})\right)-A-\exp(1)$

where ${\mathcal{B}}$ is a hyperrectangle of ${\mathbb{R}}^{d}$. $(0,\dots,0)$
is the only global optimizer of $Ackley_{d}$, at which the function is equal
to $0$. We use BoTorch’s default values for $A,B,c$, which are $20,0.2,2\pi$
respectively.

In our experiments, we used ${\mathcal{B}}=[-10,15]^{d}$ for all dimensions
$d$.

For the TurBO baseline, we use BoTorch’s default implementation, with Expected
Improvement as an acquisition function, and a batch size of $1$ (i.e.
acquiring one point per step).

For fair comparisons, for DEUP, we use a Gaussian Process as the main model,
and use its variance as the only input of the epistemic uncertainty predictor.
This means that we calibrate the GP variance to match the out-of-sample
squared error, using another GP to perform the regression. TurBO-DEUP is a
combination of both, in which we perform the variance calibration task for the
local GP models of TurBO. The uncertainty predictor, i.e. the GP regressor, is
trained with $\log$ targets, as in Appendix F.1, but also with $\log$
variances as inputs.

Note that only the stationarizing feature is used as input for the uncertainty
predictor. When we used the input $x$ as well, we found that the GP error
predictor overfits on the $x$ part of the input $(x,v)$, and it was
detrimental to the final performances. For all experiments, we used $20$
initial points.

##  Appendix G Reinforcement Learning Experiments

For RL experiments, we used bsuite [60], a collection of carefully designed RL
environments. bsuite also comes with a list of metrics which aim to evaluate
RL agents from different aspects. We compare the agents based on the basic
metric and average regret as they capture both sample complexity and final
performance. The default DQN agent is used as the base of our experiments with
a 3 layer fully-connected (FC) neural network as its Q-network. For the
Bootstrapped DQN baseline, we used the default implementation provided by
bsuite. To implement DQN + MC-Dropout, following the implementation from [22],
two dropout layers with dropout probability of 0.1 are used before the second
and the third FC layers. In order take an action, the agent performs a single
stochastic forward pass through the Q-network, which is equivalent to taking a
sample from the posterior over the Q-values, as done in Thompson sampling, an
alternative to $\epsilon-$greedy exploration.

As a density estimator, we used a Kernel Density Estimator (KDE) with a
Gaussian kernel and bandwidth of 1 to map states to densities. This KDE is fit
after each 10000 steps (actions) with a batch of samples from the replay
buffer (which is of size 10000). The uncertainty estimator network (E-network)
has the same number of layers as the Q-network, with an additional Softplus
layer at the end. All other hyperparameters are the same as the default
implementation by [60]. One complete training run for the DEUP-DQN with 5
seeds experiments takes about 0.04-0.05 GPU days on a V100 GPU. In total RL
experiments took about 0.15 GPU days on a Nvidia V100 GPU.

Initialize replay buffer $\mathcal{D}$ with capacity $\mathcal{N}$
$Q_{\theta}(s,a)$: state-action value predictor
$E_{\phi}(\log d)$: uncertainty estimator network, which takes the log-density
of the states as the input
$d(s)$: Kernel density estimator (KDE)
K: KDE fitting frequency
W: Number of warm-up episodes
for  _episode=1 to $M$_ do

set $s_{0}$ as the initial state
for  _t=1 to max-steps-per-episode_ do

with probability $\epsilon$:  take a random action, otherwise:
if $episode\leq$ W: $a=max_{a}Q_{\theta}(s_{t},a)$, else:
$a=max_{a}\big{[}Q_{\theta}(s_{t},a)+\kappa\times E_{\phi}(\log
d(s_{t}))(a)\big{]}$
store $(s_{t},a_{t},r_{t},s_{t+1})$ in $\mathcal{D}$
Sample random minibatch B of transitions $(s_{j},a_{j},r_{j},s_{j+1})$ from
$\mathcal{D}$
if $s_{j}$ is a final state: $y_{j}=r_{j}$, else: $y_{j}=r_{j}+\gamma
max_{a}Q(s_{t},a)$
Update Q-network:
$\theta\leftarrow\theta+\alpha_{Q}.\nabla_{\theta}\mathop{\mathbb{E}_{(s,a)\sim{B}}}\Big{[}\big{(}y_{j}-Q_{\theta}(s,a)\big{)}^{2}\Big{]}$
Update E-network:
$\phi\leftarrow\phi+\alpha_{E}.\nabla_{\phi}\mathop{\mathbb{E}_{(s,a)\sim{B}}}\Bigg{[}\Big{[}\big{(}y_{j}-Q_{\theta}(s,a)\big{)}^{2}-E_{\phi}(\log
d(s_{t}))(a)\Big{]}^{2}\Bigg{]}$
if mod(total-steps, K) = 0: fit the KDE $d$ on the states of $\mathcal{D}$

end for

end for

Algorithm 4 DEUP-DQN

##  Appendix H Rejecting Difficult Examples

We adapt the standard OOD rejection task [76, 43] to measure the Spearman Rank
Correlation of the predicted uncertainty with the true generalization error,
in addition to the OOD Detection AUROC. We use MC-Dropout [22], Deep Ensemble
[40], DUE[75] and DUQ [76] as the baselines 444MC-Dropout and Deep Ensemble
baselines are based on https://github.com/google/uncertainty-baselines, DUQ
based on https://github.com/y0ast/deterministic-uncertainty-quantification and
DUE based on https://github.com/y0ast/DUE. We use these baselines as
representatives for the major approaches for uncertainty estimation in recent
literature. For all the methods, including DEUP we consider two architectures
for the main predictor, ResNet-18 and ResNet-50 [24] (Table 2) to study the
effect of model capacity. Note that for the ResNet50 DEUP model we continue
using the ResNet-18 based DUE as variance source.

Table 2: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with ResNet-50 models (3 seeds) trained on CIFAR-10. Model | ResNet-50
---|---
MC-Dropout | $0.312\pm 0.003$
Deep Ensemble | $0.401\pm 0.004$
DUQ | $0.399\pm 0.003$
DEUP (D+V) | $\bm{0.465\pm 0.002}$

##### Training

The baselines were trained with the CIFAR-10 training set with $10\%$ set
aside as a validation set for hyperparameter tuning. The hyperparameters are
presented in Table 3 and Table 4. The hyperparameters not specified are set to
the default values. For DEUP, we consider the log-density, model-variance
estimate and the seen-unseen bit as the features for the error predictor. The
density estimator we use is Masked-Autoregressive Flows [62] and the variance
estimator used is DUE [75]. Note that as indicated earlier $x$, the input
image, is not used as a feature for the error predictor. We present those
ablations in the next sub-section. For training DEUP, the CIFAR-10 training
set is divided into 5 folds, with each fold containing 8 unique classes. For
each fold, we train an instance of the main predictor, density estimator and
model variance estimator on only the corresponding 8 classes. The remaining 2
classes act as the out-of-distribution examples for training the error
predictor. Using these folds we construct a dataset for training the error
predictor, a simple feed forward network. The error predictor is trained with
the $\log$ targets (i.e. log MSE between predicted and observed error). This
helps since the scale of the errors varies over multiple orders of magnitude.
We then train the main predictor, density estimator and the variance estimator
on the entire CIFAR-10 dataset, for evaluation. The hyperparameters are
presented in Table 4. For all models, we train the main predictor for $75$ and
$125$ epochs for ResNet-18 and ResNet-50 respectively. We use SGD with
Momentum (set to $0.9$), with a multi-step learning schedule with a decay of
$0.2$ at epochs $[25,50]$ and $[45,90]$ for ResNet-18 and ResNet-50
respectively. One complete training run for DEUP takes about 1.5-2 GPU days on
a V100 GPU. In total these set of experiments took about 31 GPU days on a
Nvidia V100 GPU.

Table 3: Left: Hyperparameters for training Deep Ensemble [40]. Right:
Hyperparameters for training MC-Dropout [22].

Parameters | Model
---|---
ResNet-18 | ResNet-50
Number of members | 5 | 5
Learning Rate | 0.05 | 0.01

Parameters | Model
---|---
ResNet-18 | ResNet-50
Number of samples | 50 | 50
Dropout Rate | 0.15 | 0.1
L2 Regularization Coefficient | 6e-5 | 8e-4
Learning Rate | 0.05 | 0.01

Table 4: Left: Hyperparameters for training DUQ [76]. Right: Hyperparameters
for training DUE [75].

Parameters | Model
---|---
ResNet-18 | ResNet-50
Gradient Penalty | 0.5 | 0.65
Centroid Size | 512 | 512
Length scale | 0.1 | 0.2
Learning Rate | 0.05 | 0.025

Parameters | Model
---|---
ResNet-18
Inducing Points | 50
Kernel | RBF
Lipschitz Coefficient | 2
BatchNorm Momentum | 0.99
Learning Rate | 0.05
Weight Decay | 0.0005

Table 5: Hyperparameters for training DEUP. Parameters | Model
---|---
ResNet-18 | ResNet-50
Uncertainty Predictor Architecture | [1024] x 5 | [1024] x 5
Uncertainty Predictor Epochs | 100 | 100
Uncertainty Predictor LR | 0.01 | 0.01
Main Predictor Learning Rate | 0.05 | 0.01

##### Ablations

We also perform some ablation experiments to study the effect of each feature
for the error predictor. The Spearman rank correlation coefficient between the
generalization error and the variance feature, $V$, from DUE [75] alone is
37.84 $\pm$ 0.04, and the log-density, $D$, from MAF [62] alone is 30.52 $\pm$
0.03. With only the image ($x$) the SRCC is $36.58\pm 0.16$

Table 6 presents the results for these experiments. We observe that combining
all the features performs the best. Also note that using the log-density and
variance as features to the error predictor we observe better performance than
using them directly, indicating that the error predictor perhaps captures a
better target for the epistemic uncertainty. The boolean feature ($B$)
indicating seen examples, discussed in Section 2.2, also leads to noticeable
improvments.

Table 6: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with variants of DEUP with different features as input for the uncertainty predictor. $D$ indicates the log-density from MAF [62], $V$ indicates variance from DUQ [76] and $B$ indicates a bit indicating if the data is seen.  Features | Model
---|---
ResNet-18 | ResNet-50
$D$+$V$+$B$ | 0.426 $\pm$ 0.009 | 0.465 $\pm$ 0.002
$D$+$V$ | 0.419 $\pm$ 0.003 | 0.447 $\pm$ 0.003
$V$+$B$ | 0.401 $\pm$ 0.004 | 0.419 $\pm$ 0.004
$D$+$B$ | 0.403 $\pm$ 0.003 | 0.421 $\pm$ 0.002

###  H.1 Predicting Uncertainty under Distribution Shift

We also consider the task of uncertainty estimation in the setting of shifted
distributions [61, 25]. We evaluate the uncertainty predictions of models
trained with CIFAR-10, on CIFAR-10-C [25] which consists of images from
CIFAR-10 distorted using 16 corruptions like gassian blur, impulse noise,
among others. Figure 8 shows that even in the shifted distribution setting,
the uncertainty estimates of DEUP correlate much better with the error made by
the predictor, compared to the baselines.

Figure 8: Spearman Rank Correlation Coefficient between the predicted
uncertainty and true error for models trained with CIFAR-10, and evaluated on
CIFAR-10-C. DEUP outperforms the baselines on all types of corruptions.

##  Appendix I Drug Combination Experiments

To validate DEUP’s uncertainty estimates in a real-world setting, we measured
its performance on a regression task predicting the synergy of drug
combinations. While much effort in drug discovery is spent on finding novel
small molecules, a potentially cheaper method is identifying combinations of
pre-existing drugs which are synergistic (i.e., work well together). Indeed,
drug combinations are the current standard-of-care for a number of diseases
including HIV, tuberculosis, and some cancers [15, 58, 53].

However, due to the combinatorial nature of drug combinations, identifying
pairs exhibiting synergism is challenging. Compounding this problem is the
high monetary cost of running experiments on promising drug combinations, as
well as the length of time the experiments take to complete. Uncertainty
models could be used by practitioners downstream to help accelerate drug
combination treatment discoveries and reduce involved development costs.

To test DEUP’s performance on this task we used the DrugComb and LINCS L1000
datasets [82, 69]. DrugComb is a dataset consisting of pairwise combinations
of anti-cancer compounds tested on various cancer cell lines. For each
combination, the dataset provides access to several synergy scores, each
indicating whether the two drugs have a synergistic or antagonistic effect on
cancerous cell death. LINCS L1000 contains differential gene expression
profiles for various cell lines and drugs. Differential gene expressions
measure the difference in the amount of mRNA related to a set of influential
genes before and after the application of a drug. Because of this, gene
expressions are a powerful indicator of the effect of a single drug at the
cellular level.

In our experiments, each drug is represented by its Morgan fingerprint
[54]555The Morgan fingerprint represents a molecule by associating with it a
boolean vector specifying its chemical structure. Morgan fingerprints have
been used as a signal of various molecular characteristics to great success
[6, 84]. (with 1,024 bits and a radius of 3) as well as two differential gene
expression profiles (each of dimension 978) from two cell lines (PC-3 and
MCF-7). In order to use gene expression features for every drug, we only used
drug pairs in DrugComb where both drugs had differential gene expression data
for cell lines PC-3 and MCF-7.

We first compared the quality of DEUP’s uncertainty estimations to other
uncertainty estimation methods on the task of predicting the combination
sensitivity score [49] for drug pairs tested on the cell line PC-3 (1,385
examples). We evaluated the uncertainty methods using a train, validation,
test split of $40\%$, $30\%$, and $30\%$, respectively. The underlying model
used by each uncertainty estimation method consisted of a single drug fully
connected neural network (2 layers with 2048 hidden units and output of
dimension 1024) and a combined drug fully connected neural network (2 layers,
with 128 hidden units). The embeddings of an input drug pair’s drugs produced
by the single drug network are summed and passed to the combined drug network,
which then predicts final synergy. By summing the embeddings produced by the
single drug network, we ensure that the model is invariant to permutations in
order of the two drugs in the pair. The models were trained with Adam [33],
using a learning rate of $1\text{e-}4$ and weight decay of $1\text{e-}5$. For
MC-Dropout we used a dropout probability of $0.1$ on the two layers of the
combined drug network and 3 test-time forward passes to compute uncertainty
estimates. The ensemble used 3 constituent models for its uncertainty
estimates. Both Ensemble and MC-Dropout models were trained with the MSE loss.

We also compared against DUE [75] which combines a neural network feature
extractor with an approximate Gaussian process. Spectral normalization was
added to all the layers of the combined drug network and of the single drug
network. Let $d_{\text{emb}}$ denote the dimension of the output of the
combined drug network, which is also the input dimension of the approximate
Gaussian process. We conducted a grid-search over different values of
$d_{\text{emb}}$ (from 2 to 100), the number of inducing points (from 3 to
200), the learning rate, and the kernel used by the Gaussian process. The
highest correlation of uncertainty estimates with residuals was attained with
$d_{\text{emb}}=10$, 100 inducing points, a learning rate of $1\text{e-}2$,
and the Matern12 kernel.

Data: $\mathcal{D}$ dataset of pairwise drug combinations, along with synergy
scores $((d_{1},d_{2}),y)$

Initialization:

Split training set into two halves, in-sample $\mathcal{D}_{in}$ and out-of-
sample $\mathcal{D}_{out}$

$f_{\mu}(d_{1},d_{2})$: $\hat{\mu}$ predictor which takes a pair of drugs as
input

$f_{\sigma}^{in}(d_{1},d_{2})$: In-sample $\hat{\sigma}_{in}$ error predictor

$f_{\sigma}^{out}(d_{1},d_{2})$: Out-of-sample $\hat{\sigma}_{out}$ error
predictor

Training:

while  _training not finished_ do

In-sample update Get an in-sample batch
$(d_{1,in},d_{2,in},y_{in})\sim\mathcal{D}_{in}$ Predict
$\hat{\mu}=f_{\mu}(d_{1,in},d_{2,in})$ and in-sample error
$\hat{\sigma}_{in}=f_{\sigma}^{in}(d_{1,in},d_{2,in})$ Compute NLL:
$\frac{\log(\hat{\sigma}_{in}^{2})}{2}+\frac{(\hat{\mu}-y_{in})^{2}}{2\hat{\sigma}_{in}^{2}}$
Backpropagate through $f_{\mu}$ and $f_{\sigma}^{in}$ and update. Out-of-
sample update Get an out-of-sample batch
$(d_{1,out},d_{2,out},y_{out})\sim\mathcal{D}_{out}$ Estimate
$\hat{\mu}=f_{\mu}(d_{1,out},d_{2,out})$ and out-of-sample error
$\hat{\sigma}_{out}=f_{\sigma}^{out}(d_{1,out},d_{2,out})$ Compute NLL:
$\frac{\log(\hat{\sigma}_{out}^{2})}{2}+\frac{(\hat{\mu}-y_{out})^{2}}{2\hat{\sigma}_{out}^{2}}$
Backpropagate through $f_{\sigma}^{out}$ and update.

end while

Algorithm 5 DEUP for Drug Combinations

The DEUP model we used outputs two heads
$\bigl{[}\begin{smallmatrix}\hat{\mu}\\\
\hat{\sigma}\end{smallmatrix}\bigr{]}$ and is trained with the NLL
$\frac{\log(\hat{\sigma}^{2})}{2}+\frac{(\hat{\mu}-y)^{2}}{2\hat{\sigma}^{2}}$
in a similar fashion as in [40]. To obtain a predictor of the out-of-sample
error, we altered our optimization procedure so that the $\mu$ and $\sigma$
heads were not backpropagated through at all times. Specifically, we first
split the training set into two halves, terming the former the in-sample set
$\mathcal{D}_{in}$ and the latter the out-of-sample set $\mathcal{D}_{out}$.
We denote as $f_{\sigma}^{in}$ the in-sample error predictor and
$f_{\sigma}^{out}$ the out-of-sample error predictor. $f_{\sigma}^{out}$ is
used to estimate total uncertainty. Note that in this setting,
$f_{\sigma}^{out}$ predicts the square root of the epistemic uncertainty
($\hat{\sigma}_{out}$) rather than the epistemic uncertainty itself
($\hat{\sigma}_{out}^{2}$).

In our experiments, an extra bit is added as input to the model in order to
indicate whether a given batch is from $\mathcal{D}_{in}$ or
$\mathcal{D}_{out}$. Through this, the same model is used to estimate
$f_{\sigma}^{in}$ and $f_{\sigma}^{out}$ with the model estimating
$f_{\sigma}^{in}$ when the bit indicates an example is drawn from
$\mathcal{D}_{in}$ and $f_{\sigma}^{out}$ otherwise. When the batch is drawn
from $\mathcal{D}_{in}$, both heads are trained using NLL using a single
forward pass. However, when the data is drawn from $\mathcal{D}_{out}$ only
the $\hat{\sigma}$ head is trained. To do this, we must still predict
$\hat{\mu}$ in order to compute the NLL. But the $\hat{\mu}$ predictor
$f_{\mu}$ must be agnostic to the difference between $\mathcal{D}_{in}$ and
$\mathcal{D}_{out}$. To solve this, we perform two separate forward passes.
The first pass computes $\hat{\mu}$ and sets the indicator bit to 0 so
$f_{\mu}$ has no notion of $\mathcal{D}_{out}$, while the second pass computes
$\hat{\sigma}$, setting the bit to 1 to indicate the true source of the batch.
Finally, we backpropagate through the $\hat{\sigma}$ head only. The training
procedure is described in Algorithm 5

We report several measures for the quality of uncertainty predictions on a
separate test set in Table 7.

Model Corr. w. res. U. Bound Ratio Log Likelihood Coverage Probability CI
width MC-Dropout $0.14\pm 0.07$ $0.56\pm 0.05$ $0.25\pm 0.12$ $-20.1\pm 6.8$
$11.4\pm 0.2$ $3.1\pm 0.1$ Deep Ensemble $0.30\pm 0.09$ $0.59\pm 0.04$
$0.50\pm 0.13$ $-14.3\pm 4.7$ $10.8\pm 1.4$ $3.4\pm 0.6$ DUE $0.12\pm 0.12$
$0.15\pm 0.03$ $\bm{0.80\pm 0.79}$ $-13.0\pm 0.52$ $15.2\pm 1.0$ $3.5\pm 0.1$
DEUP $\bm{0.47\pm 0.03}$ $0.63\pm 0.05$ $\bm{0.75\pm 0.07}$ $\bm{-3.5\pm
0.25}$ $\bm{36.1\pm 2.5}$ $\bm{13.1\pm 0.9}$

Table 7: Drug combinations: quality of uncertainty estimates from different
methods. Corr. w. res. shows correlation between model residuals and predicted
uncertainties $\hat{\sigma}$. A best-case Upper Bound on Corr. w. res. is
obtained from the correlation between $\hat{\sigma}$ and true samples from
$\mathcal{N}(0,\hat{\sigma})$. Ratio is the ratio between col. 1 and 2 (larger
is better). Log-likelihood: average over 3 seeds of per sample predictive log-
likelihood. Coverage Probability: Percentage of test samples which are covered
by the 68% confidence interval. CI width: width of the 86% confidence
interval.

For each model, we report the per sample predictive log-likelihood, coverage
probability and confidence interval width, averaged over 3 seeds.

We also computed the correlation between the residuals of the model
$|\hat{\mu}(x_{i})-y_{i}|$ and the predicted uncertainties
$\hat{\sigma}(x_{i})$. We noted that the different uncertainty estimation
methods lead to different distributions $p(\hat{\sigma}(x))$. For example,
predicted uncertainties obtained with DUE always have a similar magnitude. By
contrast, DEUP yields a wide range of different predicted uncertainties.

These differences between the distributions $p(\hat{\sigma}(x))$ obtained with
the different methods may have an impact on the correlation metric, possibly
biasing the comparison of the different methods. In order to account for
differences in the distribution $p(\hat{\sigma}(x))$ across methods, we report
another metric which is the ratio between the observed correlation
$Corr(|\hat{\mu}(x)-y|,\hat{\sigma}(x))$ and the maximum achievable
correlation given a specific distribution $p(\hat{\sigma}(x))$.

This maximum achievable correlation (refered to as the upper bound) is not per
se a comparison metric, and is estimated (given a specific
$p(\hat{\sigma}(x))$) as follows: we assume that, for each example
$(x_{i},y_{i})$, the predictive distribution of the model
$\mathcal{N}(\hat{\mu}(x_{i}),\hat{\sigma}(x_{i}))$ corresponds exactly to the
distribution of the target, i.e.
$y_{i}\sim\mathcal{N}(\hat{\mu}(x_{i}),\hat{\sigma}(x_{i}))$. Under this
assumption, the residual of the mean predictor follows a distribution
$\mathcal{N}(0,\hat{\sigma}(x_{i}))$. We can then estimate the upper bound by
computing the correlation between the predicted uncertainties
$\hat{\sigma}(x_{i})$ and samples from the corresponding Gaussians
$\mathcal{N}(0,\hat{\sigma}(x_{i}))$. 5 samples were drawn from each Gaussian
for our evaluation. This upper bound is reported in the Table.

Finally, we reported our comparison metric: the ratio between the correlation
$Corr(|\hat{\mu}(x)-y|,\hat{\sigma}(x))$ and the upper bound. The higher the
ratio is, the closer the observed correlation is to the estimated upper bound
and the better the method is doing.

It is interesting to note that the upper bound is much lower for DUE compared
to other methods, as its predicted uncertainties lie within a short range of
values.

Predicted $\hat{\mu}$ and uncertainty estimates can be visualized in Figure 9
for different models. MC-dropout, Ensemble and DUE consistently underestimate
uncertainty, while the out-of-sample uncertainties predicted by DEUP are much
more consistent with the order of magnitude of the residuals. Moreover, we
observed that DUE predicted very similar uncertainties for all samples,
resulting in a lower upper-bound for the correlation between residuals and
predicted uncertainties compared to other methods. We observed a similar
pattern when experimenting with the other kernels available in the DUE
package, including the standard Gaussian kernel.

Finally, we note that in the context of drug combination experiments,
aleatoric uncertainty could be estimated by having access to replicates of a
given experiment (c.f. Section E), allowing us to subtract the aleatoric part
from the out-of-sample uncertainty, leaving us with the epistemic uncertainty
only.

(a) MC-dropout

(b) Ensemble

(c) DUE

(d) DEUP

Figure 9: Predicted mean and uncertainty for different models on a separate
test set. 50 examples from the test set are ordered by increasing value of
true synergy score (orange). Model predictions and uncertainties are
visualized in blue. MC-Dropout, Ensemble and DUE consistently underestimate
the uncertainty while DEUP seems to capture the right order of magnitude.
Figures made using The Uncertainty Toolbox [14].

One complete training run for the drug combination experiments takes about
0.01 GPU days on a V100 GPU. In total these set of experiments took about 0.2
GPU days on a Nvidia V100 GPU.