(LABEL:)Equation

# Bayesian Active Learning for Censored Regression

Frederik Boe Hüttel  Technical University of Denmark  Christoffer Riis
Technical University of Denmark  Filipe Rodrigues  Technical University of
Denmark  Francisco C. Pereira  Technical University of Denmark

###### Abstract

Bayesian active learning is based on information theoretical approaches that
focus on maximising the information that new observations provide to the model
parameters. This is commonly done by maximising the Bayesian Active Learning
by Disagreement (BALD) acquisitions function. However, we highlight that it is
challenging to estimate BALD when the new data points are subject to
censorship, where only clipped values of the targets are observed. To address
this, we derive the entropy and the mutual information for censored
distributions and derive the BALD objective for active learning in censored
regression ($\mathcal{C}$-BALD). We propose a novel modelling approach to
estimate the $\mathcal{C}$-BALD objective and use it for active learning in
the censored setting. Across a wide range of datasets and models, we
demonstrate that $\mathcal{C}$-BALD outperforms other Bayesian active learning
methods in censored regression.

##  1 Introduction

Figure 1: Illustration of a 1-dimensional censored dataset, in which the
dashed black line represents the underlying function that generated the data.
The (blue) circles denote non-censored observations, while the (green) crosses
represent observations that have been censored. The grey background indicates
areas where the observations are censored.

Active learning is a framework where a model learns from a small amount of
labelled data and chooses the data it wants to acquire a label for [Settles,
2009]. This acquisition of new data points is done iteratively to improve the
model’s predictive performance and reduce model uncertainty [MacKay, 1992b].
This naturally poses the challenge: which new data points can improve the
model the most? Information theoretical approaches are often the basis to
solve this challenge by reasoning about the information that new labels can
provide to the model’s parameters [MacKay, 1992a]. A common method to estimate
the information is with the _Bayesian Active Learning by Disagreement_ (BALD)
acquisition function [Houlsby et al., 2011], which estimates the mutual
information between the model parameters and the labels to identify which
labels to acquire. It has successfully been applied to various domains, such
as computer vision [Gal et al., 2017, Kirsch et al., 2019] and natural
language processing [Shen et al., 2018].

Even though existing Bayesian active learning methods have proven useful,
using them for censored regression problems remains challenging. The challenge
in censored regression is that the target variable is only partially observed
[Tobin, 1958, Cox, 1972, Powell, 1986]. To illustrate the concept, consider
the dataset shown in Figure 1. The data points in this set are generated from
a true (_latent_) function, where some targets are censored, meaning that only
clipped values of the true function are observed. The green crosses in the
figure represent censored observations whose $y$-values, as a consequence of
the censoring, are distributed below the mean of the true function. In
contrast, the blue circles in the figure are non-censored observations and
follow the true generating function.

Censored regression models try to approximate the true function using _both_
censored and non-censored observations [Hüttel et al., 2022]. Naturally,
censored data presents unique modelling challenges to determine which new data
points provide the most information, as censored observations do not provide
the same information about the true function as uncensored ones [Hollander et
al., 1985, 1987, 1990]. As a result, the BALD objective can not directly be
applied to censored regression problems because it measures the information
gained by observing the true function, where, in reality, we might observe a
clipped value.

This issue has considerable practical implications, for example, in shared
mobility services, such as shared bikes, shared electric vehicles or electric
vehicle charging, where the supply censors the observed demand [Gammelli et
al., 2020, Hüttel et al., 2023]. Modern machine learning models are becoming
more integral to managing these systems. This raises the question [Golsefidi
et al., 2023]: where should operators focus their attention to gather the most
informative data for their model in a city? The same challenges can be found
beyond the transport domain, such as in subscription-based businesses (e.g.
what customers will cancel their subscription) [Fader and Hardie, 2007,
Chandar et al., 2022, Maystre and Russo, 2022], and in health survival
applications where labels are expensive to collect (e.g. in medical imagining
or invasive diagnosing) [Nezhad et al., 2019, Lian et al., 2022].

Motivated by these challenges, we study Bayesian active learning for censored
regression problems, focusing on estimating the information new observations
provide to the model parameters. This is challenging because new labels might
be subject to censorship and, therefore, do not have the potential to provide
the same amount of information as uncensored ones. Concretely, this paper
makes the following contributions:

  1. 1.

Formulation and derivation of the mutual information between observations and
model parameters when the observations are subject to censoring.

  2. 2.

A novel acquisition function for active learning using the derived mutual
information and the entropy of censored distributions and a novel modelling
approach to estimate the entropy.

  3. 3.

Evaluation of the proposed acquisitions on synthetic and real-world datasets
compared to Bayesian active learning methods.

##  2 Background & Setting

We are interested in the supervised learning of a probabilistic regression
model, $p(y^{*}_{i}|\mathbf{x}_{i},\theta)$, where
$\mathbf{x}_{i}\in\mathcal{X}\subseteq\mathbb{R}^{d}$ for $d\geq 1$,
$y^{*}_{i}\in\mathcal{Y}^{*}\subseteq\mathbb{R}$, and $\theta$ is a set of
stochastic model parameters. We assume that we can sample a set of model
parameters, $\theta$, from the posterior distribution $p(\theta|\mathcal{D})$.
We consider the special regression case, where $y^{*}_{i}$ is subject to
censoring, meaning that for some observations in our dataset, $y^{*}_{i}$ is
unknown. Specifically, we consider _right-censored_ data, which means that
instead of observing $y^{*}$, we observe $y_{i}=\min(y^{*}_{i},z_{i})$, where
$z_{i}\in\mathcal{Z}\subseteq\mathbb{R}$. $z_{i}$ is a censoring threshold. In
addition, we also observe a censoring indicator
$\ell_{i}=\mathbbm{1}\\{y^{*}_{i}\leq z_{i}\\}$, which indicates whether
$y_{i}$ is censored or not. A censored dataset of size $n$ can thus be denoted
$\mathcal{D}=\\{\mathbf{x}_{i},y_{i},\ell_{i}\\}_{i=1}^{n}$. For readability
and to simplify the notation, we will use $p(x)$ to denote the density
function of a random variable $x$.

In the case of censored regression, the objective is to infer the true
distribution $p(y_{i}^{*}|x_{i},\theta)$ and the model parameters, $\theta$,
based on the censored dataset $\mathcal{D}$. In censored regression, one
typically assumes that the distributions of $p(y_{i}^{*}|\mathbf{x}_{i})$ and
$p(z_{i}|\mathbf{x}_{i})$ are independent given the covariates,
$\mathbf{x}_{i}$ [Tobin, 1958]. This assumption is more general than other
assumptions, such as fixed-value censoring, i.e., $z_{i}=\text{constant
}\forall i$ [Powell, 1986]. We formally state the following assumption.

Assumption 1. (Independent censoring) We assume that conditioned on the
covariates, $\mathbf{x}_{i}$, the censoring distribution and the true
distribution of the target are independent. That is, we assume that
$y_{i}^{*}\perp z_{i}|\mathbf{x}_{i}$.

Under Assumption 1, the right censored log-likelihood is defined as,

$\displaystyle\mathcal{L}_{\text{C}}\left(\theta\right)=-\sum_{i\in\mathcal{D}}$
$\displaystyle\Big{(}\underbrace{\ell_{i}\log\left(\varphi\left(y_{i}|\mathbf{x}_{i},\theta\right)\right)}_{\text{Observed
loss}}+$ (1)
$\displaystyle\underbrace{(1-\ell_{i})\log\left(1-\Phi\left(z_{i}|\mathbf{x}_{i},\theta\right)\right)}_{\text{Censored
loss}}\Big{)}\,,$

where $\Phi$ is the Cumulative Distribution Function (CDF) and $\varphi$ is
the Probability Density Function (PDF) of
$p(y^{*}_{i}|\mathbf{x}_{i},\theta)$. The term $(1-\Phi)$ is often called the
censoring or survival distribution. While we focus on right-censoring, left
censoring (i.e. $y_{i}=\max(y^{*}_{i},z_{i})$) can be handled by inverting
$y_{i}\forall i$.

The class of models that can fit the distribution for $\varphi$ and $\Phi$ is
broad. It includes, in practice, all Bayesian models for which
$p(y_{i}^{*}|x_{i},\theta)$ has a fixed PDF and CDF, and
$p(\theta)=p(\theta|\mathcal{D})$ is the posterior distribution given the
observed dataset $\mathcal{D}$. Common models include deep ensembles
[Lakshminarayanan et al., 2017] and neural networks with stochastic parameters
[Gal and Ghahramani, 2016, Sharma et al., 2023].

###  2.1 Active Learning

In the supervised setting, active learning involves having a model select
which labels to acquire during training to increase the model performance
[MacKay, 1992a, Settles, 2009]. It maximises an acquisition function, which
captures the utility of acquiring the label for a given input [Kirsch and Gal,
2022]. We are interested in such settings, but where the data points are
subject to censoring.

Typically, one starts with a small training dataset,

$\mathcal{D}^{\text{train
}}=\left\\{\left(\mathbf{x}_{i},y_{i},\ell_{i}\right)\right\\}_{i=1}^{n}\,.$
(2)

which is used to train a probabilistic model with likelihood
$p(y_{i}^{*}|\mathbf{x}_{i},\theta)$. Then, from a larger (finite or infinite)
pool of unlabelled data,

$\mathcal{D}^{\text{pool }}=\left\\{\mathbf{x}_{i}\right\\}_{t=1}^{m},$ (3)

the model is used to actively select $\mathbf{x}_{i}$ to acquire a label for
[Kirsch and Gal, 2022]. Once the label is acquired, the sample is added to the
training set. In the pool, $\mathcal{D}^{\text{pool }}$, the censorship status
of new observations is _unknown_ , i.e., during acquitions of new
observations, both $y_{i}$ and $\ell_{i}$ are unknown [Nezhad et al., 2019].
Thus, acquiring new labels involves obtaining its label $y_{i}$ and its
censorship status $\ell_{i}$ [Vinzamuri et al., 2014].

###  2.2 Bayesian Experimental design

Bayesian experimental design is a formal framework for quantifying the
information gained from an experiment [Lindley, 1956]. In active learning, we
can view the input $\mathbf{x}_{i}$ as the design of an experiment and the
acquired label $y_{i}$ as the experiment’s outcome and formalise the
information gained from observing $y_{i}$ [Bickford Smith et al., 2023].

Let $\theta$ be the quantity we are trying to infer. Given a prior (or the
most recent knowledge), $p(\theta)$, and a likelihood function,
$p(y_{i}|\mathbf{x}_{i},\theta)$, then we can quantify the information gain
($\operatorname{IG}$) in $\theta$ due to an acquisition of
$(\mathbf{x}_{i},y_{i})$, as the reduction in Shannon entropy in $\theta$ that
results from observing $(\mathbf{x}_{i},y_{i})$:

$\operatorname{IG}_{\theta}(\mathbf{x}_{i},y_{i})=\operatorname{H}[p(\theta)]-\operatorname{H}[p(\theta|\mathbf{x}_{i},y_{i})]\,.$
(4)

Since $y_{i}$ is a random variable, the expected information of $y_{i}$, can
be computed across multiple simulated outcomes, using

$p_{\theta}(y_{i}|\mathbf{x}_{i})=\mathbb{E}_{p(\theta)}[p(y_{i}|\mathbf{x}_{i},\theta)],$
(5)

which leads to the expected information gain,

$\operatorname{EIG}_{\theta}(\mathbf{x}_{i})=\mathbb{E}_{p_{\theta}(y_{i}|\mathbf{x}_{i})}\left[\operatorname{H}\left[p(\theta)\right]-\operatorname{H}\left[p(\theta|\mathbf{x}_{i},y_{i})\right]\right]\,.$
(6)

This is the expected reduction in uncertainty of $\theta$ after conditioning
on $(\mathbf{x}_{i},y_{i})$. Equivalently it is the mutual information between
$\theta$, and $y_{i}$ given $\mathbf{x}_{i}$, denoted
$\operatorname{I}(y_{i},\theta|\mathbf{x}_{i})$ [Bickford Smith et al., 2023].

###  2.3 Bayesian active learning

The expected information gain has often been the basis for Bayesian active
learning, seeking to acquire data points that provide high information gain in
the model parameters $\theta$. This acquisition function is referred to as the
_Bayesian Active Learning by Disagreement_ (BALD) [Houlsby et al., 2011]:

BALD
$\displaystyle\left(\mathbf{x}_{i}\right)=\operatorname{I}(y_{i},\theta|\mathbf{x}_{i})$
(7)
$\displaystyle=\mathbb{E}_{p_{\theta}(y_{i}|\mathbf{x}_{i})}[\operatorname{H}[p(\theta)]-\operatorname{H}[p(\theta|\mathbf{x}_{i},y_{i})]]$
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i}|\mathbf{x}_{i})]-\operatorname{H}[p_{\theta}(y_{i}|\mathbf{x}_{i},\theta)]]\,.$

The BALD score is often used when the update to the model parameters is non-
Bayesian, for example, when applying Monte Carlo dropout in a neural network
[Gal et al., 2017]. For Bayesian active learning without censoring, the
$\text{BALD}(\mathbf{x}_{i})$ acquisition function can be used for
classification and regression methods, as the entropies are well-defined for
these tasks [Gal et al., 2017, Jesson et al., 2021].

##  3 Censoring and Information

Ideally, we would still like to use the BALD objectives for the censored
regression case. However, we must consider that, for a new observation
$\mathbf{x}_{i}$ in the pool, the corresponding label $y_{i}$ can provide a
varying amount of information for the distribution of $y^{*}_{i}$ and
$\theta$, depending on the censorship status of the label [Baxter, 1989,
Hollander et al., 1985, 1987, 1990]. The censorship status of new
observations, $\mathbf{x}_{i}$, is unknown before we acquire the label, which
means that the information provided by the label is unknown at the time of
acquisition.

To use the $\operatorname{EIG}$ and BALD acquisition functions, we will derive
the Shannon entropy for a model trained with Equation 1 and adopt Bayesian
experimental design ideas to reason about the information gain that $y_{i}$
provides to the true distributions of $y^{*}_{i}$. Using the derived entropy,
we extend the BALD objective to the censored case and use this as an
acquisition function for Bayesian active learning in this setting. For the
entropy equations in the following, we omit the dependency on $\mathbf{x}_{i}$
and $\theta$ for readability.

###  3.1 Information of two experiments

We consider the acquisition of a new label $y$, where $y$ can come from two
different probability distributions. Following Lindley [1956], the information
that $y$ holds (i.e., Shannon entropy) is either the entropy from the first or
the second probability distributions.

#### Definition 1.

If $p(y)$ is a mixture of two distributions, that is, $p(y)=\lambda
p_{1}(y)+(1-\lambda)p_{2}(y)$ with $\lambda$, $0<\lambda<1$, then observation
$y$ is from to the density $p_{1}(y)$ with probability $\lambda$ and with
probability $1-\lambda$, $y$ is from $p_{2}(y)$. The the entropy of $y$ is

$\displaystyle H[p(y)]$ $\displaystyle=H[\lambda
p_{1}(y)+(1-\lambda)p_{2}(y)]$ (8) $\displaystyle=\lambda
H\left[p_{1}(y)\right]+(1-\lambda)H\left[p_{2}(y)\right].\ $

We refer the reader to Lindley [1956] for theoretical analysis of this
information measure.

###  3.2 Censored information

In censored regression, the censoring status of new data points,
$\mathbf{x}_{i}$, from the pool, $\mathcal{D}^{\text{pool }}$, is unknown, and
we do not know if $y=y^{*}$ or $y=z$. Therefore, an observed label $y$ will
have varying entropy about $p(y^{*})$ depending on its censoring status. This
case is analogous to Definition 1, where $y$ is either obtained according to
the distribution $\varphi$ if $y$ is uncensored, or from the censoring
distribution, $(1-\Phi)$, if $y$ is censored. We consider the entropy of
$y^{*}$ as:

#### Proposition 1.

(Information of censored experiments) Let $\lambda$ be the probability that a
new observation is censored, i.e. $p(\ell=1)=\lambda$. Then with probability
$\lambda$, $y$ is an uncensored value and is from the density $\varphi(y)$.
With probability ($1-\lambda)$, the observation is censored, in which case the
probability density of $y$ is $(1-\Phi(z))$. The entropy of $y$ when it is
subject to censoring is,

$\operatorname{H}[p(y)]=H\left[\lambda\varphi(y)+(1-\lambda)(1-\Phi(z))\right]\,.$
(9)

#### Proof

In the case of non-censorship, the entropy of $y$ and the continuous
distribution $p(y)$ corresponds to the Shannon differential entropy, defined
as,

$\operatorname{H}[p(y)]=-\int p(y)\log p(y)dy\,.$ (10)

However, given the right censorship at $z$, $y$ is defined on the measurable
space $(-\infty;z]$, and we split the integral at the censoring threshold $z$
into an uncensored and censored case. For the case where the observation $y$
is not censored and in the set $(-\infty;z]$, then $p(y)=\varphi(y)$ and if
$y^{*}>z$, then the observations are censored and $p(y)=1-\Phi(z)$. Therefore,
we can rewrite the entropy as,

$\displaystyle\operatorname{H}[p(y)]=$
$\displaystyle-\int_{-\infty}^{\tau}p(y^{*})\log\varphi(y^{*})\,dy^{*}$ (11)
$\displaystyle-\int_{\tau}^{\infty}p(y^{*})\log(1-\Phi(z))\,dy^{*}\,.$

If we assume that $z$ is known (or constant), we can approximate this integral
with Monte Carlo samples of $y^{*}$ as,

$\displaystyle\operatorname{H}[p(y)]=-\mathbb{E}_{y^{*}\sim
p\left(y^{*}\right)}[$ $\displaystyle\mathbbm{1}\\{y^{*}\leq
z\\}]\log\varphi\left(y^{*}\right)+$ (12)
$\displaystyle\mathbbm{1}\\{y^{*}>z\\}\log(1-\Phi(z))]\,.$

The entropy can be interpreted as if we know the censoring; we know how much
information we can expect from a new observation. However, the censoring
threshold is _unknown_ , and instead of a hard assignment, we can consider the
probability $\lambda$ of an observation being censored or not. This leads to
the following entropy,

$\displaystyle\operatorname{H}\left[p(y)\right]=-\mathbb{E}_{y^{*}\sim
p\left(y^{*}\right)}[$ $\displaystyle\lambda\log\varphi\left(y\right)+$ (13)
$\displaystyle(1-\lambda)\log(1-\Phi(z))]\,.$

The entropy can be interpreted as follows: if we know the censoring status, we
understand how much information to expect from a new observation [Hollander et
al., 1990].

###  3.3 Conditional entropy

The entropy derived in Equation 13, involves the expectation with respect to
the distribution of $p(y^{*})$, and is essentially a form of conditional
entropy by conditioning on the censorship status of $y$. The entropy captures
the uncertainty in the mixture distribution, considering the influence of the
censorship status. It is, therefore, easy to show that this leads to the
following entropy,

$\displaystyle\operatorname{H}\left[p(y|\ell)\right]=-\mathbb{E}_{y^{*}\sim
p\left(y^{*}\right)}[$ $\displaystyle p(\ell)\log\varphi\left(y\right)+$ (14)
$\displaystyle(1-p(\ell))\log(1-\Phi(z))]\,.$

We refer the reader to Hollander et al. [1985, 1987, 1990] for a theoretical
analysis and a discussion of the entropy with censored data.

##  4 Expected information gain in censored acquisitions

We can use the derived entropy to calculate the information that new observed
targets $y_{i}$ provide to the distribution of $y^{*}$. However, the
acquisition of new labels $y_{i}$ not only requires obtaining new values of
$y_{i}$, but it also involves acquiring new censoring indicators $\ell_{i}$
[Nezhad et al., 2019]. Consequently, it is necessary to account for the mutual
information between $y_{i}$ and $\theta$ and consider the information provided
by $\ell_{i}$. As a result, we jointly compute the mutual information between
$(y_{i},\ell_{i})$ and $\theta$. This leads to the following mutual
information,

$\displaystyle\mathcal{C}$
$\displaystyle\text{-BALD}(\mathbf{x}_{i})=\operatorname{I}\left[(y_{i},\ell_{i}),\theta|\mathbf{x}_{i}\right]$
(15)
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i},\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p(y_{i},\ell_{i}|\mathbf{x}_{i},\theta)]]\,.$
$\displaystyle=\operatorname{I}\left[y_{i},\theta|\ell_{i},\mathbf{x}_{i}\right]+\operatorname{I}\left[\ell_{i},\theta|\mathbf{x}_{i}\right]\,.$

We provide the proof in the Appendix A.1.

In the censored regression, the information gained from observing $y_{i}$ and
$\ell_{i}$ is the information provided by observing the label $y_{i}$ given
the censoring indicator $\ell_{i}$, plus the information from observing the
censoring indicator $\ell_{i}$. The mutual information criteria can be
computed as the BALD objectives,

$\displaystyle\operatorname{I}\left[y_{i},\theta|\ell_{i},\mathbf{x}_{i}\right]$
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i}|\ell_{i},\mathbf{x}_{i})]-\operatorname{H}[p(y_{i}|\ell_{i},\mathbf{x}_{i},\theta)]]\,,$
(16)
$\displaystyle\operatorname{I}\left[\ell_{i},\theta|\mathbf{x}_{i}\right]$
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p(\ell_{i}|\mathbf{x}_{i},\theta)]]\,.$
(17)

###  4.1 Modelling approach

A fundamental limitation of $\mathcal{C}\text{-BALD }$for active learning is
that the primary model from Equation 1 only approximates the distribution
$\varphi$ and $\Phi$. This means that during acquisition, there is no
knowledge of the potential censoring status of new observations $\ell_{i}$,
which should be used in the mutual information (Equation 15), and there is no
knowledge of the potential censoring threshold $z_{i}$, which should be used
to compute the entropy of $y_{i}$ (Equation 14). Therefore, applying
$\mathcal{C}\text{-BALD }$in practice is not straightforward. To overcome
these challenges, we propose explicitly modelling the probability of being
censored $\ell_{i}$ and the censoring threshold $z_{i}$, which we describe
below. Figure 2 provides an overview of the proposed modelling approach.

$\mathbf{x}_{i}$$y_{i}^{*}$$z_{i}$$y_{i}$$\ell_{i}$$\theta$$N$ Figure 2:
Overview of the modelling approach. We propose to model the distributions
$p_{\theta}(\ell_{i}|\mathbf{x}_{i})$, $p_{\theta}(y_{i}|\mathbf{x}_{i})$, and
$p_{\theta}(y^{*}_{i}|\mathbf{x}_{i})$. Shaded circles indicate observed
values and unshaded circles indicate latent values.

#### Modelling of $\mathbf{\ell_{i}}$:

Recall that the censoring indicator $\ell_{i}=\mathbbm{1}\\{y^{*}_{i}\leq
z_{i}\\}$ is observed for each data point in a censored dataset. It is a
binary indicator of whether the observations are censored or not. We propose
to approximate the distribution of $p(\ell_{i}|\mathbf{x}_{i},\theta)$. We
parameterise $p(\ell_{i}|\mathbf{x}_{i},\theta)$, as a Bernoulli distribution,
and infer the parameters $\theta$ using the binary cross entropy
($\mathcal{L}_{\text{BCE}}(\theta)$). We approximate the probability of
censoring new observations using this distribution, i.e., $\lambda_{i}$.
Consequently, this explicit modelling of $\ell_{i}$ allows us to compute the
mutual information $\operatorname{I}[\ell_{i},\theta|\mathbf{x}_{i}]$.

#### Modelling of $\mathbf{z_{i}}$:

Explicit modelling of $z_{i}$ is more challenging, as it is not fully observed
(similar to $y^{*}_{i}$) Recall that we observe,
$y_{i}=\min(y^{*}_{i},z_{i})$, i.e. when an observation is censored, we
observe $z_{i}$ or if it is not censored, $y^{*}$. We only observe $z_{i}$
when an observation is censored. To approximate $z_{i}$, we propose
approximating the continuous distribution of
$p(y_{i}|\mathbf{x}_{i},\theta_{i})$ for new observations. We use
$p(y_{i}|\mathbf{x}_{i},\theta_{i})$ to estimate the _observed_ values,
$\hat{y}_{i}$, for new data points in the $\mathcal{D}^{\text{pool }}$.

#### Entropy estimation:

With this explicit modelling approach, we can approximate the entropy from the
observation $y$ to the distribution $y^{*}$,

$\displaystyle\operatorname{H}[p(y_{i}|\ell_{i})]=$
$\displaystyle-\mathbb{E}_{y^{*}_{i}\sim
p\left(y_{i}\right)}[{\color[rgb]{1,0.49,0.05}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.49,0.05}p_{\theta}(\ell_{i}|\mathbf{x}_{i})}\log{\color[rgb]{0.8594,0.2695,0.1914}\definecolor[named]{pgfstrokecolor}{rgb}{0.8594,0.2695,0.1914}\varphi}\left({\color[rgb]{0.122,0.467,0.706}\definecolor[named]{pgfstrokecolor}{rgb}{0.122,0.467,0.706}\hat{y}}\right)+$
(18)
$\displaystyle(1-{\color[rgb]{1,0.49,0.05}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.49,0.05}p_{\theta}(\ell_{i}|\mathbf{x}_{i})})\log(1-{\color[rgb]{0.8594,0.2695,0.1914}\definecolor[named]{pgfstrokecolor}{rgb}{0.8594,0.2695,0.1914}\Phi}({\color[rgb]{0.122,0.467,0.706}\definecolor[named]{pgfstrokecolor}{rgb}{0.122,0.467,0.706}\hat{y}}))]\,.$

###  4.2 Summary and implementation details

We want to use the mutual information between observations of $y_{i}$,
$\ell_{i}$, and the model parameters $\theta$ to acquire new labels to reduce
model uncertainty about $y^{*}_{i}$. Since the distribution of $y^{*}$ is not
fully observed, we use the entropy defined in Equation 14 to compute the
mutual information. However, the entropy relies on the knowledge of unknown
variables $z_{i}$ and $\ell_{i}$. We propose to model them explicitly,
resulting in the estimated entropy of Equation 18.

Figure 3: Overview of the fit of the proposed modelling approach on the 1-D
synthetic dataset. Grey areas indicate where $y^{*}\leq z$, and the black line
is the true function we are trying to approximate. Red: Estimated distribution
of the true function. Blue: Estimated distribution of the observed values.
Orange: Estimated probability of being censored (scaled between 0 and 2 for
illustration purposes.)

Figure 4: Left): The mutual information calculations for the label $y$ and the
censoring status. Grey areas indicate areas with complete censoring. Most
information comes from the cross-over point between the censored and non-
censored values. Right): The right censored NLL for the models across
different acquisition functions on the synthetic dataset (mean $\pm$ standard
error). $\mathcal{C}\text{-BALD }$achieves the best overall fit on the test
set.

Implementation: We will use Gaussian distributions for $y^{*}_{i}$ and $y_{i}$
and a Bernoulli distribution for $\ell_{i}$. When $y^{*}_{i}$ is assumed to be
Gaussian, the model corresponds to a Tobit model [Tobin, 1958]. We enforce the
constraint that $\sigma^{*}_{i}$ and $\sigma_{i}$ should be positive by
applying the softplus activation function on these parameters. The parameters
of the Gaussian $p(y_{i}|\mathbf{x}_{i},\theta)$ are approximated with the
maximum likelihood of the parameters ($\mathcal{L}_{\text{MLE}}(\theta))$. To
summarise,

$\displaystyle p(y^{*}_{i}|\mathbf{x}_{i},\theta)$
$\displaystyle\sim\underbrace{\mathcal{N}(\mu_{i}^{*},\sigma_{i}^{2*}|\mathbf{x}_{i},\theta)}_{{{\color[rgb]{0.8594,0.2695,0.1914}\definecolor[named]{pgfstrokecolor}{rgb}{0.8594,0.2695,0.1914}\text{True
distribution of $y^{*}_{i}$.}}}}\,,$ (19) $\displaystyle
p(y_{i}|\mathbf{x}_{i},\theta)$
$\displaystyle\sim\underbrace{\mathcal{N}(\mu_{i},\sigma_{i}^{2}|\mathbf{x}_{i},\theta)}_{{{\color[rgb]{0.122,0.467,0.706}\definecolor[named]{pgfstrokecolor}{rgb}{0.122,0.467,0.706}\text{Distribution
of $y_{i}$.}}}}\,,$ (20) $\displaystyle p(\ell_{i}|\mathbf{x}_{i},\theta)$
$\displaystyle\sim\underbrace{\operatorname{Ber}(\lambda_{i}|\mathbf{x}_{i},\theta)}_{{{\color[rgb]{1,0.49,0.05}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.49,0.05}\text{Distribution
of $\ell_{i}$.}}}}\,.$ (21)

We model all these distributions with a single Bayesian neural network with
stochastic parameters. The output of the Bayesian Neural network is the
distributional parameters of the distributions
$p(y_{i}^{*}|\mathbf{x}_{i},\theta)$, $p(y_{i}|\mathbf{x}_{i},\theta)$ and
$p(\ell_{i}|\mathbf{x}_{i},\theta)$, i.e. five outputs neurons for the set
$\\{\mu_{i}^{*},\sigma_{i}^{*},\mu_{i},\sigma_{i},\lambda_{i}\\}$.

The parameters of the neural network, $\theta$, are inferred using the total
loss from the maximum likelihood estimation of all these distributions,

$\mathcal{L}(\theta)=\mathcal{L}_{\mathcal{C}}(\theta)+\mathcal{L}_{\text{MLE}}(\theta)+\mathcal{L}_{\text{BCE}}(\theta)\,.$
(22)

Figure 3 shows the fit of the proposed model for all the different
distributions on a synthetic dataset. Using all the explicit modelling of
$y^{*}_{i}$, $y_{i}$ and $\ell_{i}$, we can compute the
$\mathcal{C}\text{-BALD }$objective and use it as an acquisition function in
active learning.

##  5 Experiments

In this section, we present the results of the proposed acquisition function
with multiple experiments on synthetic and real-world datasets.

Models: We implement the Bayesian Neural Network with stochastic parameters
using Monte Carlo Dropout [Gal and Ghahramani, 2016]. We use three layers, 128
hidden units, a dropout probability of 0.25, and the ADAM optimiser with a
learning rate of $0.3\cdot 10^{-3}$ [Kingma and Ba, 2014] and the ReLU
activation function111In Appendix C, we experiment with different model
architectures..

Baselines: We compare the proposed acquisition function with the following
baselines: Random acquisitions, which randomly acquires data points in
$\mathcal{D}^{\text{pool }}$, the Entropy (Entropy) of Bayesian neural
networks, which is proportional to variance between the individual’s models in
the sampled ensemble, $\operatorname{Var}_{\theta\sim
p(\theta|\mathcal{D})}[p(y_{i}|\mathbf{x}_{i})]$, and the BALD objective from
Equation 7.

Name | Num. features | Censorship | $n_{0}$ | Acquisition size | Acquisition steps | repetitions | $\mathcal{D}^{\text{pool }}$ | $\mathcal{D}^{\text{val }}$ | $\mathcal{D}^{\text{test }}$
---|---|---|---|---|---|---|---|---|---
Synthetic | 1 | $44\%$ | 10 | 3 | 150 | 50 | 9000 | 250 | 500
BreastMSK | 5 | $77\%$ | 5 | 3 | 150 | 50 | 1285 | 183 | 366
Metabric | 9 | $42\%$ | 5 | 3 | 150 | 50 | 1523 | 76 | 305
Whas | 6 | $58\%$ | 5 | 3 | 150 | 50 | 1310 | 65 | 263
GBSG | 7 | $37\%$ | 5 | 3 | 150 | 50 | 1546 | 137 | 549
Support | 14 | $32\%$ | 5 | 3 | 150 | 50 | 7098 | 355 | 1420
Churn | 26 | $53\%$ | 5 | 3 | 150 | 50 | 1276 | 136 | 546
Credit Risk | 47 | $30\%$ | 5 | 3 | 150 | 50 | 650 | 70 | 280
SurvMNIST | $28\times 28$ | $53\%$ | 100 | 5 | 100 | 25 | 60000 | 5000 | 5000

Table 1: Overview of the various datasets used in this analysis, including the
number of features and the percentage of censorship in
$\mathcal{D}^{\text{pool }}$. We also include $n_{0}$ as the initial data
points in $\mathcal{D}^{\text{train }}$

Evaluation: We evaluate the acquisition function based on the method outlined
by Riis et al. [2022]. To quantify the performance of the acquisition
function, we evaluate the relative decrease in the area under the curve (RD-
AUC) across the entire active learning experiment. We compare the relative
decrease to a baseline acquisition function (Random) and evaluate the models’
right censored negative log-likelihood (NLL) on a test set
($\mathcal{D}^{\text{test }}$). Since the NLL is not bounded by 0, we use the
lowest NLL obtained across all the acquisition functions as a lower bound for
the metric. We compute the average across all the number of acquisitions,
$N_{\text{Acq}}$. The RD-AUC is defined as follows:

$\text{RD-
AUC}=\frac{1}{N_{\text{Acq}}}\sum_{i=0}^{N_{\text{Acq}}}\left(\frac{NLL_{\text{Random}}-NLL_{s}}{NLL_{\text{Random}}}\right)\,,$
(23)

where $NLL_{s}$ is the negative log-likelihood of the model with the
acquisition function $s$ and $NLL_{\text{Random}}$ is the negative log-
likelihood of from Random acquisition.

Synthetic Data: We begin our empirical evaluation of the proposed acquisition
by considering the following 1D synthetic dataset, with
$x_{i}=\mathcal{N}(5,1)$, and,

$\displaystyle y_{i}^{*}$
$\displaystyle=\frac{1}{2}\sin(2x_{i})+2+\varepsilon_{i}\,,$ (24)
$\displaystyle z_{i}$
$\displaystyle=\frac{1}{2}\cos(2x_{i})+2+\varepsilon_{i}\,,$ (25)

$y_{i}=\min(y^{*}_{i},z_{i})$, and $\ell_{i}=\mathbbm{1}\\{y^{*}_{i}\leq
z_{i}\\}$ and $\varepsilon_{i}\sim\mathcal{N}(0,0.01|x_{i}|)$. The dataset is
depicted in Figure 1 and our proposed modelling fit in Figure 3. We generate a
small pool of labelled data points ($n_{0}=10$), a larger set of unlabelled
data points $|$$\mathcal{D}^{\text{pool }}$$|=9000$, and a
$|$$\mathcal{D}^{\text{test }}$$|=500$.

We train a model of the small pool of labelled data and acquire three new data
points with labels every iteration. During each training step, we use a small
validation set $\mathcal{D}^{\text{val }}$with 250 observations to evaluate
the models and apply early stopping on the right censored maximum likelihood.

Figure 4 shows the $\mathcal{C}\text{-BALD }$scores across the entire range of
$x$. $\mathcal{C}$-BALD assigns a high mutual information value in regions
where the censoring status changes, i.e. when the model is uncertain about the
information that new samples will provide. In the right of Figure 4, we show
the right censored negative log-likelihood for the different acquisition
functions. We find that $\mathcal{C}\text{-BALD }$achieves the best overall
fit of the data with the lowest NLL, which shows that it identifies which data
point provides the most information to the model.

Real Datasets: We test the proposed functions on seven real-world datasets:
five from a biomedical context [Katzman et al., 2018] and two from a
predictive analytics context [Fotso et al., 2019]. Three datasets focus on
estimating the survival time for various types of cancer patients (BreastMSK,
METABRIC, and GBSG), one dataset for modelling the survival time of myocardial
infarction (WHAS), and the last dataset estimates the survival time for
critically-ill hospital patients (SUPPORT). For the predictive analytics
datasets, we focus on predicting the time customers remain subscribed to a
service (Churn) and the other on estimating the time for borrowers to repay
their credit (Credit Risk). These datasets contain _real_ censoring, i.e., no
synthetic censoring is applied to them222A more extensive summary of these
datasets can be found in Appendix B.3..

Table 1 summarises the datasets used in the experiments, including the number
of features, the percentage of censored observations, and the total number of
observations. Additionally, it includes a summary of the parameters used for
the active learning experiments for each dataset. The results reported are
averages over the number of repetitions for each dataset and acquisition
function (mean $\pm$ standard error).

Table 2 reports the RD-AUC compared across the different scoring functions.
Figure 5 shows the right-censored NLL across the different runs for two real-
world datasets. We find that the proposed acquisition function leads to better
acquisition of new data points by obtaining a superior fit on the test set
compared to the baselines.

Dataset | Random | Entropy | BALD | $\mathcal{C}\text{-BALD }$
---|---|---|---|---
Synthetic | $0.00\pm 0.00$ | $8.65\pm 0.42$ | $-0.12\pm 0.14$ | $\mathbf{33.49\pm 1.11}$
BreastMSK | $0.00\pm 0.00$ | $8.21\pm 1.43$ | $-1.89\pm 0.66$ | $\mathbf{8.75\pm 1.42}$
Metabric | $0.00\pm 0.00$ | $-0.67\pm 0.39$ | $2.25\pm 0.34$ | $\mathbf{18.26\pm 0.94}$
whas | $0.00\pm 0.00$ | $0.42\pm 0.27$ | $\mathbf{1.68\pm 0.17}$ | $0.26\pm 0.32$
GBSG | $0.00\pm 0.00$ | $-0.81\pm 0.05$ | $-0.04\pm 0.05$ | $\mathbf{5.58\pm 0.05}$
support | $0.00\pm 0.00$ | $0.70\pm 0.02$ | $-0.53\pm 0.01$ | $\mathbf{4.55\pm 0.02}$
churn | $0.00\pm 0.00$ | $5.14\pm 0.31$ | $0.17\pm 0.21$ | $\mathbf{32.75\pm 0.87}$
credit risk | $0.00\pm 0.00$ | $-0.72\pm 0.36$ | $-0.17\pm 0.33$ | $\mathbf{22.11\pm 0.64}$
Survmnist | $0.00\pm 0.00$ | $-0.05\pm 0.28$ | $1.06\pm 0.30$ | $\mathbf{13.47\pm 0.66}$

Table 2: Relative decrease in the area under the curve (RD-AUC) compared to
the Random scoring function. A higher value in the table represents better
performance, with the best performance highlighted in bold.

Figure 5: Results of the real-world experiments on two of the seven datasets,
namely the METABRIC and CREDIT RISK datasets, respectively. The figure shows
the NLL (mean $\pm$ standard error) across the multiple repetitions of the
experiment.

High-dimensional data: Lastly, we evaluate the performance of our proposed
scoring functions with Bayesian convolutional neural networks on the SurvMNIST
dataset [Goldstein et al., 2020]. In SurvMNIST, each label is replaced with a
random draw from a Gamma distribution, with different distributional
parameters across the labels [Pearce et al., 2022]. The observations in the
dataset are censored uniformly, between the minimum and the 90th percentile in
the training set [Goldstein et al., 2020]. The initial training set contains
ten samples from each class in the dataset333The details of the gamma
distributions and the model architecture can be found in Appendix B.4..

The experiment on the SurvMNIST dataset shows that the proposed scoring
functions outperformed the baseline functions, as shown in Table 2.

##  6 Related work

The study of the information that an experiment or observation provides was
introduced by Lindley [1956] and has often been the basis for new acquisition
functions in active learning [MacKay, 1992a, b]. The study of information in
censored experiments has traditionally focused on survival experiments, where
observations are studied over time [Hollander et al., 1985, 1990]. In survival
experiments, an individual is observed for an amount of time and is considered
censored if the person drops out of the experiment [Hollander et al., 1987].
For the discrete and continuous case, the entropy calculations come down to
the integral over the time an individual was observed [Baxter, 1989].
Hollander et al. [1987] shows that information decreases after censorship and
that uncensored observations provide the most information Traditionally,
survival experiments were the focus because conducting such studies in the
past was expensive. Due to electronic health records, survival analysis today
often employs large-scale datasets [Qi et al., 2023]. However, in other areas
where censored regression is applied, such as transportation systems [Hüttel
et al., 2022, 2023], subscription-based businesses [Fader and Hardie, 2007,
Chandar et al., 2022, Maystre and Russo, 2022], and in health survival
applications [Nezhad et al., 2019, Lian et al., 2022], data can be expensive
to collect and label, necessitating the need for active learning in this
context.

Despite the challenges of censored data, there is limited research on active
learning in this context. Two notable exceptions from the survival analysis
literature include the work of Vinzamuri et al. [2014], who proposed a query
strategy based on discriminative gradients to identify the most informative
points, and the work of Nezhad et al. [2019], who suggested a query strategy
for acquiring data points with the highest expected performance increase if
their labels were known.

A popular approach is Bayesian Active Learning with the BALD objective
[Houlsby et al., 2011], specifically with its ability to work in conjunction
with deep neural networks [Gal et al., 2017] and extensions to batch-
acquisitions [Kirsch et al., 2019]. In the Deep Bayesian active learning, the
BALD objective has primarily been used for classification tasks with MC
Dropout models but has recently seen applications for deep regression tasks,
such as estimating causal treatment effects [Jesson et al., 2021] and for
black-box models [Kirsch, 2023]. While plenty of research has focused on the
BALD objective, to our knowledge, we are the first to explore the BALD
objective in censored regression.

Our work contributes to Bayesian and Censored regression by extending the
common BALD objective to the censored regression case. We propose a novel
modelling approach to approximate the information gain for new observations
when they are subject to censoring. Compared to previous work in active
learning with censored regression models, we quantify the information
theoretical quantities in the prediction space instead of approximating them
in the parameter space. Future work could explore the implications of
censoring on acquisition functions in the parameter space, such as other
state-of-the-art methods [Ash et al., 2020, 2021, Kothawade et al., 2022], as
well as explore other censoring schemes, such as interval-censored data.

##  7 Conclusion

This paper studies Bayesian active learning for censored regression problems.
We have shown that the BALD objective is inadequate for censored data, as the
expected information gain for new data points depends on the points’
censorship status. Motivated by this challenge, we derive the entropy for
censored distributions and propose the $\mathcal{C}$-BALD acquisition
function, which accounts for censored observations. Since $\mathcal{C}$-BALD
relies on unknown variables, we propose to model the variables with minimal
computational overhead to compute the mutual information. Empirically, across
synthetic and real datasets, we show that $\mathcal{C}\text{-BALD
}$outperforms BALD with both synthetic and real censoring.

###### Acknowledgements.

The research leading to these results has received funding from the
Independent Research Fund Denmark (Danmarks Frie Forskningsfond) under the
grant no. 0217-00065B.

## References

  * Ash et al. [2020] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  In _International Conference on Learning Representations_ , 2020.
  * Ash et al. [2021] Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham M. Kakade.  Gone fishing: Neural active learning with fisher embeddings.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.
  * Baxter [1989] L. A. Baxter.  A note on information and censored absolutely continuous random variables.  _Statistics & Risk Modeling_, 7(1-2):193–198, 1989.  doi:10.1524/strm.1989.7.12.193.
  * Bickford Smith et al. [2023] Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth.  Prediction-oriented bayesian active learning.  In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics_ , volume 206 of _Proceedings of Machine Learning Research_ , pages 7331–7348. PMLR, 25–27 Apr 2023.
  * Chandar et al. [2022] Praveen Chandar, Brian Thomas, Lucas Maystre, Vijay Pappu, Roberto Sanchis-Ojeda, Tiffany Wu, Ben Carterette, Mounia Lalmas, and Tony Jebara.  Using survival models to estimate user engagement in online experiments.  pages 3186–3195, 04 2022.  10.1145/3485447.3512038.
  * Cox [1972] D. R. Cox.  Regression models and life-tables.  _Journal of the Royal Statistical Society. Series B (Methodological)_ , 34(2):187–220, 1972.  ISSN 00359246.
  * Fader and Hardie [2007] Peter S. Fader and Bruce G.S. Hardie.  How to project customer retention.  _Journal of Interactive Marketing_ , 21(1):76–90, 2007.  ISSN 1094-9968.  https://doi.org/10.1002/dir.20074.
  * Foekens et al. [2000] J A Foekens, H A Peters, M P Look, H Portengen, M Schmitt, M D Kramer, N Brünner, F Jänicke, M E Meijer-van Gelder, S C Henzen-Logmans, W L van Putten, and J G Klijn.  The urokinase system of plasminogen activation and prognosis in 2780 breast cancer patients.  _Cancer Res_ , 60(3):636–643, February 2000.
  * Fotso et al. [2019] Stephane Fotso et al.  PySurvival: Open source package for survival analysis modeling, 2019.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * Gal et al. [2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep Bayesian active learning with image data.  In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1183–1192. PMLR, 06–11 Aug 2017.
  * Gammelli et al. [2020] Daniele Gammelli, Inon Peled, Filipe Rodrigues, Dario Pacino, Haci A. Kurtaran, and Francisco C. Pereira.  Estimating latent demand of shared mobility through censored gaussian processes.  _Transportation Research Part C: Emerging Technologies_ , 120, 2020.  10.1016/j.trc.2020.102775.
  * Goldstein et al. [2020] Mark Goldstein, Xintian Han, Aahlad Puli, Adler Perotte, and Rajesh Ranganath.  X-cal: Explicit calibration for survival analysis.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18296–18307, 2020.
  * Golsefidi et al. [2023] Atefeh Hemmati Golsefidi, Frederik Boe Hüttel, Inon Peled, Samitha Samaranayake, and Francisco Câmara Pereira.  A joint machine learning and optimization approach for incremental expansion of electric vehicle charging infrastructure.  _Transportation Research Part A: Policy and Practice_ , 178:103863, 2023.  ISSN 0965-8564.  https://doi.org/10.1016/j.tra.2023.103863.  URL https://www.sciencedirect.com/science/article/pii/S0965856423002835.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hollander et al. [1985] M Hollander, F Proschan, J Sconing, and FLORIDA STATE UNIV TALLAHASSEE DEPT OF STATISTICS.  Information in censored models.  _FSU Statistics Report M_ , 701, 1985.
  * Hollander et al. [1987] Myles Hollander, Frank Proschan, and James Sconing.  Measuring information in right-censored models.  _Naval Research Logistics (NRL)_ , 34(5):669–681, 1987.
  * Hollander et al. [1990] Myles Hollander, Frank Proschan, and James Sconing.  Information, censoring, and dependence.  _Lecture Notes-Monograph Series_ , 16:257–268, 1990.  ISSN 07492170.
  * Houlsby et al. [2011] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.  Bayesian active learning for classification and preference learning, 2011.
  * Hüttel et al. [2022] Frederik Boe Hüttel, Inon Peled, Filipe Rodrigues, and Francisco C. Pereira.  Modeling censored mobility demand through censored quantile regression neural networks.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(11):21753–21765, 2022.  10.1109/TITS.2022.3190194.
  * Hüttel et al. [2023] Frederik Boe Hüttel, Filipe Rodrigues, and Francisco Câmara Pereira.  Mind the gap: Modelling difference between censored and uncensored electric vehicle charging demand.  _Transportation Research Part C: Emerging Technologies_ , 153:104189, 2023.  ISSN 0968-090X.  https://doi.org/10.1016/j.trc.2023.104189.
  * Jesson et al. [2021] Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal.  Causal-bald: Deep bayesian active learning of outcomes to infer treatment-effects from observational data.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 30465–30478. Curran Associates, Inc., 2021.
  * Katzman et al. [2018] Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger.  Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network.  _BMC Medical Research Methodology_ , 18(1):24, Feb 2018.  ISSN 1471-2288.  10.1186/s12874-018-0482-1.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirsch [2023] Andreas Kirsch.  Black-box batch active learning for regression.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  Expert Certification.
  * Kirsch and Gal [2022] Andreas Kirsch and Yarin Gal.  Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities.  _Transactions on Machine Learning Research_ , 2022.  ISSN 2835-8856.  Expert Certification.
  * Kirsch et al. [2019] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal.  Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Knaus et al. [1995] William Knaus, Frank Harrell, Joanne Lynn, L Goldman, Russell Phillips, Alfred Connors, Jr, Neal Dawson, W Fulkerson, R Califf, N Desbiens, Peter Layde, Robert Oye, P Bellamy, Rabia Hakim, and D Wagner.  The support prognostic model. objective estimates of survival for seriously ill hospitalized adults. study to understand prognoses and preferences for outcomes and risks of treatments.  _Annals of internal medicine_ , 122:191–203, 03 1995.
  * Kothawade et al. [2022] Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer.  Prism: A rich class of parameterized submodular information measures for guided data subset selection.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Lemeshow et al. [2011] Stanley Lemeshow, Susanne May, and David W Hosmer Jr.  _Applied survival analysis: regression modeling of time-to-event data_.  John Wiley & Sons, 2011.
  * Lian et al. [2022] Jie Lian, Yonghao Long, Fan Huang, Kei Shing Ng, Faith M. Y. Lee, David C. L. Lam, Benjamin X. L. Fang, Qi Dou, and Varut Vardhanabhuti.  Imaging-based deep graph neural networks for survival analysis in early stage lung cancer using ct: A multicenter study.  _Frontiers in Oncology_ , 12, 2022.  ISSN 2234-943X.  10.3389/fonc.2022.868186.
  * Lindley [1956] D. V. Lindley.  On a Measure of the Information Provided by an Experiment.  _The Annals of Mathematical Statistics_ , 27(4):986 – 1005, 1956.  10.1214/aoms/1177728069.
  * MacKay [1992a] David J. C. MacKay.  Information-Based Objective Functions for Active Data Selection.  _Neural Computation_ , 4(4):590–604, 07 1992a.  ISSN 0899-7667.  10.1162/neco.1992.4.4.590.
  * MacKay [1992b] David J. C. MacKay.  The evidence framework applied to classification networks.  _Neural Computation_ , 4(5):720–736, 1992b.  10.1162/neco.1992.4.5.720.
  * Maystre and Russo [2022] Lucas Maystre and Daniel Russo.  Temporally-consistent survival analysis.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.
  * Nezhad et al. [2019] Milad Zafar Nezhad, Najibesadat Sadati, Kai Yang, and Dongxiao Zhu.  A deep active survival analysis approach for precision treatment recommendations: Application of prostate cancer.  _Expert Systems with Applications_ , 115:16–26, 2019.  ISSN 0957-4174.  https://doi.org/10.1016/j.eswa.2018.07.070.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Pearce et al. [2022] Tim Pearce, Jong-Hyeon Jeong, yichen jia, and Jun Zhu.  Censored quantile regression neural networks for distribution-free survival analysis.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.
  * Powell [1986] James L. Powell.  Censored regression quantiles.  _Journal of Econometrics_ , 32(1):143–155, 1986.  ISSN 0304-4076.  https://doi.org/10.1016/0304-4076(86)90016-3.
  * Qi et al. [2023] Shiang Qi, Neeraj Kumar, Mahtab Farrokh, Weijie Sun, Li-Hao Kuan, Rajesh Ranganath, Ricardo Henao, and Russell Greiner.  An effective meaningful way to evaluate survival models, 2023.
  * Riis et al. [2022] Christoffer Riis, Francisco Antunes, Frederik Boe Hüttel, Carlos Lima Azevedo, and Francisco C. Pereira.  Bayesian active learning with fully bayesian gaussian processes.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.
  * Schumacher et al. [1994] M Schumacher, G Bastert, H Bojar, K Hübner, M Olschewski, W Sauerbrei, C Schmoor, C Beyerle, R L Neumann, and H F Rauschecker.  Randomized 2 x 2 trial evaluating hormonal treatment and the duration of chemotherapy in node-positive breast cancer patients. german breast cancer study group.  _J Clin Oncol_ , 12(10):2086–2093, October 1994.
  * Settles [2009] Burr Settles.  Active learning literature survey, 2009.
  * Sharma et al. [2023] Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth.  Do bayesian neural networks need to be fully stochastic?  In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics_ , volume 206 of _Proceedings of Machine Learning Research_ , pages 7694–7722. PMLR, 25–27 Apr 2023.
  * Shen et al. [2018] Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar.  Deep active learning for named entity recognition.  In _International Conference on Learning Representations_ , 2018.
  * Tobin [1958] James Tobin.  Estimation of Relationships for Limited Dependent Variables.  _Econometrica_ , 26(1):24–36, 1958.  ISSN 00129682, 14680262.
  * Vinzamuri et al. [2014] Bhanukiran Vinzamuri, Yan li, and Chandan Reddy.  Active learning based survival regression for censored data.  _CIKM 2014 - Proceedings of the 2014 ACM International Conference on Information and Knowledge Management_ , pages 241–250, 11 2014.  10.1145/2661829.2662065.

Bayesian Active Learning for Censored Regression
(Supplementary Material)

##  Appendix A Derivations of the entropy of two independent probability
densities.

Here, we show the joint entropy between two independent distributions $x$ and
$y$. Under the assumption of independence between the distributions, the joint
entropy equals the sum of the entropy of $x$ and $y$. 2nd line is under the
assumption of independence between $x$ and $y$, and 4th line is the linearity
of expectations.

$\displaystyle\operatorname{H}\left[p(x,y)\right]$
$\displaystyle=\mathbb{E}\left[-\ln p(x,y)\right]$ (26)
$\displaystyle=-\mathbb{E}\left[\ln p(x,y)\right]$ (27)
$\displaystyle=-\mathbb{E}\left[\ln\left(p(x)p(y)\right)\right]$ (28)
$\displaystyle=-\mathbb{E}\left[\ln p(x)+\ln p(y)\right]$ (29)
$\displaystyle=-\mathbb{E}\left[\ln p(x)\right]-E\left[\ln p(y)\right]$ (30)
$\displaystyle=\operatorname{H}\left[p(x)\right]+\operatorname{H}\left[p(y)\right]$
(31)

###  A.1 Proof of C-BALD

Here, we derive the C-bald objective from Equation 15.

$\displaystyle\mathcal{C}\text{-BALD}$
$\displaystyle=\operatorname{I}\left[y_{i},l,\theta\mid\mathbf{x}_{i}\right]$
(32)
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i},\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p(y_{i},\ell_{i}|\mathbf{x}_{i},\theta)]]\,$
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i}|\ell_{i},\mathbf{x}_{i})]+\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p(y_{i}|\ell_{i},\mathbf{x}_{i},\theta)]-\operatorname{H}[p(\ell_{i}|\mathbf{x}_{i},\theta)]]\,$
$\displaystyle=\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i}|\ell_{i},\mathbf{x}_{i})]-\operatorname{H}[p(y_{i}|\ell_{i},\mathbf{x}_{i},\theta)]+\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i},\theta)]]\,$
$\displaystyle=\underbrace{\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(y_{i}|\ell_{i},\mathbf{x}_{i})]-\operatorname{H}[p(y_{i}|\ell_{i},\mathbf{x}_{i},\theta)]]}_{\text{Mutual
information between $y_{i}$ and $\theta$, assuming we know
$\ell$}}+\underbrace{\mathbb{E}_{p(\theta)}[\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i})]-\operatorname{H}[p_{\theta}(\ell_{i}|\mathbf{x}_{i},\theta)]]}_{\text{Mutual
information between $\ell_{i}$ and $\theta$}}\,$
$\displaystyle=\operatorname{I}\left[y_{i},\theta|\ell_{i},\mathbf{x}_{i}\right]+\operatorname{I}\left[\ell_{i},\theta|\mathbf{x}_{i}\right]\,.$

##  Appendix B Additional information of Experiments

###  B.1 Computational resources

The real and synthetic data experiments were run parallel on single-core CPUs
with varying computing power. The longest experiments ran for 8 hours for the
25 repetitions, i.e. an approximate time to run repetitions in 20 minutes. All
models are implemented in PyTorch [Paszke et al., 2019] The experiments on
SurvMNIST were run on a GV100 Volta (Tesla V100 - SXM2) with 32GB. The running
time for each scoring function is approximately 25 minutes per 1 experiment.
The 25 repetitions resulted in a total of 12 hours.

###  B.2 Synthetic 1D dataset

The synthetic dataset is generated using a simple sine function with the
following censorship,

$\displaystyle x_{i}$ $\displaystyle=\mathcal{N}(5,1)$ (33) $\displaystyle
y_{i}^{*}$ $\displaystyle=\frac{1}{2}\sin(2x_{i})+2+\varepsilon_{i}$ (34)
$\displaystyle z_{i}$
$\displaystyle=\frac{1}{2}\cos(2x_{i})+2+\varepsilon_{i}$ (35) $\displaystyle
y_{i}$ $\displaystyle=\begin{cases}y^{*}_{i}&\text{ if }y^{*}_{i}\leq z_{i}\\\
z_{i}&\text{ else}\end{cases}$ (36)

and $\ell_{i}=1\\{y^{*}_{i}<=z_{i}\\}$ and
$\varepsilon_{i}\sim\mathcal{N}(0,0.01|x_{i}|)$. We construct the test set
using the same approach. However, we extend it as $x\sim U(1.5,8.5)$. This
allows us to evaluate the fit across the entire $x$ range.

###  B.3 Real datasets

Here, we provide a brief description of the real-world datasets. Four (GSBG,
IHC4, Support, Whas) of the datasets are obtained from
https://github.com/jaredleekatzman/DeepSurv/tree/master/experiments/data.
Katzman et al. [2018] provides detail introduction of these datasets.
BreastMSK are obtained from
https://github.com/TeaPearce/Censored_Quantile_Regression_NN/tree/main/02_datasets.
Pearce et al. [2022] provides an introduction to this. The churn and credit
risk datasets are from  https://square.github.io/pysurvival/. Fotso et al.
[2019] provides an introduction to these. SurvMNIST was introduced in
[Goldstein et al., 2020].

Here we provide a short introduction to the datasets:

  * •

Rotterdam & German Breast Cancer Study Group (GBSG) requires prediction of
survival time for breast cancer patients [Foekens et al., 2000, Schumacher et
al., 1994]. We follow the same pre-processing steps as Katzman et al. [2018]
and Pearce et al. [2022]

  * •

Molecular Taxonomy of Breast Cancer International Consortium (METABRIC (IHC4))
requires prediction of survival time for breast cancer patients. Features
include clinical and expressions for four genes [Katzman et al., 2018, Pearce
et al., 2022].

  * •

Study to Understand Prognoses Preferences Outcomes and Risks of Treatment
(Support) requires predicting survival time in seriously ill hospitalised
patients. The 14 features are age, sex, race, number of comorbidities,
presence of diabetes, presence of dementia, presence of cancer, mean arterial
blood pressure, heart rate, respiration rate, temperature, white blood cell
count, serum sodium, and serum creatinine [Knaus et al., 1995].

  * •

Worcester Heart Attack Study (WHAS) requires prediction of acute myocardial
infarction survival. The five features are age, sex, body-mass-index, left
heart failure complications and order of MI [Lemeshow et al., 2011, Katzman et
al., 2018, Pearce et al., 2022]

  * •

BreastMSK requires prediction of survival time for patients with breast cancer
using tumour information. Features include ER, HER, HR, mutation count, and
TMB [Pearce et al., 2022]. Original from
https://www.cbioportal.org/study/clinicalData?id=breast_msk_2018

  * •

Credit risk, The task is to predict the time it takes a borrower to repay a
loan [Fotso et al., 2019]. Original from:
https://square.github.io/pysurvival/tutorials/credit_risk.html

  * •

Churn is the percentage of customers that stop using a company’s products or
services. The task is to predict when it will happen [Fotso et al., 2019].
Original from:  https://square.github.io/pysurvival/tutorials/churn.html

  * •

SurivalMNIST (SurvMNIST) was used in Goldstein et al. [2020], who modified it
from Sebastian Pölsterl’s blog:  https://k-d-w.org/blog/2019/07/survival-
analysis-for-deep-learning/. We follow the outline introduced by Pearce et al.
[2022]. It is based on MNIST  http://yann.lecun.com/exdb/mnist/, but each
target is drawn from a Gamma distribution according to the class with means
[11.25, 2.25, 5.25, 5.0, 4.75, 8.0, 2.0, 11.0, 1.75, 10.75] and variance [0.1,
0.5, 0.1, 0.2, 0.2, 0.2, 0.3, 0.1, 0.4, 0.6].

For all the datasets, we log-transform the response $y\sim\log y$.

###  B.4 SurvMNIST

Here we provide a brief overview of the parameters used for the gamma
distributions and the model architecture for the SurvMNIST experiments.

###  B.5 Parameters for gamma distributions

We use the same parameters for the gamma distributions as Pearce et al. [2022]

Digit |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9
---|---|---|---|---|---|---|---|---|---|---
Risk | 11.25 | 2.25 | 5.25 | 5.0 | 4.75 | 8.0 | 2.0 | 11.0 | 1.75 | 10.75
Variance | 0.1 | 0.5 | 0.1 | 0.2 | 0.2 | 0.2 | 0.3 | 0.1 | 0.4 | 0.6
Table 3: Overview of the SurvMNIST dataset and the corresponding gamma
distributions and their digits.

###  B.6 Architecture

The architecture for the SurvMNIST experiments is inspired by the
architectures proposed by Pearce et al. [2022] and Goldstein et al. [2020].

Table 4: Overview of the SurvMNIST model architecture.. Layers
---
Conv2d(64, ($5\times 5$))
GeLU() [Hendrycks and Gimpel, 2016]
Consistent Dropout ($0.25$) [Gal and Ghahramani, 2016]
AvgPool($2\times 2$)
Conv2d(128, ($5\times 5$))
GeLU()
Consistent Dropout($0.25$)
AvgPool ($2\times 2$)
Conv2d (258, ($3\times 3$))
GeLU()
Flatten()
Linear (128)
GeLU()
Consistent Dropout($0.25$)
Output

##  Appendix C Experiment with different sizes of Bayesian Neural networks

Here we provide additional results with varying amount of hidden param

Dataset | Hidden size | Layers | Random | Entropy | BALD | $\mathcal{C}\text{-BALD }$
---|---|---|---|---|---|---
BreastMSK | 64 | 2 | $0.00\pm 0.00$ | $3.57\pm 0.67$ | $0.80\pm 0.36$ | $-12.49\pm 0.33$
BreastMSK | 128 | 2 | $0.00\pm 0.00$ | $9.31\pm 0.96$ | $0.74\pm 0.41$ | $0.62\pm 0.60$
BreastMSK | 256 | 2 | $0.00\pm 0.00$ | $6.33\pm 0.86$ | $-0.73\pm 0.36$ | $9.67\pm 0.81$
BreastMSK | 64 | 3 | $0.00\pm 0.00$ | $8.29\pm 1.41$ | $0.04\pm 0.73$ | $-12.02\pm 0.73$
BreastMSK | 128 | 3 | $0.00\pm 0.00$ | $8.21\pm 1.43$ | $-1.89\pm 0.66$ | $8.75\pm 1.42$
BreastMSK | 256 | 3 | $0.00\pm 0.00$ | $5.50\pm 1.19$ | $-0.79\pm 0.68$ | $13.86\pm 1.55$
BreastMSK | 64 | 4 | $0.00\pm 0.00$ | $1.44\pm 1.36$ | $-0.98\pm 0.90$ | $-2.18\pm 1.45$
BreastMSK | 128 | 4 | $0.00\pm 0.00$ | $5.44\pm 1.39$ | $-0.53\pm 0.78$ | $15.50\pm 2.01$
BreastMSK | 256 | 4 | $0.00\pm 0.00$ | $5.05\pm 1.47$ | $-0.40\pm 0.91$ | $21.08\pm 2.45$
metabric | 64 | 2 | $0.00\pm 0.00$ | $-1.00\pm 0.39$ | $1.79\pm 0.36$ | $29.27\pm 0.99$
metabric | 128 | 2 | $0.00\pm 0.00$ | $-0.86\pm 0.34$ | $3.86\pm 0.34$ | $29.12\pm 1.18$
metabric | 256 | 2 | $0.00\pm 0.00$ | $-1.48\pm 0.32$ | $-0.17\pm 0.25$ | $33.61\pm 1.36$
metabric | 64 | 3 | $0.00\pm 0.00$ | $-1.42\pm 0.36$ | $1.84\pm 0.31$ | $19.20\pm 0.82$
metabric | 128 | 3 | $0.00\pm 0.00$ | $-0.67\pm 0.39$ | $2.25\pm 0.34$ | $18.26\pm 0.94$
metabric | 256 | 3 | $0.00\pm 0.00$ | $0.15\pm 0.37$ | $0.14\pm 0.30$ | $19.25\pm 0.76$
metabric | 64 | 4 | $0.00\pm 0.00$ | $-1.79\pm 0.36$ | $-1.69\pm 0.34$ | $7.42\pm 0.72$
metabric | 128 | 4 | $0.00\pm 0.00$ | $-1.47\pm 0.40$ | $-1.88\pm 0.37$ | $7.46\pm 0.72$
metabric | 256 | 4 | $0.00\pm 0.00$ | $1.78\pm 0.51$ | $0.04\pm 0.41$ | $15.47\pm 0.93$
whas | 64 | 2 | $0.00\pm 0.00$ | $0.46\pm 0.24$ | $1.22\pm 0.18$ | $-11.63\pm 0.17$
whas | 128 | 2 | $0.00\pm 0.00$ | $0.37\pm 0.20$ | $0.32\pm 0.12$ | $-4.28\pm 0.17$
whas | 256 | 2 | $0.00\pm 0.00$ | $1.86\pm 0.20$ | $0.83\pm 0.10$ | $2.22\pm 0.16$
whas | 64 | 3 | $0.00\pm 0.00$ | $-0.01\pm 0.36$ | $0.68\pm 0.23$ | $-5.81\pm 0.40$
whas | 128 | 3 | $0.00\pm 0.00$ | $0.42\pm 0.27$ | $1.68\pm 0.17$ | $0.26\pm 0.32$
whas | 256 | 3 | $0.00\pm 0.00$ | $-2.89\pm 0.25$ | $-1.92\pm 0.15$ | $2.84\pm 0.27$
whas | 64 | 4 | $0.00\pm 0.00$ | $3.19\pm 0.36$ | $0.35\pm 0.26$ | $-2.22\pm 0.43$
whas | 128 | 4 | $0.00\pm 0.00$ | $-0.40\pm 0.30$ | $-0.33\pm 0.19$ | $2.89\pm 0.36$
whas | 256 | 4 | $0.00\pm 0.00$ | $-1.09\pm 0.24$ | $-1.02\pm 0.19$ | $6.46\pm 0.35$
gsbg | 64 | 2 | $0.00\pm 0.00$ | $1.51\pm 0.07$ | $0.20\pm 0.06$ | $5.85\pm 0.06$
gsbg | 128 | 2 | $0.00\pm 0.00$ | $0.28\pm 0.07$ | $1.70\pm 0.05$ | $6.85\pm 0.08$
gsbg | 256 | 2 | $0.00\pm 0.00$ | $0.20\pm 0.06$ | $0.81\pm 0.05$ | $7.31\pm 0.08$
gsbg | 64 | 3 | $0.00\pm 0.00$ | $-0.07\pm 0.04$ | $-0.11\pm 0.04$ | $3.86\pm 0.04$
gsbg | 128 | 3 | $0.00\pm 0.00$ | $-0.81\pm 0.05$ | $-0.04\pm 0.05$ | $5.58\pm 0.05$
gsbg | 256 | 3 | $0.00\pm 0.00$ | $1.20\pm 0.06$ | $-1.20\pm 0.05$ | $5.67\pm 0.07$
gsbg | 64 | 4 | $0.00\pm 0.00$ | $0.12\pm 0.04$ | $0.06\pm 0.03$ | $1.49\pm 0.04$
gsbg | 128 | 4 | $0.00\pm 0.00$ | $0.90\pm 0.05$ | $0.81\pm 0.03$ | $3.00\pm 0.04$
gsbg | 256 | 4 | $0.00\pm 0.00$ | $1.36\pm 0.05$ | $0.28\pm 0.04$ | $5.08\pm 0.06$
support | 64 | 2 | $0.00\pm 0.00$ | $0.80\pm 0.02$ | $-0.08\pm 0.02$ | $5.81\pm 0.02$
support | 128 | 2 | $0.00\pm 0.00$ | $0.07\pm 0.02$ | $-0.27\pm 0.01$ | $5.98\pm 0.02$
support | 256 | 2 | $0.00\pm 0.00$ | $0.53\pm 0.02$ | $-0.71\pm 0.01$ | $5.83\pm 0.02$
support | 64 | 3 | $0.00\pm 0.00$ | $0.57\pm 0.02$ | $-0.42\pm 0.01$ | $4.56\pm 0.02$
support | 128 | 3 | $0.00\pm 0.00$ | $0.70\pm 0.02$ | $-0.53\pm 0.01$ | $4.55\pm 0.02$
support | 256 | 3 | $0.00\pm 0.00$ | $-0.03\pm 0.02$ | $-0.44\pm 0.01$ | $5.17\pm 0.02$
support | 64 | 4 | $0.00\pm 0.00$ | $-0.43\pm 0.02$ | $-0.46\pm 0.01$ | $2.67\pm 0.01$
support | 128 | 4 | $0.00\pm 0.00$ | $0.35\pm 0.02$ | $-0.79\pm 0.01$ | $4.04\pm 0.02$
support | 256 | 4 | $0.00\pm 0.00$ | $1.39\pm 0.02$ | $-0.18\pm 0.01$ | $4.30\pm 0.02$

Table 5: Relative decrease in the area under the curve (RD-AUC) compared to
the Random scoring function. A higher value in the table represents better
performance, with the best performance highlighted in bold.
Dataset | Hidden size | Layers | Random | Entropy | BALD | $\mathcal{C}\text{-BALD }$
---|---|---|---|---|---|---
Churn | 64 | 2 | $0.00\pm 0.00$ | $0.56\pm 0.19$ | $0.10\pm 0.17$ | $20.87\pm 0.47$
Churn | 128 | 2 | $0.00\pm 0.00$ | $4.88\pm 0.25$ | $1.25\pm 0.16$ | $18.79\pm 0.42$
Churn | 256 | 2 | $0.00\pm 0.00$ | $-0.32\pm 0.23$ | $-0.05\pm 0.11$ | $19.46\pm 0.50$
Churn | 64 | 3 | $0.00\pm 0.00$ | $1.49\pm 0.29$ | $0.38\pm 0.21$ | $29.77\pm 0.74$
Churn | 128 | 3 | $0.00\pm 0.00$ | $4.10\pm 0.30$ | $0.08\pm 0.20$ | $32.29\pm 0.81$
Churn | 256 | 3 | $0.00\pm 0.00$ | $5.40\pm 0.32$ | $0.24\pm 0.17$ | $30.57\pm 0.92$
Churn | 64 | 4 | $0.00\pm 0.00$ | $-0.66\pm 0.23$ | $0.66\pm 0.19$ | $29.17\pm 0.49$
Churn | 128 | 4 | $0.00\pm 0.00$ | $3.88\pm 0.26$ | $0.06\pm 0.16$ | $33.34\pm 0.64$
Churn | 256 | 4 | $0.00\pm 0.00$ | $2.12\pm 0.20$ | $0.30\pm 0.13$ | $35.96\pm 0.85$
Credit risk | 64 | 2 | $0.00\pm 0.00$ | $-0.48\pm 0.29$ | $-0.40\pm 0.29$ | $15.37\pm 0.45$
Credit risk | 128 | 2 | $0.00\pm 0.00$ | $-1.08\pm 0.28$ | $-0.15\pm 0.27$ | $16.87\pm 0.46$
Credit risk | 256 | 2 | $0.00\pm 0.00$ | $1.00\pm 0.25$ | $0.03\pm 0.21$ | $21.53\pm 0.45$
Credit risk | 64 | 3 | $0.00\pm 0.00$ | $-0.57\pm 0.32$ | $-0.47\pm 0.30$ | $20.21\pm 0.47$
Credit risk | 128 | 3 | $0.00\pm 0.00$ | $0.50\pm 0.32$ | $0.71\pm 0.28$ | $21.17\pm 0.51$
Credit risk | 256 | 3 | $0.00\pm 0.00$ | $0.99\pm 0.33$ | $0.16\pm 0.30$ | $23.53\pm 0.61$
Credit risk | 64 | 4 | $0.00\pm 0.00$ | $-0.48\pm 0.29$ | $1.26\pm 0.28$ | $17.60\pm 0.42$
Credit risk | 128 | 4 | $0.00\pm 0.00$ | $0.07\pm 0.28$ | $-0.27\pm 0.28$ | $18.38\pm 0.44$
Credit risk | 256 | 4 | $0.00\pm 0.00$ | $1.43\pm 0.28$ | $-0.21\pm 0.23$ | $19.40\pm 0.46$
Table 6: Relative decrease in the area under the curve (RD-AUC) compared to
the Random scoring function. A higher value in the table represents better
performance, with the best performance highlighted in bold.