# Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active
Learning

Wei Tan, Lan Du , Wray Buntine  Wei Tan and Lan Du are with Dept. of Data
Science and AI, Monash University, Clayton, Australia. E-mail:
<EMAIL_ADDRESS>Wray Buntine is now at VinUniversity, Hanoi, Vietnam.
Correspondence author: Lan Du. Manuscript received June 30th, 2022

###### Abstract

The effectiveness of active learning largely depends on the sampling
efficiency of the acquisition function. Expected Loss Reduction (ELR) focuses
on a Bayesian estimate of the reduction in classification error, and more
general costs fit in the same framework. We propose Bayesian Estimate of Mean
Proper Scores (BEMPS) to estimate the increase in strictly proper scores such
as log probability or negative mean square error within this framework. We
also prove convergence results for this general class of costs. To facilitate
better experimentation with the new acquisition functions, we develop a
complementary batch AL algorithm that encourages diversity in the vector of
expected changes in scores for unlabeled data. To allow high-performance
classifiers, we combine deep ensembles, and dynamic validation set
construction on pretrained models, and further speed up the ensemble process
with the idea of Monte Carlo Dropout. Extensive experiments on both texts and
images show that the use of mean square error and log probability with BEMPS
yields robust acquisition functions and well-calibrated classifiers, and
consistently outperforms the others tested. The advantages of BEMPS over the
others are further supported by a set of qualitative analyses, where we
visualise their sampling behaviour using data maps and t-SNE plots.

###### Index Terms:

active learning, machine learning, artificial intelligence, text
classification, image classification.

##  I Introduction

Classification is widely used and its performance has substantially improved
due to the advent of deep learning. However, a significant obstacle to its
utilization is the paucity of labeled or annotated data.  The data annotation
process performed by domain experts is often time-consuming, expensive and
tedious, especially in the medical field, due to the need for expertise
combined with privacy concerns [1]. Active Learning (AL) offers a solution by
judiciously selecting the most informative data points for annotation,
ensuring efficient use of a limited annotation budget [2]. The elegance of AL
lies in that it primarily involves an expert assigning a class label. Yet,
even for a simple task such as classification, a generally accepted theory for
AL remains elusive.

Creating an appropriate acquisition function is the central aspect of AL. In
the AL literature, acquisition functions are predominantly classified into two
major categories: uncertainty-based methods and diversity-based methods. The
former aims to select samples with the highest predictive uncertainty of a
model. This uncertainty can be measured by metrics such as the least confident
(LC) [3, 2], entropy [4, 2], mutual information [5] and other methods.
However, uncertainty-based methods are prone to adversarial examples [6] and
noisy datasets, often resulting in sampling redundant and even biased
instances, thus degrading the training efficiency and the model performance.
Conversely, diversity-based methods strive to acquire samples that best
capture the data distribution of the entire unlabeled set [7, 8]. Their
success largely depends on the representations of the unlabeled data used in
k-Means clustering. For example, ALPS and BERT-KM [7] compute surprisal
embeddings from the predictive word probabilities generated by a pretrained
language model. Even though existing methods exhibit promising performance in
different tasks, we argue that diversity alone without considering uncertainty
or vice versa is sub-optimal, particularly when the estimated uncertainty or
the computed embeddings have nothing to do with the model performance [9, 10].

Recently, some batch AL approaches focus on the uncertainty-diversity trade-
off with hybrid methods [10]. One notable method is BADGE [11], which uses the
gradient representation generated by the last layer of the model to
incorporate prediction uncertainty and sample diversity. While BADGE could
achieve a balance between uncertainty and diversity without requiring
adjustments to model hyperparameters, it faces the challenge of time-consuming
distance calculations in the gradient space [7]. Nevertheless, many hybrid
approaches prioritize diversifying samples through the data representation of
deep neural networks (DNNs), often neglecting prediction uncertainty.

In formulating AL systems using DNNs, especially in real-world scenarios, it
is essential to consider a comprehensive list of factors in a holistic manner.
For instance, one should account for the costs involved in both retraining a
deep neural network (DNN) and using validation data for training [12]. The
role played by transformer language models [13] is also essential, as is
understanding the intricacies of batch mode AL, especially with an emphasis on
diversity [10]. Moreover, one shouldn’t overlook the necessary expertise and
associated professional costs [14, 15]. When designing experiments, it’s vital
to consider the feasible size of the validation set. In many real-world
scenarios, practitioners can be constrained to an expert annotation budget of,
for instance, 1000 data samples, due to the inherent cost of experts. This
limitation suggests that larger validation set sizes, as observed in [11, 16],
might not always be feasible.

With these desiderata, we propose a new set of acquisition functions based on
the principle of strictly proper scoring rules and develop a hybrid approach
that considers both uncertainty and diversity, for effective batch active
learning using DNNs. As our first step, frameworks for cost reduction due to
data acqusition are the expected value of sample information (EVSI) [17] and
similarly mean objective cost of uncertainty (MOCU) [18]. We develop our AL
model using strictly proper scoring rules or Bregman divergences [19] within
these frameworks. MOCU applied to errors required modifications of the
formula, resulting in WMOCU [20] and SMOCU [21], in order to achieve
convergence and avoid getting stuck in error bands.  In contrast, strictly
proper scoring rules avoid this problem by utilizing expected scores generated
by strictly convex functions. The convexity of the scoring functions
guarantees convergence of AL. The scoring rules can be adapted to different
inference tasks (e.g., different utilities, precision-recall trade-offs,
etc.). This property is preferable and beneficial for applications such as
medical domains where measures other than errors are relevant for an inference
task.

This work represents a substantial extension of our previous NeurIPS 2021
conference paper BEMPS [22], where we developed a family of novel acquisition
functions based on strictly proper scoring rules for categorical variables,
which generalize existing functions based on ELR and BALD. It was then
instantiated with two scoring functions, namely CoreMSE and CoreLog. The
contributions of this extension are summarised as follows:

  1. 1.

Theoretical properties of BEMPS: We provide a detailed proof demonstrating
that active learning with the Bayesian estimate of mean strictly proper scores
guarantees convergence. Moreover, we show that our acquisition function
measures epistemic uncertainty, not aleatoric uncertainty.

  2. 2.

Comprehensive quantitative experimental results: Besides the results reported
in [22], we further validate BEMPS’ performance on the image classification
task, using two benchmark datasets, namely, MINST and CIFAR10. Additionally,
we perform an analysis of model calibration, measured by the expected
calibration error. This also lets us explore the computational properties of
our methods and two different ensembling techniques, deep ensembles and Monte-
Carlo dropout.

  3. 3.

Qualitative analysis of sampling behaviors: We further employ visualization
techniques (i.e., Data Maps [23] and t-SNE [24] ) to delve into the sampling
behavior of different AL methods, in order to gain insight into the underlying
mechanism that drives the acquisition of samples from different regions (i.e.,
the easy-to-learn, the ambiguous and the hard-to-learn regions) and along the
decision boundaries within the sample space.

##  II Related Work

Active learning has been researched extensively since the emergence of deep
learning, generating significant interest. As elucidated by previous
studies[2, 10, 9, 20, 21, 25, 26], there is no effective universal AL method.
Instead, researchers often rely on heuristics tailored to their specific
tasks. In this context, we review existing acquisition functions (aka query
strategies) proposed in some recent AL models that are most related to ours.
These models cover a spectrum of perspectives, including uncertainty-based,
diversity-based and hybrid methods. Readers interested in a thorough discourse
on contemporary strategies are directed to [10] for a comprehensive
discussion.

Uncertainty-based methods: The proposed BEMPS [22], as a general Bayesian
model for acquisition functions, quantifies the model uncertainty using the
theory of (strictly) proper scoring rules for categorical variables [27].
Thus, we first review existing acquisition functions proposed in some recent
uncertainty-based AL models that are most related to ours. One common and
straightforward heuristic often used in AL is maximum entropy [28, 29, 30,
31], where one chooses samples that maximize the predictive entropy.
Similarly, Bayesian AL by disagreement (BALD) [32] and its batch version
(i.e., BatchBALD) [5] instead compute the mutual information between the model
predictions and the model parameters, which indeed chooses samples that
maximize the decrease in expected entropy [16]. Improving on ELR [33], MOCU
with minimum errors was extended by WMOCU using weights [20], also obtaining a
theoretical guarantee of convergence. WMOCU directly minimizes the objective
uncertainty impacting classification by tuning a hyperparameter of the
weighting function.  Published alongside WMOCU, SMOCU [21] achieves
convergence through the utilization of a log-sum-exp function to approximate
the max function in computing the OBC error. This adaptation renders the MOCU
function strictly concave. Without modifying the EVSI formula, BEMPS uses
proper scores directly in the expected value of evidence.  This requires no
additional manipulation of the formula needed by the MOCU-based methods to
achieve convergence.

Diversity-based and hybrid methods: Query strategies considering just
uncertainty do not work well in a batch setting because similar samples can be
acquired in one batch [34, 10, 35]. To overcome this problem, there have been
many AL methods that achieve batch diversity by acquiring samples that are
both informative and diverse, such as [36, 37, 38, 11, 7, 15, 39]. For
instance, BADGE [11] and ALPS [7] are the two recent AL methods focusing on
batch diversity. BADGE uses gradient embeddings of unlabeled samples as inputs
of $k$-MEANS++ to select a set of diverse samples, which relies on fine-tuning
pretrained models. Inspired by BADGE, ALPS uses surprisal embeddings computed
from the predictive word probabilities generated by a masked language model
for cold-start AL. Whereas our BEMPS computes an embedding vector for a
candidate sample data by the expected change in the proper scores induced by
acquiring a label for the data. This is a vector because it is computed for
each unlabeled sample in a pool, answering the question “how are proper scores
expected to change across the pool if we acquire this one label?”

Ensemble methods: Ensemble methods combine multiple distinct models to improve
generalization performance. They are now widely utilized in machine learning
[40, 41]. For example, [42, 43] uses ensembles to estimate the uncertainty of
DNN predictions in the context of outlier detection and reinforcement
learning. In recent years, ensembles are also used to obtain better
uncertainty estimates with deep learning, including deep ensembles [42] and
Monte-Carlo dropout (MC-Dropout) [44, 5, 6]. Deep ensembles combine the
benefits of both deep learning and ensemble learning to produce a model with
superior generalization performance and uncertainty estimation, but they are
very expensive to train and evaluate. MC-Dropout is a well-known ensemble
method that is less costly but less reliable [45]. BatchBALD utilizes the MC-
dropout technique to approximate inference within Bayesian Neural Networks
(BNNs). This approach aids in estimating uncertainty, enabling effective
scalability to handle high-dimensional image inputs.

Validation setup: In experiments, using a validation set to train deep
learning models in active learning is not uncommon, for instance, for early-
stopping. Some existing methods assume that there is a large validation set
available a priori [11, 5, 31], which means the cost of labeling the
validation set is not factored into the labeling budget. We argue that the
availability of a separate validation set is impractical in real-world AL
scenarios. Although [7] uses fixed epochs to train the classifier without a
validation set to save the cost, the classifier could either be under-fit or
over-fit. We instead use a dynamic approach to generate alternative validation
sets from the ever-increasing labeled pool after each iteration.

##  III Bayesian Estimate of Mean Proper Scores

We first review the general Bayesian model for acquisition functions,
including ELR, MOCU and BALD; and provide the preliminaries of proper scoring
rules. We then develop BEMPS a new uncertainty quantification framework with a
theoretical foundation based on strictly proper scoring rules [27].

###  III-A Expected Loss Reduction

The ELR method attempts to quantify the generalization error that is expected
to decrease when one new sample is added to the labeled dataset.  MOCU [18] is
similar to a broader category of scoring systems known as the value of
information [17]. Most relevant for us is the concept of the expected value of
sample information (EVSI) [17]. This concept explores how the cost measure can
be anticipated to improve upon acquiring a single sample point. Our distinct
contribution lies in adopting a proper scoring rule as the designated cost
metric within the EVSI framework.

TABLE I: Notation Notation |  Description
---|---
$\mathbf{L}$ |  a set of labeled data
$\mathbf{U}$ |  a set of unlabeled data
$\mathbf{X}$ |  a set of unlabeled data used to estimate the expected score change
$\mathbf{x}$ |  a unlabeled sample from $U$
$y$ |  the class label
$\mathbf{x}^{\prime}$ |  a unlabeled sample sampled from $X$
${\bf\it\theta}$ |  the model parameters
$\mathbf{\Theta}$ |  a set of model parameters
$\mathbb{E}_{\mu}[f(\mu,\nu)]$ |  the expectation of a function $f(\mu,\nu)$ with respect to a random variable $\mu$
$p(y|{\bf\it\theta},\mathbf{x})$ |  the probability of label $y$ for an instance $x$ on running model ${\bf\it\theta}$
$p({\bf\it\theta}|\mathbf{L})$ |  the probability of ${\bf\it\theta}$ based on learning from $\mathbf{L}$
$p({\bf\it\theta}|\mathbf{L},(\mathbf{x},y))$ |  the probability of ${\bf\it\theta}$ based on the learning from $\mathbf{L}$ and an additional labeled instance $(\mathbf{x},y)$
$p(y|\mathbf{L},\mathbf{x})$ |  the Bayesian optimal estimate of label $y$ based on ${\bf\it\theta}$ after learning $p({\bf\it\theta}|\mathbf{L})$, i.e., ${\mathbb{E}}_{p({\bf\it\theta}|\mathbf{L})}\left[p(y|{\bf\it\theta},\mathbf{x})\right]$
$Q(\cdot)$ |  the score that measures the quality of the posterior of a model
$\Delta Q(\mathbf{x}|\mathbf{L})$ |  the expected score change by acquiring the label of $\mathbf{x}$, i.e., the acquisition function in AL
$S(\cdot)$ |  a strictly proper scoring function
$G(\cdot)$ |  a strictly convex function computed as the expectation of $S(\cdot)$

Suppose models of our interest are parameterised by parameters
${\bf\it\theta}\in\mathbf{\Theta}$, $\mathbf{L}$ indicates labeled data,
probability of label $y$ for data $\mathbf{x}$ is given by
$p(y|{\bf\it\theta},\mathbf{x})$, and $p(\cdot|{\bf\it\theta},\mathbf{x})$
presents a vector of label probabilities. With a fully conditional model, the
posterior of ${\bf\it\theta}$ is unaffected by unlabeled data, which means
$p({\bf\it\theta}|\mathbf{L},\mathbf{U})=p({\bf\it\theta}|\mathbf{L})$ for any
unlabeled data $\mathbf{U}$. Moreover, we assume without loss of generality
that this model family is well-behaved in a statistical sense, so the model is
identifiable. The “cost” of the posterior $p({\bf\it\theta}|\mathbf{L})$ can
be measured by some functional $Q(p({\bf\it\theta}|\mathbf{L}))$, denoted
$Q(\mathbf{L})$ for short, where $Q(\mathbf{L})\geq 0$ and $Q(\mathbf{L})=0$
when some convergence objective has been achieved. For our model, this is when
$p({\bf\it\theta}|\mathbf{L})$ has converged to a point mass at a single
model. A suitable objective function is to measure the expected decrease in
$Q(\cdot)$ due to acquiring the label for a data point $\mathbf{x}$. The
corresponding acquisition function for AL is formulated [32, Eq (1)], [21, Eq
(3)] as

$\Delta
Q(\mathbf{x}|\mathbf{L})=Q(\mathbf{L})-\mathbb{E}_{p(y|\mathbf{L},\mathbf{x})}\big{[}Q(\mathbf{L}\cup\\{(\mathbf{x},y)\\})\big{]},$
(1)

whereas for ELR the expression is split over an inequality sign [33, Eq (2)].
It estimates how much the cost is expected to reduce when a new data point $x$
is acquired. Since the true label for the new data $\mathbf{x}$ is unknown a
prior, we have to use expected posterior proportions from our model,
$p(y|\mathbf{L},\mathbf{x})$ to estimate the likely label. For BALD [32] using
Shannon’s entropy, $Q_{I}(\mathbf{L})=I(p({\bf\it\theta}|\mathbf{L}))$, which
measures uncertainty in the parameter space and thus has no strong
relationship to actual errors [20]. MOCU[20] is equivalent to ELR that uses a
Bayesian regret given by the expected loss difference between the optimal
Bayesian classifier and the optimal classifier:

$\displaystyle Q_{MOCU}(\mathbf{L})=$ (2)
$\displaystyle\mathbb{E}_{p(\mathbf{x})}\Big{[}\min_{y}(1-p(y|\mathbf{L},\mathbf{x}))-\mathbb{E}_{p(\theta|\mathbf{L})}\big{[}\min_{y}(1-p(y|{\bf\it\theta},\mathbf{x}))\big{]}\Big{]}.$

WMOCU uses a weighting function defined by Eq (11) in [20] to have a more
amenable definition of $\Delta Q(\mathbf{x|L})$ than the MOCU method. Although
WMOCU guarantees $\Delta Q(\mathbf{x|L})$ converging to the optimal classifier
(under minimum errors) according to the $Q(\mathbf{L})$ with the strictly
concave function by Eq (15) in [20], the optimal approximation of the
convergences can only be solved by controlling a hyperparameter of the
weighting function manually. To allow theoretical guarantees of convergence
under more general loss functions, we propose a different definition for
$Q(\mathbf{L})$ based on strictly proper scoring rules.

###  III-B Proper Scoring Rules

Proper scoring rules assess the quality of probabilistic forecasts and offer a
measure of predictive uncertainty. They assign a numerical score based on the
predictive distribution and the predictive outcome, favouring more accurate
and calibrated forecasts over less accurate ones. We focus on scoring
functions in which a higher numeric score indicates a better performance. In
this paper, we are interested in the quality of the probabilistic prediction
of categorical variables, where proper scoring rules are often used in
training a classification algorithm. Readers interested in a more detailed
discussion of proper scoring cores can refer to [46, 47, 48, 49, 27].

Let us consider the prediction of a categorical variable $y$ with the sample
space $\Omega=\\{0,\dots,K-1\\}$. In this context, a scoring function
$S(p(\cdot|{\bf\it\theta},\mathbf{x}),y)$ evaluates the quality of the
predictive distribution $p(\cdot|{\bf\it\theta},\mathbf{x})$ over $\Omega$
relative to an observed event $y|\mathbf{x}\sim q(y|\mathbf{x})$ where
$q(\cdot)$ denotes the true distribution on $(y,\mathbf{x})$, which $p(\cdot)$
tries to estimate. Drawing upon the insights of [46] and [27], it becomes
apparent that a scoring rule is intricately linked to a tangent line of a
convex, real-valued function $G(p(\cdot))$ at $p(\cdot)$ evaluated at
$q(\cdot)$. The derivation of the scoring rule pertaining to an observed event
and the expected scoring rule follows in the subsequent manner:

$\displaystyle S(p(\cdot),y)$ $\displaystyle=$ $\displaystyle
G(p(\cdot))+\langle\gradient{G(p(\cdot))},(\delta_{y}-p(\cdot))\rangle\,,$ (3)
$\displaystyle S(p(\cdot),q(\cdot))$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{y\sim q(\cdot)}\left[S(p(\cdot),y)\right]$ (4)
$\displaystyle=$ $\displaystyle
G(p(\cdot))+\langle\gradient{G(p(\cdot))},\,(q(\cdot)-p(\cdot))\rangle\,,$ (5)

where $\gradient{G(p(\cdot))}$ is a subgradient of $G(\cdot)$ at the point
$p(\cdot)$. Note that the expectation of a scoring rule according to the
supplied probability $p(\cdot)$ takes a simple form $G(p(\cdot))$, the proof
of which is straightforward by replacing $y\sim q(\cdot)$ with $y\sim
p(\cdot)$ in Eq (4); and the second term in Eq (5) will be zero.

The scoring rule is proper if $S(q(\cdot),q(\cdot))\geq S(p(\cdot),q(\cdot))$
for all $p(\cdot),q(\cdot)\in\mathcal{P}$111$\mathcal{P}$ is a convex set of
probability measures on $(\mathcal{A},\Omega)$, where $\mathcal{A}$ is a
$\sigma$-algebra of subsets of the sample space $\Omega$ (i.e., classes in
classification)., and it is strictly proper if the expected score is minimized
if and only if $p(\cdot)=q(\cdot)$ [27]. With $G(\cdot)$ being a convex
function, the proof of the inequality is derived as follows:

$\displaystyle S(q(\cdot),q(\cdot))$ $\displaystyle=$ $\displaystyle
G(q(\cdot))$ (6) $\displaystyle\geq$ $\displaystyle
G(p(\cdot))+\langle\gradient{G(p(\cdot))},\,(q(\cdot)-p(\cdot))\rangle$
$\displaystyle=$ $\displaystyle S(p(\cdot),q(\cdot))\,,$

where the inequality holds because $G(\cdot)$ is a convex function that is
always above its tangent line computed at $p(\cdot)$. In the case where
$G(\cdot)$ exhibits strict convexity, the previously outlined proper scoring
rule transitions into a strictly proper scoring rule, as indicated by [27].

A strictly proper scoring rule has the behaviour that in the limit of infinite
labeled data $\mathbf{L}_{n}$, as $n\rightarrow\infty$, the average score
$\frac{1}{n}\sum_{(\mathbf{x},y)\in\mathbf{L}_{n}}S(p(\cdot|{\bf\it\theta},\mathbf{x}),y)$
has a unique maximum for ${\bf\it\theta}$ at the “true” model (for our
identifiable model family).  With these characteristics, one can tune
$G(\cdot)$ for different tasks [48, 49]. For instance, [42] used Gibbs
inequality to prove that the scoring function corresponding to maximizing
likelihood is indeed a proper scoring rule. In the following sections, we will
develop two acquisition functions using the Brier score and the logarithmic
score that correspond to the mean square error and the Kullback–Leibler
divergence.

###  III-C Strictly Proper Scores for Active Learning

With strictly proper scoring rules, we develop a generalized class of
acquisition functions built using the posterior (i.e., w.r.t.
$p({\bf\it\theta}|\mathbf{L})$) expected difference between the score for the
Bayes optimal classifier and the score for the “true” model. This is
inherently Bayesian due to the use of $p({\bf\it\theta}|\mathbf{L})$.

$\displaystyle\begin{split}&Q_{S}(\mathbf{L})=\\\
&\,\,\,\mathbb{E}_{p(\mathbf{x})p({\bf\it\theta}|\mathbf{L})}\Big{[}\mathbb{E}_{p(y|{\bf\it\theta},\mathbf{x})}\big{[}S(p(\cdot|{\bf\it\theta},\mathbf{x}),y)-S(p(\cdot|\mathbf{L,x}),y)\big{]}\Big{]}\end{split},$
(7) $\displaystyle
Q_{S}(\mathbf{L})=\mathbb{E}_{p(\mathbf{x})p({\bf\it\theta}|\mathbf{L})}\big{[}B(p(\cdot|\mathbf{L,x}),p(\cdot|{\bf\it\theta},\mathbf{x}))\big{]},$
(8) $\displaystyle
Q_{S}(\mathbf{L})=\mathbb{E}_{p(\mathbf{x})}\Big{[}\mathbb{E}_{p({\bf\it\theta}|\mathbf{L})}\big{[}G(p(\cdot|{\bf\it\theta},\mathbf{x}))\big{]}-G(p(\cdot|\mathbf{L,x}))\Big{]},$
(9) $\displaystyle\begin{split}&\Delta Q_{S}(\mathbf{x|L})=\\\
&\,\,\,\mathbb{E}_{p(\mathbf{x^{\prime}})}\Big{[}\mathbb{E}_{p(y|\mathbf{L,x})}\big{[}G(p(\cdot|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}}))\big{]}-G(p(\cdot|\mathbf{L,x^{\prime}}))\Big{]}\end{split},$
(10)

where $\mathbf{x}^{\prime}$ is a unlabeled sample, different from
$\mathbf{x}$, which is used to estimate the expected score change $\Delta
Q_{x}(\mathbf{x}|\mathbf{L})$.

The $Q_{S}(\mathbf{L})$ has three equivalent variations, one for an arbitrary
strictly proper scoring rule $S(q(\cdot),y)$ (Eq (7)), one for a corresponding
Bregman divergence $B(\cdot,\cdot)$ (Eq (8)) and the third for an arbitrary
strictly convex function $G(\cdot)$ (Eq (9)). Their connections are given in
[27]. The form for $\Delta Q_{S}(\mathbf{x|L})$ corresponds to [21, Equation
(6)]. using the same simplification.

######  Lemma III.1 (Properties of scoring).

In the context of a fully conditional classification model
$p(y|{\bf\it\theta},\mathbf{x})$, the $Q_{I}(\mathbf{L})$,
$Q_{S}(\mathbf{L})$, $\Delta Q_{I}(\mathbf{x|L})$, $\Delta
Q_{S}(\mathbf{x|L})$ as defined above are all non-negative.

###### Proof.

$Q_{I}(\mathbf{L})\geq 0$ by definition of entropy. Now an identity for
entropy is that
$\mbox{I}\left(p(A|B,C)\right)\leq\mbox{I}\left(p(A|B)\right)$, which means
given more evidence $C$, the conditional entropy of $A$ cannot increase. So
the decrease in log volume is never negative. This means $\Delta
Q_{I}(\mathbf{x|L})\geq 0$.  The result for $Q_{S}(\mathbf{L})$ follows
directly by the definition of a scoring rule. The expression
$S(q(\cdot),q(\cdot))-S(p(\cdot),q(\cdot))$ can be matched inside the outer
square brackets of Eq (7), and it is non-negative for a proper scoring rule.

For $\Delta Q_{S}(\mathbf{x|L})$ we work as follows. Start with Equation 10
and reverse back in the scores:

$\displaystyle\mathbb{E}_{p(y|\mathbf{L,x})}\Big{[}\mathbb{E}_{p(\mathbf{x^{\prime}})p(y^{\prime}|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}})}\big{[}S(p(\cdot|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}}),y^{\prime})\big{]}\Big{]}$
$\displaystyle\quad-\mathbb{E}_{p(\mathbf{x^{\prime}})p(y^{\prime}|\mathbf{L,x^{\prime}})}\big{[}S(p(\cdot|\mathbf{L,x^{\prime}}),y^{\prime})\big{]}$
$\displaystyle=\mathbb{E}_{p(y|\mathbf{L,x})}\Big{[}\mathbb{E}_{p(\mathbf{x^{\prime}})p(y^{\prime}|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}})}\big{[}S(p(\cdot|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}}),y^{\prime})$
$\displaystyle\quad-S(p(\cdot|\mathbf{L},\mathbf{x^{\prime}}),y^{\prime})\big{]}\Big{]}$

where the second line is done by changing
$\mathbb{E}_{p(y^{\prime}|\mathbf{L,x^{\prime}})}[\cdot]$ to
$\mathbb{E}_{p(y^{\prime}|\mathbf{L,x,x^{\prime}})}[\cdot]$ (the model is
fully conditional), then to
$\mathbb{E}_{p(y,y^{\prime}|\mathbf{L,x,x^{\prime}})}[\cdot]$ and rearranging.
The second line is $\geq 0$ due to the maximum properties of scoring functions
used earlier. Moreover, they guarantee learning will converge to the “truth”
as follow. ∎

######  Theorem III.2 (Convergence of active learning).

We have a fully conditional classification model
$p(y|{\bf\it\theta},\mathbf{x})$, for ${\bf\it\theta}\in\mathbf{\Theta}$ with
finite discrete classes $y$ and input features $\mathbf{x}$. Moreover, there
is a unique “true” model parameter ${\bf\it\theta}_{r}$ with which the data is
generated, the prior distribution $p({\bf\it\theta})$ satisfies
$p({\bf\it\theta}_{r})>0$, and the model is identifiable. After being applied
for $n$ steps, the AL algorithm with the acquisition functions defined above
(i.e., $\Delta Q_{I}(\mathbf{x|L})$ or $\Delta Q_{S}(\mathbf{x|L})$) gives
labeled data $\mathbf{L}_{n}$, then $\lim_{n\rightarrow\infty}\Delta
Q_{I}(\mathbf{x|L}_{n})=0$ and likewise for $Q_{S}(\cdot)$. Moreover,
$\lim_{n\rightarrow\infty}p({\bf\it\theta}|\mathbf{L}_{n})$ is a delta
function at ${\bf\it\theta}={\bf\it\theta}_{r}$ for data acquired by both
$\Delta Q_{I}(\mathbf{x|L})$ or $\Delta Q_{S}(\mathbf{x|L})$.

###### Proof.

The proof for Lemma 5 in [20] can be readily adapted to show for $\mathbf{x}$
occurring infinitely often in $\mathbf{L}_{n}$ and $n\rightarrow\infty$
$\Delta Q_{I}(\mathbf{x|L}_{n})$ and $\Delta Q_{S}(\mathbf{x|L}_{n})$ both
approach zero as $n\rightarrow\infty$ since
$Q(\mathbf{L}\cup\\{(\mathbf{x},y)\\})\rightarrow Q(\mathbf{L})$ when
$p({\bf\it\theta}|\mathbf{L}\cup\\{(\mathbf{x},y)\\})\rightarrow
p({\bf\it\theta}|\mathbf{L})$. Then one adapts the proof of Theorem 1 in [20],
which requires finiteness and discreteness of $\mathbf{x}$, to show that
$\Delta Q_{I}(\mathbf{x|L}_{n})$ and $\Delta Q_{S}(\mathbf{x|L}_{n})$ both
approach zero as $n\rightarrow\infty$ for all $\mathbf{x}$.

Now consider $\Delta Q_{I}(\mathbf{x|L}_{n})$ which by properties of the KL
function is equal to
$\mathbb{E}_{p({\bf\it\theta}|\mathbf{L}_{n})}[\mbox{KL}(p(y|{\bf\it\theta},\mathbf{x})||p(y|\mathbf{L}_{n},\mathbf{x}))]$.
Let

$\Theta_{NZ}=\left\\{{\bf\it\theta}\,:\,\left(\lim_{n\rightarrow\infty}p({\bf\it\theta}|\mathbf{L}_{n})\right)>0\right\\}$
(11)

Now ${\bf\it\theta}_{r}\in\mathbf{\Theta}_{NZ}$ due to the arguments of
Theorem 1 in [20]. From the KL approaching zero, it follows that for all $x$
and as $n\rightarrow\infty$, $p(y|{\bf\it\theta},\mathbf{x})$ approaches
$p(y|\mathbf{L}_{n},\mathbf{x})$ for all
${\bf\it\theta}\in\mathbf{\Theta}_{NZ}$. This means that all
${\bf\it\theta}\in\mathbf{\Theta}_{NZ}$ yield identical
$p(y|{\bf\it\theta},\mathbf{x})$. Since the model is identifiable,
$\mathbf{\Theta}_{NZ}$ only has one element, ${\bf\it\theta}_{r}$.

Suppose $\Delta Q_{S}(\mathbf{x|L}_{n})\rightarrow 0$ as $n\rightarrow\infty$.
Considering the final equation from the proof of Lemma III.1, since
$\mathbb{E}_{p(y^{\prime}|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}})}[S(p(\cdot|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}}),y^{\prime})-S(p(\cdot|\mathbf{L}_{n},\mathbf{x^{\prime}}),y^{\prime})]\geq
0$ from properties of the proper scoring rule, it follows that
$\mathbb{E}_{p(y|\mathbf{L}_{n},\mathbf{x})p(y^{\prime}|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}})}[S(p(\cdot|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}}),y^{\prime})-S(p(\cdot|\mathbf{L}_{n},\mathbf{x^{\prime}}),y^{\prime})]$
approaches 0 for all $\mathbf{x}$, $\mathbf{x^{\prime}}$. Substitute the
result from Savage about strictly proper scoring rules into the above
simplification:

$\displaystyle\mathbb{E}_{p(y|\mathbf{L}_{n},\mathbf{x})p(y^{\prime}|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}})}\big{[}S(p(\cdot|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}}),y^{\prime})$
$\displaystyle\quad-S(p(\cdot|\mathbf{L}_{n},\mathbf{x^{\prime}}),y^{\prime})\big{]}$
(12)
$\displaystyle=\mathbb{E}_{p(y|\mathbf{L}_{n},\mathbf{x})}\big{[}G(p(\cdot|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}}))\big{]}-G(p(\cdot|\mathbf{L}_{n},\mathbf{x^{\prime}}))~{},$

which by above must mean $\rightarrow 0$ as $n\rightarrow\infty$. Because

$\displaystyle\mathbb{E}_{p(y|\mathbf{L}_{n},\mathbf{x})}\big{[}p(y^{\prime}|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}})\big{]}=p(y^{\prime}|\mathbf{L}_{n},\mathbf{x^{\prime}})$

and $G(\cdot)$ is stricly convex, it must mean that
$p(y^{\prime}|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x^{\prime}})$ approaches
$p(y^{\prime}|\mathbf{L}_{n},\mathbf{x^{\prime}})$ for all $y,y^{\prime}$ and
$\mathbf{x,x^{\prime}}$ as $n\rightarrow\infty$.

Now consider as $n\rightarrow\infty$,

$\displaystyle\mathbb{E}_{p({\bf\it\theta}|\mathbf{L}_{n})}\big{[}p(y|{\bf\it\theta},\mathbf{x})^{2}\big{]}$
(13)
$\displaystyle~{}~{}~{}=p(y|\mathbf{L}_{n},\mathbf{x})\int_{\theta}\frac{p({\bf\it\theta}|\mathbf{L}_{n})p(y|{\bf\it\theta},\mathbf{x})}{p(y|\mathbf{L}_{n},\mathbf{x})}p(y|{\bf\it\theta},\mathbf{x})\textrm{d}\theta$
$\displaystyle~{}~{}~{}=p(y|\mathbf{L}_{n},\mathbf{x})p(y|\mathbf{L}_{n},(\mathbf{x},y),\mathbf{x})$
$\displaystyle~{}~{}~{}\rightarrow p(y|\mathbf{L}_{n},\mathbf{x})^{2}$

and thus the variance of $p(y|{\bf\it\theta},\mathbf{x})$ w.r.t.
$p({\bf\it\theta}|\mathbf{L}_{n})$ approaches zero for all $\mathbf{x},y$.
Therefore, $\lim_{n\rightarrow\infty}p({\bf\it\theta}|\mathbf{L}_{n})$ must be
non-zero on a set of ${\bf\it\theta}$ yielding identical
$p(y|{\bf\it\theta},\mathbf{x})=\lim_{n\rightarrow\infty}p(y|\mathbf{L}_{n},\mathbf{x}).$
Again, there must be a unique $\theta$ with non-zero limit using
identifability. ∎

Finiteness and discreteness of $\mathbf{x}$ is used to adapt results from [20]
to show for all $\mathbf{x}$ that $\Delta Q(\mathbf{x|L}_{n})\rightarrow 0$ as
$n\rightarrow\infty$, not an issue since real data is always finite.
Interestingly $\Delta Q_{I}(\mathbf{x|L})$, i.e., BALD, achieves convergence
too, which occurs because the model is identifiable and fully conditional,
during AL we are free to choose $\mathbf{x}$ values that would distinguish
different parameter values ${\bf\it\theta}$. Full conditionality also supports
BEMPS because it means any inherent bias in the AL selection is nullified with
the use of the data distribution $p(\mathbf{x})$. But it also means that the
theory has not been shown to hold for semi-supervised learning algorithms,
where full conditionality does not apply.

Compared with BALD, MOCU and WMOCU, the advantage of using strictly proper
scoring rules in BEMPS is that, as mentioned in Section III-B, the expected
scores can be tailored for different inference tasks. BALD has its quality
$Q_{I}(\cdot)$ as the certainty (negative entropy) of the model’s parameter
space. This does not support AL when the uncertainty of some parameters does
not strongly influence classification performance.  This is reflected by its
poor performance in our experiments. MOCU however has convergence issues as
ELR, as pointed out by [20], which were overcome by WMOCU and SMOCU by either
augmenting or manipulating the scoring functions.

###  III-D Scoring functions: CoreMSE and CoreLog

Scoring rules promote the quality of predictive distributions by rewarding
calibrated predictive distributions. For example, scoring rules can be
developed [27] for some different inference tasks, including Brier score,
logarithmic score, the beta family, etc. Many loss functions used by neural
networks, like cross-entropy loss, are indeed strictly proper scoring rules
[42].

It is noteworthy that the acquisition function $\Delta Q_{S}(\mathbf{x|L})$
defined in Eq (10) is in a general form, applicable to any strictly proper
scoring function for categorical variables. For instance, using a logarithmic
scoring rule, we have $S_{log}(p(\cdot),y)=\log p(y)$ and
$G_{log}(p(\cdot))=-I(p(\cdot))$. The corresponding materialization of Eq (7)
is

$\displaystyle Q_{CoreLog}(\mathbf{L})$
$\displaystyle={\mathbb{E}}_{p(\mathbf{x})p({\bf\it\theta}|\mathbf{L})}\left[KL(p(y|{\bf\it\theta},\mathbf{x})\;\|\;p(y|\mathbf{L,x}))\right]$
(14)

Similarly, using the squared error scoring rule, known as a Brier score, we
have
$S_{MSE}(p(\cdot),y)=-\sum_{\hat{y}}\left(p(\hat{y})-1_{y=\hat{y}}\right)^{2}$
and $G_{MSE}(p(\cdot))=\sum_{y}p(y)^{2}-1$. The definition of $Q(\mathbf{L})$
in Eq (7) is

$\displaystyle Q_{CoreMSE}(\mathbf{L})$
$\displaystyle={\mathbb{E}}_{p(\mathbf{x})p({\bf\it\theta}|\mathbf{L})}\left[\sum_{y}\big{(}p(y|{\bf\it\theta},\mathbf{x})-p(y|\mathbf{L,x})\big{)}^{2}\right]$
(15)

Algorithm 1 Estimating point-wise $\Delta Q(\mathbf{x|L})$ with Equation (10)

1:unlabeled data point $\mathbf{x}$, existing labeled data $\mathbf{L}$,
estimation point $\mathbf{x^{\prime}}$

2:model/network ensemble
$\mathbf{\Theta}=\\{{\bf\it\theta}_{1},...,{\bf\it\theta}_{E}\\}$ built from
labeled data $\mathbf{L}$,

3:strictly convex function $G(\cdot)$ taking as input a probability density
over $y$

4:$Q=0$

5:$qx(\cdot)=\sum_{{\bf\it\theta}\in\mathbf{\Theta}}p({\bf\it\theta}|\mathbf{L})p(\cdot|{\bf\it\theta},\mathbf{x})$

6:for $y$ do

7:
$q(\cdot)=\sum_{{\bf\it\theta}\in\mathbf{\Theta}}p({\bf\it\theta}|\mathbf{L},(\mathbf{x},y))p(\cdot|{\bf\it\theta},\mathbf{x^{\prime}})$

8: $Q~{}+\\!\\!=qx(y)G(q(\cdot))$

9:$q(\cdot)=\sum_{{\bf\it\theta}\in\mathbf{\Theta}}p({\bf\it\theta}|\mathbf{L})p(\cdot|{\bf\it\theta},\mathbf{x^{\prime}})$

10:$Q~{}-\\!\\!=G(q(\cdot))$

11:return $Q$

Algorithm 2 Estimate of
$\operatorname*{arg\,max}_{\mathbf{x}\in\mathbf{U}}\Delta Q(\mathbf{x|L})$

1:unlabeled pool $\mathbf{U}$, estimation pool $\mathbf{X}$

2:for $\mathbf{x}\in\mathbf{U}$ do

3: $Q_{\mathbf{x}}=0$

4: for $\mathbf{x^{\prime}}\in\mathbf{X}$ do

5: $Q_{\mathbf{x}}~{}+\\!\\!=\Delta Q(\mathbf{x|L,x^{\prime}})$

6:return $\operatorname*{arg\,max}_{\mathbf{x}\in\mathbf{U}}Q_{\mathbf{x}}$

Viewed above, minimising Brier score gets the probability right in a least
squares sense, i.e., minimising the squared error between the predictive
probability and the one-shot label representation, which pays less attention
to very low probability events. Meanwhile, log probability gets the
probability scales right, paying attention to all events. In most cases, we
can create a particular model to match just about any Bregman divergence
(e.g., minimum squared errors is a Gaussian). In practice, we can also use
robust models (e.g., a Dirichlet-multinomial rather than a multinomial, a
negative binomial rather than a Poisson, Cauchy rather than Gaussian) in our
log probability. Combining the two $G(\cdot)$ functions above with Equation
(10) yields two acquisition functions for the different scoring rules.

$\displaystyle\Delta Q_{CoreMSE}(\mathbf{x}|\mathbf{L})$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{p(\mathbf{x^{\prime}})p(y|\mathbf{L,x})}\left[\sum_{y^{\prime}}\left(p(y^{\prime}|\mathbf{L},(y,\mathbf{x}),\mathbf{x^{\prime}})-p(y^{\prime}|\mathbf{L,x^{\prime}})\right)^{2}\right]$
$\displaystyle\Delta Q_{CoreLog}(\mathbf{x}|\mathbf{L})$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{p(\mathbf{x^{\prime}})p(y|\mathbf{L,x})}\left[KL(p(y^{\prime}|\mathbf{L},(y,\mathbf{x}),\mathbf{x^{\prime}})\;\|\;p(y^{\prime}|\mathbf{L,x^{\prime}}))\right]$

###  III-E Understanding Uncertainty and Diversity

How do the $\Delta Q$ functions of Eqs (LABEL:eq:QMSE) and (LABEL:eq:QLog)
relate to uncertainty, the traditional class of functions used in early AL?
The $\Delta Q_{CoreMSE}$ measures the average change in model probabilities
when the label of $\mathbf{x}$ is acquired and $\Delta Q_{CoreLog}$ measures
the average KL between model probabilities before and after acquisition. Thus
they measure the average model change induced by acquiring the label of
$\mathbf{x}$. These are a measure of epistemic uncertainty revealed by the
data $\mathbf{x}$, revealing how uncertain the model is around the data point,
and do not measure aleatoric uncertainty. Note, $p(y|\mathbf{L,x})$ is a proxy
for epistemic certainty in the common situation where the true model admits
high accuracy prediction, common in AL experiments.

What about diversity, the other traditional function used in AL? Eqs
(LABEL:eq:QMSE) and (LABEL:eq:QLog) also hold if $\mathbf{x}$ is a batch of
data to be labeled. Moreover, they give a fundamental theoretical objective to
maximise for batch AL, whereas diversity we argue is an observed feature of
good batch AL, that should be derivable from fundamental theory, such as ours.

###  III-F Enhanced Batch Diversity for BEMPS

Algorithm 2 gives an implementation of BEMPS for an arbitrary strictly convex
function $G(\cdot)$, returning the data point with the best-estimated measure.
To work with a Bregman divergence or score, the corresponding strictly convex
function $G(\cdot)$ should first be derived. When $G(\cdot)$ is negative
entropy, we call this CoreLog (Eq (14)) and when $G(\cdot)$ is the sum of
squares we call this CoreMSE (Eq (15), corresponding to the logarithmic or
Brier scoring rules respectively. Both Algorithms 2 and 3 use a fixed
estimation pool, $\mathbf{X}$, a fixed random subset of the initial unlabeled
data used to estimate expected values
$\mathbb{E}_{p(\mathbf{x^{\prime}})}[\cdot]$. Algorithm 2 calls Algorithm 1
which implements Eq (10) and estimates the expected score change with respect
to $\mathbf{x^{\prime}}\in\mathbf{X}$, while assuming the label of
$\mathbf{x}$. Note $p({\bf\it\theta}|\mathbf{L},(\mathbf{x},y))$ is computed
from $p({\bf\it\theta}|\mathbf{L})$ via ensembles as follows:

$\displaystyle
p({\bf\it\theta}|\mathbf{L},(\mathbf{x},y))\approx\frac{p({\bf\it\theta}|\mathbf{L})p(y|{\bf\it\theta},\mathbf{x})}{\sum_{{\bf\it\theta}\in{\bf\it\Theta}}p({\bf\it\theta}|\mathbf{L})p(y|{\bf\it\theta},\mathbf{(}x))}$
(16)

Algorithm 3 returns $B$ data points representing a batch with enhanced
diversity: it first calls Algorithm 1 to compute, for each data point
$\mathbf{x}$ in the unlabeled pool, a vector of expected changes in score
values over the estimation pool $\mathbf{X}$. Thus, this vector conveys
information about uncertainty directly related to the change in score due to
the addition of $\mathbf{x}$. While the gradient embedding used in [11]
represents a data point’s impact on the model, our vector represents a data
point’s direct impact on the mean proper score. Concurrently Algorithm 3
computes the estimate of $\Delta Q(\mathbf{x|L})$ for these same
$\mathbf{x}$s. The top $T$% of scoring data $\mathbf{x}$ are then clustered
with $k$-Means and a representative of each cluster closest to the cluster
mean is returned. The intuition is that 1) only higher scoring data $x$ should
appear in a batch; 2) those clusters capture the pattern of expected changes
in score values deduced by samples in the unlabeled pool; 3) samples in the
same cluster can affect the learning similarly, so it should not co-occur in a
batch.

Algorithm 3 Finding a diverse batch

1:unlabeled pool $\mathbf{U}$, batch size $B$

2:estimation pool $\mathbf{X}$, top fraction $T$

3:$\forall_{\mathbf{x}\in\mathbf{U}}Q_{\mathbf{x}}=0$

4:for $\mathbf{x}\in\mathbf{U}$, $\mathbf{x^{\prime}}\in\mathbf{X}$ do

5: $Q_{\mathbf{x}}~{}+\\!\\!=vec_{\mathbf{x,x^{\prime}}}=\Delta
Q(\mathbf{x|L,x^{\prime}})$

6:$V\leftarrow topk(Q,T*|\mathbf{U}|)$

7:$batch=\emptyset$

8:$centroids$ = $k$-Means centers $(vec_{\mathbf{x}\in V},B)$

9:for $c\in centroids$ do

10: $batch~{}\cup\\!\\!=\\{\operatorname*{arg\,min}_{\mathbf{x}\in
V}||c-vec_{\mathbf{x}}||\\}$

11:return $batch$

Algorithm 4 Bayesian Estimate of Mean Proper Scores with Ensembles

1:initial unlabeled data $\mathbf{U}$, initial labeled data $\mathbf{L}$,

2:batch size $B$, estimation pool $\mathbf{X}$, the number of acquisition
iteration $N$

3:Initialize:
$n=0,\mathbf{L}_{0}\leftarrow\mathbf{L},\mathbf{U}_{0}\leftarrow\mathbf{U}$

4:while $n<N$ do

5: if Ensemble method is Deep Ensembles then

6: while $e<E$ do

7: Randomly split $\mathbf{L}_{i}$ into training and validation

8: Train a model ${\bf\it\theta}_{i,e}$

9: if Ensemble method is MC-Dropout then

10: Randomly split $\mathbf{L}_{n}$ into training and validation

11: Train a single model ${\bf\it\theta}_{n}$ on the split

12: while $e<E$ do

13: Compute ${\bf\it\theta}_{n,e}$ via a single forward pass through

14: the current trained model ${\bf\it\theta}_{n}$ with MC-Dropout

15:
${\bf\it\Theta}_{n}=\\{{\bf\it\theta}_{n,1},{\bf\it\theta}_{n,2},...,{\bf\it\theta}_{n,E}\\}$

16: $batch=\emptyset$

17: if $B$ is one then

18: $batch=\operatorname*{arg\,max}_{\mathbf{x}\in\mathbf{U}}\Delta
Q(\mathbf{x|L})$ implemented by
Algorithm 2

19: else

20: $batch=B$ samples acquired with Algorithm 3

21: $\mathbf{L}_{n+1}\leftarrow\mathbf{L}_{n}\cup batch$

22: $\mathbf{U}_{n+1}\leftarrow\mathbf{U}_{n}\setminus batch$

###  III-G Ensembles for BEMPS

Computing $p(\cdot|\mathbf{L},x)$ and
$p(\cdot|\mathbf{L},(\mathbf{x},y),\mathbf{x}^{\prime})$ is intractable. We
approximate the integral over ${\bf\it\theta}$ via  two ensemble methods in
Algorithm 4: Deep Ensembles and MC-Dropout. We first implement Deep Ensembles
since the method has been successfully used to improve predictive performance
[42],  MC-Dropout, trying to peed up the acquisition process

Deep Ensembles. Let $\mathbf{\Theta}$ denote a set of DNN models (i.e.,
DistilBERT [50] for text, and VGG-16 [51] for images in our experiments) with
size $E$. Each individual model, denoted as ${\bf\it\theta}_{e}$, is trained
with a randomly generated training-validation split of incrementally augmented
labeled pool $\mathbf{L}$ at each acquisition iteration. The training-
validation ratio is set to 70/30. In other words, different models are trained
utilizing different train/validation splits, which fosters ensemble
diversification. We call this split process dynamic validation set (aka
Dynamic VS). Once the ensemble models are trained, a batch of samples can be
acquired via Algorithm 3. This dynamic approach to validation was found to
increase the training efficiency and model’s performance, as demonstrated in
our ablation studies.

MC-Dropout. One drawback of using Deep Ensembles with DNNs is its high
computational cost in terms of training the active learner, i.e., the
classifier. [44] demonstrated that Monte Carlo samples of the posterior can be
obtained by conducting several stochastic forward passes at test time. Thus,
an ensemble of $E$ models often used in standard deep ensemble methods can be
replaced with MC-Dropout models in the AL process to approximate the integral
over ${\bf\it\theta}$ [6]. In other words, the predictive distribution
$p(\cdot|\mathbf{L,x})$ and
$p(\cdot|\mathbf{L},(\mathbf{x},y),\mathbf{x^{\prime}})$ could be estimated
through $E$ forward passes on a singled trained model,  rather than training
the classifier $E$ times, and thus saving computational cost, particularly
when the value of $E$ becomes large. Refer to Table V.

##  IV Experiments

We conducted comprehensive sets of experiments on various classification tasks
to demonstrate the efficacy of BEMPS by comparing CoreMSE and CoreLog to some
recent AL methods. To show the reliability and robustness of our BEMPS
framework, we consider four benchmark text datasets and two benchmark image
datasets for either binary classification or multi-class classification tasks.
Moreover, going beyond the quantitative studies, we further visualise
different AL methods using Data Maps [23] and t-SNE to have an in-depth
understanding of their sampling behaviors along with model calibration.

TABLE II: Four benchmark text datasets and the used language model

Dataset |  Unlabeled/ Test sizes |  Lang. Model |  Initial labeled size
---|---|---|---
AG NEWS | 120,000 / 7,600 | DistilBERT | 26
PUBMED | 15,000 / 2,500 | DistilBERT | 26
IMDB | 25,000 / 25,000 | DistilBERT | 26
SST5 | 8544 / 2210 | DistilBERT | 26

TABLE III: Image datasets and the used models

Dataset |  Unlabeled/ Test sizes |  NN Model |  Initial labeled size
---|---|---|---
MNIST | 60,000 / 10,000 | BNN | 20
CIFAR10 | 50,000 / 10,000 | VGG-16bn | 20

###  IV-A Text classification

Figure 1: Performance on SST5 dataset. The left half illustrates the learning
curve, while the right half illustrates the matrix of paired comparisons. The
dashline represents the performance of the backbone classifier trained on the
entire dataset.

####  IV-A1 Datasets

We use four benchmark datasets: IMBD, AG NEWS, PUBMED and SST5 for text
classification as shown in Table III. IMDB contains 50K movie reviews,
categorized as either Positive or Negative [52]. AG NEWS includes 120K texts
distributed across four classes: Science/Technology, World, Business, and
Sports [53]. [53]. PUBMED 20k is designed for sentence classification and
contains approximately 20K medical abstracts divided into five classes:
Objective, Background, Conclusions, Results, and Methods [54]. Similarly, SST5
includes 11K sentences from movie reviews, labeled with five sentiment classes
[55].

####  IV-A2 Baselines

Let $B$ indicate the batch side. We consider two types of performance
comparisons: non-batch active learning ($B=1$) and batch active learning
($B>1$). In the case of non-batch active learning, CoreMSE and CoreLog are
compared against Max-Entropy [56], BALD [32], MOCU [20], and WMOCU [20], along
with a random baseline. For batch active learning, CoreMSE and CoreLog are
compared to WMOCU, BADGE [11], and ALPS [7], also alongside a random baseline.

CoreMSE & CoreLog. In the default configuration of BEMPS we used deep
ensembles. For non-batch active learning, we chose a sample with the highest
uncertainty score related to classification error. For batch active learning,
each sample in the unlabeled pool is represented as a vector of scores
computed by Eq (10). We then used $k$-MEANS to generate $B$ clusters and
select from each cluster the sample closest to the cluster center to form the
batch.

Random. We sampled $B$ samples uniformly from the unlabeled pool.

Max-Ent. We chose a sample with the highest entropy of the predictive
distribution [57, 30].

BALD. Like Max-Entropy, we chose a sample with the maximum mutual information
based on how well labeling those samples would improve the model parameters
[32].

MOCU & WMOCU. Following [20], for non-batch active learning, we selected a
sample with the highest uncertainty score related to classification error.
Similarly to CoreMSE and CoreLog, we represent each sample in the unlabeled
pool as a vector of scores, as computed by Eqs (5)and (10) in [20]. For batch
active learning, we selected $B$ samples using the k-MEANS clustering
approach.

BADGE. Following [11], we represented each sample with a gradient embedding
generated from a pretrained language model, specifically DistilBERT in our
experiments. Then, using $k$-MEANS++, we generate $B$ clusters and choose a
representative from each cluster that is closest to the mean.

ALPS. Different from BADGE, we followed [11] to generate surprisal embeddings
from DistilBERT as inputs to $k$-MEANS. Then, a similar approach was used to
choose the $B$ samples in a batch.

####  IV-A3 Model configuration

We used a small and fast pretrained language model, DistilBERT [50] as the
backbone classifier in our experiments. We fine-tuned DistilBERT on each
dataset after each AL iteration with a random re-initialization [58], proven
to improve the model performance over the use of incremental fine-tuning with
the newly acquired samples [31]. The maximum sequence length was set to 128,
and a maximum of 30 epochs was used in fine-tuning DistilBERT with early
stopping [59]. We used AdamW [60] as the optimizer with a learning rate 2e-5
and betas 0.9/0.999. All experiments were run on 8 Tesla 16GB V100 GPUs.

Each AL method was run for 5 times with different random seeds on each
dataset. The batch size $b$ was set to 1 for the non-batch active learning,
and for batch active learning, the batch size $B$ is set to {5, 10, 50, 100}.
We trained five DistilBERTs with Dynamic VS as members of a deep ensemble, as
described in Algorithm 4. Note that our implementation of CoreMSE and CoreLog
does not rely on a separate large validation set.

Figure 2: Learning curves of batch size 50 for PUBMED, IMDB, SST5 and AG NEWS.
The dashline represents the performance of the backbone classifier trained on
the entire dataset.

(a) F1-based pairwise comparison

(b) Accuracy-based pairwise comparison

Figure 3: Pairwise comparison matrices of batch active learning strategies on
four datasets with four batch sizes.

####  IV-A4 Comparative performance metrics

We followed [11] to compute a pairwise comparison matrix but used a counting-
based algorithm [61], as shown in the right of Fig. 1. The rows and columns of
the matrix correspond to the AL methods used in our experiments. Each cell
represents the outcome of the comparison between method $i$ and method $j$
over all datasets (D). Let $C_{i,j,d}=1$ when method $i$ beats method $j$ on
dataset $d$, and 0 otherwise. The value of each cell is computed as
$C_{i,j}=\sum_{d}^{D}C_{i,j,d}$. To determine the value of each $C_{i,j,d}$
entry, we used a two-side paired $t$-test to compare their performance for
five weighted F1 scores (or accuracy) at maximally spaced labeled sample sizes
$\\{l_{i,j,d}^{1},l_{i,j,d}^{2},...,l_{i,j,d}^{5}\\}$ from the learning curve.
We computed the $t$-score as $t=\sqrt{5}\hat{\mu}/\hat{\sigma}$, where
$\hat{\mu}$ and $\hat{\sigma}$ are the usual sample mean and standard
deviation. For example, in Fig. 1, five samples according to a step size 50
were used: the first sample $l_{i,j,d}^{1}$ was chosen after $50$ labels, the
second $l_{i,j,d}^{1}$ after $100$, etc. Whereas, for Fig. 2, a step size 100
was used.
$\hat{\mu}=\frac{1}{5}\sum_{k=1}^{5}\left(l_{i,j,d}^{k}\right),\hat{\sigma}=\sqrt{\frac{1}{4}\sum_{k=1}^{5}\left(l_{i,j,d}^{k}-\hat{\mu}^{2}\right)}$.
The value $C_{i,j,d}$ is set to 1 if method $i$ outperforms method $j$ with a
$t$-score greater than 2.776 (corresponding to a $p$-value less than 0.05). We
summed the outcomes of each pairwise comparison to determine the total
performance score for each method, as displayed in the ’Total’ column of the
matrix. The method with the highest total score is ranked as the best among
the AL methods evaluated.

####  IV-A5 Results and Analysis

Non-batch active learning. We first compared our CoreMSE and CoreLog based on
Algorithm 2 to the baselines on the SST5 datasets to demonstrate how those
methods perform particularly in a hard classification setting where classes
are imbalanced. The learning curve sitting on the left of Fig. 1 shows
CoreMSE, CoreLog and WMOCU outperform all the other methods considered, we can
attribute this to their estimation of uncertainty being better related to
classification accuracy. Among these three methods, our CoreMSE performs the
best in terms of F1 score. The matrix at the right of Fig. 1 then presents a
statistical summary of comparative performance. CoreMSE has the highest total
quantity which further confirms its effectiveness in acquiring informative
samples in AL.

Batch active learning. We compared batch CoreMSE and CoreLog implemented based
on Algorithm 4 with BADGE, ALPS and batch WMOCU on the four datasets listed in
Table III, We extended WMOCU with our Algorithm 4 to build its batch
counterpart. Specifically, we generated $vec_{\mathbf{x}}$ using its point-
wise error estimates, i.e., Eq (10) in [20]. The random baseline selects $B$
unlabeled samples randomly. Here we present the results derived with $B=50$ as
an example. More comprehensive results with different batch sizes, including
accuracy, can be found in Appendix B.

The learning curves in Fig. 2 show that batch CoreMSE and CoreLog almost
always outperform the other AL methods as the number of acquired samples
increases. Batch WMOCU devised with our batch algorithm compare favourably
with BADGE and ALPS that use gradient/surprisal embeddings to increase batch
diversity. These results suggest that selecting the representative samples
from clusters learned with vectors of expected changes in scores (i.e., Eq
(10)) is appropriate, leading to an improved AL performance. Moreover, the
performance differences between our methods and others on PUBMED and SST-5
indicate that batch CoreMSE and CoreLog can still achieve good results when
the annotation budget is limited in those imbalanced datasets.

We also created two pairwise comparison matrices for different batch sizes
using either F1 score or accuracy. Fig. 3 show the sum of the four matrices,
summarizing the comparative performance on the four datasets. The maximum cell
value is now $4\times 4=16$. In other words, if a method beats another on all
the four datasets across the four different batch sizes, the corresponding
cell value will be 16. Both matrices computed with F1 score and accuracy
respectively show both CoreMSE and CoreLog are ranked higher than the other
methods. Thus, we attribute the better performance of CoreMSE and CoreLog to
the fact that their estimation of uncertainty is more correlated to prediction
performance than other methods studied.

###  IV-B Ablation Studies

(a) Learning curves of various batch size for CoreMSE.

(b) Learning curves of the model training with diversity.

Figure 4: Learning curves for batch size and batch diversity on PUBMED and
SST5. The dashline represents the performance of the backbone classifier
trained on the entire dataset.

Batch size. In Fig. 4(a), we plotted the learning curves of batch CoreMSE with
various batch sizes (i.e., $B\in\\{1,5,10,50,100\\}$) for PUBMED and SST5. The
curves demonstrate that the performance of smaller batch sizes (5 or 10) is
better to that of larger batch sizes (50 or 100), particularly in the early
training iterations. Moveover, the batch version of CoreMSE and CoreLog
perform substantially better than their non-batch counterparts. In order to
acquire $B$ samples, the non-batch (i.e., $B=1$) case must perform multiple
one-step-look-ahead optimising acquisitions consecutively (Algorithm 1 and
Algorithm 2). In contrast, the batch case must only perform the acquisition
once, heuristically, introducing a form of diversity based directly on the
error surface (Algorithm 1 and Algorithm 3). While the importance of diversity
is illustrated in [62], they used larger batch sizes in their experiments. a
phenomena also seen with BatchBALD [5]. Thus we hypothesise that the one-step-
look-ahead of Equation (1) is only greedy, and not optimal.  In other words,
diversity of the batch of data is an advantage for the $\Delta Q$ functions of
Equations (LABEL:eq:QMSE) and (LABEL:eq:QLog), and this occurs mainly for
smaller batch sizes where the projections for model probability estimates
after acquiring labeled data retain some accuracy. Overall, the learning
curves show that batch algorithms outperform non-batch algorithms, and that
smaller batch sizes are preferable to large batch sizes.

Batch diversity. To further study the effectiveness of Algorithm 3, we
considered the following variants: 1) Pretrained-LM: We used the embedding
generated by DistilBERT for $k$-Means clustering, which is similar to the
BERT-KM in [7]; and 2) CoreMSE-topK and CoreLog-topK: We simply chose the top
$B$ samples ranked by $Q_{x}$. the results shown in Fig. 4(b) show that batch
CoreMSE and CoreLog perform much better than the corresponding CoreMSE-topK
and CoreLog-topK, which showcases Algorithm 3 can promote batch diversity that
benefits AL for text classification. The performance difference between
Pretrained-LM and batch CoreMSE/CoreLog indicates that representing each
unlabeled sample as a vector of expected changes in scores (i.e., Eq (10)) is
effective in capturing the information to be used for diversification.

Figure 5: Learning curves of the model training with a dynamic validation set,
constant validation set, fixed # epochs without validation set, fixed length #
labels validation set for CoreMSE. The dashline represents the performance of
the backbone classifier trained on the entire dataset.

Dynamic VS. We studies how Dynamic VS impacts the ensemble model training by
comparing batch CoreMSE with Dynamic VS to its following variations: 1) 3/30
epochs without VS: ensemble model training without VS and each model is
trained for 3 or 30 epochs [59], 2) Fixed length (#1000) VS: a pre-fixed
validation set with 1000 labeled samples separate from the labeled pool, used
in some existing AL empirical work; 3) Constant VS: a variant of Dynamic VS
where one random split is generated after each AL iteration and then shared by
all the ensemble models. Fig. 5 shows Dynamic VS gains an advantage after the
third acquisition iteration on PUBMED and SST5. It is not surprising that 30
epochs without VS and Fixed length VS perform better in the early acquisition
iterations, since they use the whole augmented labeled pool in training
DistilBERT, whereas CoreMSE used 70%. But choosing the number of epochs
without a validation set is simply heuristic otherwise. Also Fixed length VS
is midway between Constant VS and Dynamic VS, indicating the variability in
ensembles inherent in the dynamic training sets is a source of improvement.

Deep Ensemble vs MC-Dropout. CoreMSE and CoreLog use deep ensembles, and when
MC-Dropout is used instead, they are denoted as CoreMSE-MC and CoreLog-MC.
Fig. 2 also shows that the performance of CoreMSE-MC and CoreLog-MC with MC-
dropout are quite similar to CoreMSE and CoreLog with deep ensembles with the
same annotation budget (i.e., 500) and the same batch size (i.e., 50). The F1
and accuracy rankings not only closely match between CoreMSE-MC and CoreMSE in
deep ensembles but also exhibit a similar trend between CoreLog-MC and
CoreLog, as illustrated in Fig. 3. This result is unexpected due to the
literature on ensembles [45]. However, MC-Dropout achieves similar performance
to Deep Ensembles with substantially less training time, as shown in both
Table IV and Table Table V. The training speedup it offers ranges from a
4-fold increase to a 7-fold increase, depending on the size of the ensemble.
and details on the interaction with ensembling can be found in Table V.

(a) Learning curves of batch size 50

(b) F1-based pairwise comparison

Figure 6: Learning curves and pairwise comparison matrices of batch size 50 for the image datasets based on F1 scores.  TABLE IV: Average training (T) and query (Q) time in seconds, measured after acquiring 500 samples with a batch size of 50 and using 5 ensembles. The times are calculated using five random seeds for various AL methods on DistilBERT AL method | AG NEWS | PUBMED | IMDB | SST5
---|---|---|---|---
| T / Q | T / Q | T / Q | T / Q
Rand | 75 / 1 | 75 / 1 | 120 / 1 | 45/1
Max-Ent | 75 / 319 | 75 / 52 | 120 / 74 | 45 / 7
BALD | 75 / 319 | 75 / 47 | 120 / 73 | 45 / 4
MOCU | 75 / 327 | 75 / 48 | 120 / 77 | 45 / 6
WMOCU | 75 / 333 | 75 / 52 | 120 / 78 | 45 / 7
BADGE | 75 / 858 | 75 / 115 | 120 / 130 | 45 / 22
ALPS | 75 / 609 | 75 / 86 | 120 / 116 | 45 / 8
CoreMSE | 75 / 320 | 75 / 49 | 120 / 74 | 45 / 5
CoreLog | 75 / 325 | 75 / 50 | 120 / 80 | 45 / 8
CoreMSE-MC | 15 / 320 | 15 / 49 | 24 / 74 | 9 / 5
CoreLog-MC | 15 / 325 | 15 / 50 | 24 / 80 | 9 / 8
TABLE V: Average training time, query time, and F1 score (F1) for ensembles
with batch size 50 after acquiring 500 samples. Evaluated using five seeds on
DistilBERT for SST5.

#Ensembles | 5 | 10 | 20 | 40
---|---|---|---|---
Metrics | T/Q/F1 | T/Q/F1 | T/Q/F1 | T/Q/F1
AL Methods |  |  |  |
CoreMSE | 45/5/0.385 | 90/15/0.391 | 180/33/0.396 | 360/60/0.403
CoreLog | 45/8/0.384 | 90/18/0.390 | 180/37/0.395 | 360/68/0.402
CoreMSE-MC | 9/5/0.400 | 9/15/0.402 | 9/33/0.404 | 9/60/0.406
CoreLog-MC | 9/8/0.396 | 9/18/0.400 | 9/37/0.402 | 9/68/0.405

###  IV-C Image classification

As shown in Section IV-B, CoreMSE-MC and CoreLog-MC can significantly reduce
training time while maintaining the same level of performance as their deep
ensemble counterparts. Therefore, without losing generality, we compared our
CoreMSE-MC and CoreLog-MC with other baselines for the image classification
task.

####  IV-C1 Datasets

We used two benchmark datasets for image classification: MINST and CIFAR10, as
shown in Table III. MNIST [63] consists of 28 greyscale images of handwritten
digits ranging from 0 to 9. The images were divided into 60K and 10K images
for training and testing, respectively. CIFAR10 [64] consists of 32 × 32
coloured images in 10 classes. Each of the 10 classes has 6K images. The
training and the test sets consist of 50K and 10K images, respectively. These
two image datasets have emerged as the preferred choice for many researchers
when conducting initial benchmarking of neural networks in image
classification tasks.

####  IV-C2 Baselines

We considered WMOCU [20], BADGE [11] and BatchBALD [5] together with a random
baseline. As we utilized those acquisition functions within the context of
Algorithm 4 with MC-Dropout, we denote these variants as CoreMSE-MC, CoreLog-
MC, WMOCU-MC, BADGE-MC, and Random-MC. Furthermore, we have also included
BatchBALD [5] into our baselines as an AL method that extends BALD [32] to the
batch setting while using MC-Dropout[44]. Specifically, we followed [5] to
choose top $B$ informative samples based on their ranked mutual information,
which BALD uses to measure how well labeling those samples would improve the
the model parameters. We deliberately excluded the ALPS method due to its
reliance on a pretrained language model, making it unsuitable for image
classification tasks.

####  IV-C3 Model configuration

For the MNIST dataset, we employed a Bayesian Neural Network (BNN) as the
backbone classifier. BNNs offer enhanced scalability for image data compared
to conventional neural networks. We opted for MC-Dropout as the variational
approximation method for our ensemble models, primarily due to its effective
scalability when dealing with large models and datasets, as indicated in [31].
For the CIFAR10, we utilized PyTorch’s [65] pretrained VGG-16bn, which
incoporates a dropout layer preceding a fully connected layer with 512 hidden
units. We conducted the experiment over five trials, using five MC-Dropout
samples, each with varying batch sizes ($B$ values of 5, 10, 50, and 100). All
the neural networks were trained for 30 epochs, utilizing the Adam optimizer
with an initial learning rate set to 1e-4. All experiments were conducted on a
workstation equipped with an RTX 3090 GPU and an Intel Core i9 CPU processor.

####  IV-C4 Comparative performance metrics

Using the performance comparison metrics discussed in Section IV-A4, we
generated a pairwise comparison matrix to compare CoreMSE-MC and CoreLog-MC
with other baselines.  The maximum value within each cell of this matrix is
$2\times 4=8$, reflecting the combination of two datasets and four distinct
batch sizes. To clarify, if one method consistently outperforms another across
both datasets and all four batch sizes, the corresponding cell value will be
8.

(a) Data Maps

(b) Sampling distribution of ALs with batch size 50 per iteration

Figure 7: Learnability of ALs on PUBMED.

####  IV-C5 Result and Analysis

As depicted in Fig. 6, we can observe the performance of these methods,
derived on the two image classification datasets. The learning curves
presented in Fig. 6(a) exhibit similar trends to those observed in balanced
datasets such as IMDB and AG NEWS in text classification. CoreMSE-MC and
CoreLog-MC outperforms all other baselines, with WMOCU-MC being the next most
competitive approach, surpassing BatchBALD and BADGE-MC. The random baseline
consistently performs the worst among all the methods. In addition, The
comparative matrix shown in Fig. 6(b) supports the observation that CoreMSE-MC
and CoreLog-MC significantly outperform the other methods. Consequently, the
scoring functions integrated within the BEMPS framework consistently deliver
superior performance compared to alternative methods.

Figure 8: Running average ECE for PUBMED, IMDB, SST5 and AG NEWS.

Figure 9: The t-SNE visualization of DistilBERT’s decision boundaries for the
unlabel pool based on 500 labeled PubMed samples, highlighting 50 specially
chosen samples to demonstrate effective diverse batch selection for AL
methods.

##  V Sampling Behaviours of Different ALs

In this section, we delved into the sampling behaviours of those AL methods to
further understand their performance differences. We considered 1) using Data
Maps [23] to visualise the sampling behaviours of different AL methods by
leveraging the training dynamics; 2) evaluating the calibration of the
classifier by computing the expected calibration error method; and 3)
exploring the distribution of selected samples by each AL method along the
decision boundary.

###  V-A Interpretation of Acquisition with Data Maps

Following [23], we created a Data Map for the PUBMED dataset and analyzed the
sample selection patterns of different AL methods. Specifically, we used the
contextualisation method based on statistics derived from the behavior of the
training process, i.e., training dynamics.

As defined in [23], the training dynamics of sample $(\mathbf{x_{i}},y_{i})$
is defined as statistics derived from $N$ epochs, which is then used to
generate coordinates on our map. The first measure attempts to capture the
confidence with which the learner gives the true label to an observation,
given its probability distribution. Confidence is the average model
probability of the true label over all epochs:
$\hat{\mu_{i}}=\frac{1}{N}\sum_{n=1}^{N}p(y_{i}|{\bf\it\theta}^{(n)},\mathbf{x}_{i})$
where ${\bf\it\theta}^{(n)}$ indicates the parameters of the classifier
derived at the end of the $n^{th}$ epoch, and $y_{i}$ indicates the true
label. The variability measures the spread of
$p(y_{i}|{\bf\it\theta}^{(n)},\mathbf{x}_{i})$ across epochs:
$\hat{\sigma_{i}}=\sqrt{\frac{\sum_{n=1}^{N}(p(y_{i}|{\bf\it\theta}^{(n)},\mathbf{x}_{i})-\hat{\mu_{i}})^{2}}{N}}$.
We also considered the correctness by calculating the fraction of instances
where the model correctly labels $\mathbf{x}_{i}$ over multiple epochs.
According to these three dimensions, we map these training samples along two
axes: the $y$-axis indicates the average model confidence, and the $x$-axis
indicates the variability of the samples. The map is visualised as a 2D
representation of a dataset where samples are distributed on a map based on
statistics indicating their “learnability.”

Both confidence and variability were used to cluster samples into three
different regions based on their properties as follows: 1) “easy-to-learn”
samples are those having high confidence and low variability. The classifier
consistently predicts those samples accurately; 2)“ambiguous” samples are
those with high variability. In other words, the predicted probability of
their true classes frequently change during training. Therefore, the
classifier is indecisive about those samples; 3) “hard-to-learn” samples are
those with low confidence and low variability, so the classifier often results
in poor performance on those samples. On top of the three regions, we further
categorize the samples into the four types inspired by [66] to explore their
distribution changes in different AL methods during the sampling iterations.
The correctness score categorizes these four types from 0 to 1: Easy ([0.00,
0.25)); Medium ([0.25, 0.50)); Hard ([0.50, 0.75)); and Impossible ([0.75,
1.00)).

Fig. 7(b) shows CoreMSE and CoreLog acquired more samples in Medium, Hard and
Impossible samples than other AL methods over the sampling iterations. Thus,
CoreMSE and CoreLog can efficiently acquire samples from “ambiguous” and
“hard-to-learn” regions in Fig. 7(a).

###  V-B Expected Calibration Error

Expected Calibration Error (ECE) evaluates the correspondence between
predicted probability and empirical accuracy [67]. It places the model’s
predicted accuracy against its empirical accuracy into bins to estimate the
calibration error [68, 67]. ECE utilises equal-mass bins and selects the
maximum number of bins while maintaining monotonicity in the calibration
function. Specifically,
$ECE=\sum_{m=1}^{M}\frac{|B_{m}|}{n}|accuracy(B_{m})-confidence(B_{m})|$ where
$n$ is the number of samples and $M$ is interval bins. The difference of
accuracy and confidence values for a set of bins reflects the calibration gap.

We used ECE to measure how well our text classifier is calibrated. To do so,
we plotted the running average of ECE after each epoch in Fig. 8, where the
bin size $M$ was set to 10. Those plots show that the text classifiers trained
with either CoreMSE or CoreLog yield better calibration, which can be
attributed to that minimising a proper scoring rule should lower ECE [27]. It
is noteworthy that the classifiers trained using CoreMSE consistently achieved
the lowest ECE across all four text datasets. This observation suggests that
if the primary goal is to improve calibration, CoreMSE may be the more
favourable choice compare to CoreLog.

Figure 10: t-SNE plot: CoreMSE sample selection at acquisition sizes
$B\in{5,10,50}$, based on 500 labeled PubMed samples. White crosses indicate
acquired samples.

###  V-C Distribution of Acquired Samples

Query strategies focusing exclusively on uncertainty tend to select similar
labels that are close to the decision boundary, which can negatively impact
learning outcomes. In the t-SNE plots shown in Fig. 9, this issue is clearly
illustrated with a white cross marking each selected sample and colored dots
indicating their true labels. WMOCU, which centers on high-uncertainty
regions, fails to adequately diversify its sample selections, thereby
introducing bias and inefficiency. In contrast, ALPS and BADGE opt for a more
varied sample set, effectively covering the feature space. Similarly, CoreMSE
and CoreLog achieve an good balance between selecting uncertain samples and
ensuring diversity, highlighting the role diversity plays in the success of AL
methods. For smaller batch sizes, CoreMSE tends to pick samples with the
highest uncertainty from varied cluster regions. However, as the batch size
increases, there’s a tendency for the algorithm to repeatedly select samples
from the same region, which can convey similar information with respect to the
decision boundaries, as illustrated in Fig. 10. In contrast, smaller batch
sizes excel at choosing less-redundant samples from the unlabeled pool in each
acquisition round, effectively making their cumulative selections more
informative for learning the classifiers.

##  VI Conclusion

We proposed the BEMPS framework for active learning, focusing on acquisition
functions that are developed based on strictly proper scoring rules [27].
These developed acquisition functions specifically aim to address epistemic
uncertainty that directly influences the model’s classification performance.
For this, we developed convergence theory, borrowing techniques from [20],
that we also extended to the earlier BALD acquisition function. we
instantiated BEMPS with the Brier score and the logarithmic score, deriving
two new acquisition functions, CoreMSE and CoreLog respectively. For more
efficient and effective evaluation of those two acquisition functions, we
developed two techniques to improve performance, a batch AL strategy naturally
complement to our BEMPS algorithms, and a dynamic validation-ensemble hybrid
that generates high scoring ensembles but does not require a separate large
labeled validation set.  Empirical results have demonstrated CoreMSE and
CoreLog outperformed other AL methods on both text and image classification
with different batch sizes.

We further investigated both deep ensembles and Monte-Carlo dropout in our
BEMPS framework, aiming to speed up the process of retraining the classifiers
after each acquisition iteration. While traditional deep ensemble method can
still work on small datasets, the stochastic ensemble of Monte-Carlo dropout
models can significantly reduce the cost of retraining the classifier.
Nevertheless, both deep ensembles and those based on Monte-Carlo dropout can
achieve a similar level of classification performance with the annotation
budget.

Finally, we analyzed quantitatively the sampling behaviours of CoreMSE and
CoreLog in order to understand their performance difference. To do so, we
investigated the distribution of selected samples across three different
regions created based on the training dynamics with DataMap, and the
distribution along the decision boundary with t-SNE. Moreover, the ECE showed
that the classifier trained with CoreMSE and Corelog are the most calibrated
among those trained with the other acquisition strategies. It is noteworthy
that a limitation of BEMPS and its theory is that it does not support the use
of unlabeled data within the learning algorithm, for instance, as would be
done by state of the art semi-supervised learning methods. Given that semi-
supervised learning and AL really address a common problem, this represents an
area for future work for our techniques.

## Acknowledgment

The work has been supported by the Tides Foundation through Grant 1904-57761,
as part of the Google AI Impact Challenge, with Turning Point. Wray Buntine’s
work was also supported by DARPA’s Learning with Less labeling (LwLL) program
under agreement FA8750-19-2-0501.

## References

  * [1] A. Jiménez-Sánchez, S. Albarqouni, and D. Mateus, “Capsule networks against medical imaging data challenges,” in _Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis_. Springer, 2018, pp. 150–160.
  * [2] B. Settles, “Active learning literature survey,” Univ. of Wisconsin–Madison, Computer Sciences Technical Report 1648, 2009.
  * [3] M. Li and I. Sethi, “Confidence-based active learning,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 28, no. 8, pp. 1251–1261, 2006.
  * [4] A. Holub, P. Perona, and M. C. Burl, “Entropy-based active learning for object recognition,” in _2008 IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops_ , 2008, pp. 1–8.
  * [5] A. Kirsch, J. van Amersfoort, and Y. Gal, “BatchBALD: Efficient and diverse batch acquisition for deep Bayesian active learning,” in _Proc. 32nd Advances Neural Inf. Process. Syst._ , 2019.
  * [6] R. Pop and P. Fulop, “Deep ensemble Bayesian active learning : Addressing the mode collapse issue in Monte Carlo dropout via ensembles,” 2018. [Online]. Available: https://arxiv.org/abs/1811.03897
  * [7] M. Yuan, H.-T. Lin, and J. Boyd-Graber, “Cold-start active learning through self-supervised language modeling,” in _Proc. 2020 Conf. Empirical Methods Natural Lang. Process. (EMNLP)_ , Nov. 2020, pp. 7935–7948.
  * [8] O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” 2017. [Online]. Available: https://arxiv.org/abs/1708.00489
  * [9] C. Schröder and A. Niekler, “A survey of active learning for text classification using deep neural networks,” 2020. [Online]. Available: https://arxiv.org/abs/2008.07267
  * [10] P. Ren _et al._ , “A survey of deep active learning,” _ACM Comput. Surv._ , vol. 54, no. 9, Oct. 2021.
  * [11] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal, “Deep batch active learning by diverse, uncertain gradient lower bounds,” in _Proc. 8th Int. Conf. Learn. Representations_ , 2020.
  * [12] A. Siddhant and Z. C. Lipton, “Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study,” in _Proc. 2018 Conf. Empirical Methods Natural Lang. Process._ , Oct.-Nov. 2018, pp. 2904–2909.
  * [13] J. Lu and B. MacNamee, “Investigating the effectiveness of representations based on pretrained transformer-based language models in active learning for labelling text datasets,” 2020. [Online]. Available: https://arxiv.org/abs/2004.13138
  * [14] R. Gao and M. Saar-Tsechansky, “Cost-accuracy aware adaptive labeling for active learning,” in _Proc. AAAI Conf. Artif. Intell._ , 2020, pp. 2569–2576.
  * [15] F. Zhdanov, “Diverse mini-batch active learning,” 2019. [Online]. Available: https://arxiv.org/abs/1901.05954
  * [16] R. Pinsler, J. Gordon, E. Nalisnick, and J. M. Hernández-Lobato, “Bayesian batch active learning as sparse subset approximation,” in _Proc. 32nd Advances Neural Inf. Process. Syst._ , 2019.
  * [17] H. Raiffa and R. Schlaifer, _Applied Statistical Decision Theory_ , ser. Harvard Business School Publications. Division of Research, Graduate School of Business Adminitration, Harvard University, 1961.
  * [18] B.-J. Yoon, X. Qian, and E. R. Dougherty, “Quantifying the objective cost of uncertainty in complex dynamical systems,” _IEEE Trans. Signal Process._ , vol. 61, no. 9, pp. 2256–2266, 2013.
  * [19] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, “Clustering with Bregman divergences,” _J. Mach. Learn. Res._ , vol. 6, p. 1705–1749, Dec. 2005.
  * [20] G. Zhao, E. Dougherty, B.-J. Yoon, F. Alexander, and X. Qian, “Uncertainty-aware active learning for optimal Bayesian classifier,” in _Proc. 9th Int. Conf. Learn. Representations_ , 2021.
  * [21] G. Zhao, E. Dougherty, B.-J. Yoon, F. J. Alexander, and X. Qian, “Bayesian active learning by soft mean objective cost of uncertainty,” in _Proc. 24th Int. Conf. Artif. Intell. Statist._ , vol. 130, Apr. 2021, pp. 3970–3978.
  * [22] W. Tan, L. Du, and W. Buntine, “Diversity enhanced active learning with strictly proper scoring rules,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 10 906–10 918. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/5a7b238ba0f6502e5d6be14424b20ded-Paper.pdf
  * [23] S. Swayamdipta _et al._ , “Dataset cartography: Mapping and diagnosing datasets with training dynamics,” in _Proc. 2020 Conf. Empirical Methods Natural Lang. Process._ , Nov. 2020, pp. 9275–9293.
  * [24] L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, no. 86, pp. 2579–2605, 2008. [Online]. Available: http://jmlr.org/papers/v9/vandermaaten08a.html
  * [25] G. Zhao, E. Dougherty, B.-J. Yoon, F. Alexander, and X. Qian, “Efficient active learning for Gaussian process classification by error reduction,” in _Advances Neural Inf. Process. Syst._ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 9734–9746.
  * [26] J. Ash, S. Goel, A. Krishnamurthy, and S. Kakade, “Gone fishing: Neural active learning with Fisher embeddings,” in _Advances Neural Inf. Process. Syst._ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 8927–8939.
  * [27] T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” _J. Amer. Statistical Assoc._ , vol. 102, no. 477, pp. 359–378, 2007.
  * [28] J. Zhu, H. Wang, B. K. Tsou, and M. Ma, “Active learning with sampling by uncertainty and density for data annotations,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 18, no. 6, pp. 1323–1331, 2009.
  * [29] J. Zhu and M. Ma, “Uncertainty-based active learning with instability estimation for text classification,” _ACM Trans. Speech Lang. Process._ , vol. 8, no. 4, pp. 1–21, Feb. 2012.
  * [30] D. Wang and Y. Shang, “A new active labeling method for deep learning,” in _2014 Int. Joint Conf. Neural Netw._ , 2014, pp. 112–119.
  * [31] Y. Gal, R. Islam, and Z. Ghahramani, “Deep Bayesian active learning with image data,” in _Proc. 34th Int. Conf. Mach. Learn._ , vol. 70, Aug. 2017, pp. 1183–1192.
  * [32] N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel, “Bayesian active learning for classification and preference learning,” 2011. [Online]. Available: https://arxiv.org/abs/1112.5745
  * [33] N. Roy and A. McCallum, “Toward optimal active learning through sampling estimation of error reduction,” in _Proc. 18th Int. Conf. Mach. Learn._ , 2001, pp. 441––448.
  * [34] Z. Wang, B. Du, L. Zhang, and L. Zhang, “A batch-mode active learning framework by querying discriminative and representative samples for hyperspectral image classification,” _Neurocomputing (Amsterdam)_ , vol. 179, pp. 88–100, 2016.
  * [35] S. Ma, Z. Zeng, D. McDuff, and Y. Song, “Active contrastive learning of audio-visual video representations,” 2020. [Online]. Available: https://arxiv.org/abs/2009.09805
  * [36] H. T. Nguyen and A. Smeulders, “Active learning using pre-clustering,” in _Proc. 21st Int. Conf. Mach. Learn._ , 2004, p. 79.
  * [37] C. Yin _et al._ , “Deep similarity-based batch mode active learning with exploration-exploitation,” in _2017 IEEE Int. Conf. Data Mining_. IEEE, 2017, pp. 575–584.
  * [38] P. Liu, H. Zhang, and K. B. Eom, “Active deep learning for classification of hyperspectral images,” _IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens._ , vol. 10, no. 2, pp. 712–724, 2017.
  * [39] T. Shi, A. Benton, I. Malioutov, and O. İrsoy, “Diversity-aware batch active learning for dependency parsing,” in _Proc. 2021 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technologies_ , Jun. 2021, pp. 2616–2626.
  * [40] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conf. Comput. Vision Pattern Recognit._ , Jun. 2016, pp. 770–778.
  * [41] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in _Advances Neural Inf. Process. Syst._ , F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25, 2012, pp. 1097–1105.
  * [42] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Proc. 31st Int. Conf. Neural Inf. Process. Syst._ , 2017, pp. 6405–6416.
  * [43] N. Pawlowski, M. Jaques, and B. Glocker, “Efficient variational Bayesian neural network ensembles for outlier detection,” 2017. [Online]. Available: https://arxiv.org/abs/1703.06749
  * [44] Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in _Proc. 33rd Int. Conf. Mach. Learn._ , vol. 48, Jun. 2016, pp. 1050–1059.
  * [45] N. Durasov, T. Bagautdinov, P. Baque, and P. Fua, “Masksembles for uncertainty estimation,” in _2021 IEEE/CVF Conf. Comput. Vision Pattern Recognit._ , Jun. 2021, pp. 13 534–13 543.
  * [46] L. J. Savage, “Elicitation of personal probabilities and expectations,” _Journal of the American Statistical Association_ , vol. 66, no. 336, pp. 783–801, 1971.
  * [47] R. L. Winkler and V. R. R. Jose, _Scoring Rules_. John Wiley & Sons, Ltd, 2011. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470400531.eorms0749
  * [48] E. C. Merkle and M. Steyvers, “Choosing a strictly proper scoring rule,” _Decision Analysis_ , vol. 10, no. 4, pp. 292–304, 2013.
  * [49] A. P. Dawid and M. Musio, “Theory and applications of proper scoring rules,” _Metron_ , vol. 72, no. 2, pp. 169–183, 2014.
  * [50] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 2019. [Online]. Available: https://arxiv.org/abs/1910.01108
  * [51] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _CoRR_ , vol. abs/1409.1556, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:14124313
  * [52] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in _Proc. 49th Annu. Meeting Assoc. Comput. Linguistics: Human Lang. Technologies_ , Jun. 2011, pp. 142–150.
  * [53] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in _Proc. 28th Int. Conf. Neural Inf. Process. Syst._ , vol. 1, 2015, pp. 649–657.
  * [54] F. Dernoncourt and J. Y. Lee, “PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts,” in _Proc. 8th Int. Joint Conf. Natural Lang. Process._ , vol. 2, Nov. 2017, pp. 308–313.
  * [55] R. Socher _et al._ , “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proc. 2013 Conf. Empirical Methods Natural Lang. Process._ , Oct. 2013, pp. 1631–1642.
  * [56] Y. Yang and M. Loog, “Active learning using uncertainty information,” in _Proc. 23rd Int. Conf. Pattern Recognit._ , 2016, pp. 2646–2651.
  * [57] D. D. Lewis and W. A. Gale, “A sequential algorithm for training text classifiers,” in _Proc. 17th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval_ , 1994, pp. 3–12.
  * [58] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in _Proc. 7th Int. Conf. Learn. Representations_ , May. 2019.
  * [59] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, “Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,” 2020.
  * [60] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _Int. Conf. Learn. Representations_ , 2019.
  * [61] N. B. Shah and M. J. Wainwright, “Simple, robust and optimal ranking from pairwise comparisons,” _J. Mach. Learn. Res._ , vol. 18, no. 1, pp. 7246–7283, 2017.
  * [62] Y. Zhou, A. Renduchintala, X. Li, S. Wang, Y. Mehdad, and A. Ghoshal, “Towards understanding the behaviors of optimal deep active learning algorithms,” in _Proc. 24th Int. Conf. Artif. Intell. Statistics_ , vol. 130, Apr. 2021, pp. 1486–1494.
  * [63] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [64] A. Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009.
  * [65] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in _Autodiff Workshop, NIPS 2017_ , 2017.
  * [66] S. Karamcheti, R. Krishna, L. Fei-Fei, and C. Manning, “Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering,” in _Proc. 11th Int. Joint Conf. Natural Lang. Process._ , vol. 1, Aug. 2021, pp. 7265–7281.
  * [67] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning,” in _Proc. 29th AAAI Conf. Artif. Intell._ , 2015, p. 2901–2907.
  * [68] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _Proc. 34th Int. Conf. Mach. Learn._ , vol. 70, Aug. 2017, pp. 1321–1330.

|  Wei Tan Wei Tan is a Doctoral Researcher in the Machine Learning Group at
Monash University. He specializes in Active Learning that optimize the
labeling budget for the human annotation. He works for the Google Turning
point project, which develops a Surveillance System to capture coded ambulance
data on SITB, mental health, and AOD attendances to inform policy, strategy,
and intervention.
---|---
|  Lan Du Dr. Lan Du is currently a senior lecturer in Data Science and AI at
Monash University, Australia. His research interest lies in the joint area of
machine/deep learning and natural language processing and their cross-
disciplinary applications. He has published more than 80 high-quality research
papers in almost all top conferences/journals. He has been serving as an
editorial board member of the Machine Learning journal and ACM Transactions on
Probabilistic Machine Learning, an area chair & a senior PC of AAAI.
---|---
|  Wray Buntine Wray Buntine is full professor and Director of the Computer
Science Program at VinUniversity in Hanoi, and was previously Director of the
Machine Learning Group at Monash University. He is known for his theoretical
and applied work in probabilistic methods for document and text analysis, with
over 200 academic publications, 2 patents and some software products.
---|---