# Combining X-Vectors and Bayesian Batch Active Learning: Two-Stage Active
Learning Pipeline for Speech Recognition

Ognjen Kundacina, Vladimir Vincan, Dragisa Miskovic  This paper was realized
with the support of the Ministry of Science, Technological Development and
Innovation of the Republic of Serbia, according to the Agreement on the
realization and financing of scientific research: 451-03-66/2024-03/ 200025.
The authors are with The Institute for Artificial Intelligence Research and
Development of Serbia, Novi Sad, 21000, Serbia (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>dragisa.miskovic@ivi.ac.rs).

###### Abstract

Emphasizing a data-centric AI approach, this paper introduces a novel two-
stage active learning (AL) pipeline for automatic speech recognition (ASR),
combining unsupervised and supervised AL methods. The first stage utilizes
unsupervised AL by using x-vectors clustering for diverse sample selection
from unlabeled speech data, thus establishing a robust initial dataset for the
subsequent supervised AL. The second stage incorporates a supervised AL
strategy, with a batch AL method specifically developed for ASR, aimed at
selecting diverse and informative batches of samples. Here, sample diversity
is also achieved using x-vectors clustering, while the most informative
samples are identified using a Bayesian AL method tailored for ASR with an
adaptation of Monte Carlo dropout to approximate Bayesian inference. This
approach enables precise uncertainty estimation, thereby enhancing ASR model
training with significantly reduced data requirements. Our method has shown
superior performance compared to competing methods on homogeneous,
heterogeneous, and OOD test sets, demonstrating that strategic sample
selection and innovative Bayesian modeling can substantially optimize both
labeling effort and data utilization in deep learning-based ASR applications.

###### Index Terms:

Machine learning, active learning, speech recognition and synthesis, Bayesian
active learning, x-vectors, batch active learning.

##  I Introduction

State-of-the-art transformer models for automatic speech recognition (ASR)
require substantial volumes of labeled data. While there is an abundance of
unlabeled speech recordings, high-quality labeled data is scarce, particularly
in specialized domains or low-resource scenarios. Speech data labeling,
however, is labor-intensive and time-consuming, potentially requiring more
than eight hours to transcribe a single hour of speech recordings [1],
illustrating a significant bottleneck in the data preparation pipeline.
Embracing a data-centric AI approach [2, 3] by strategically choosing and
labeling a diverse and informative subset of audio samples can result in a
successfully trained and accurate ASR model, while reducing the labeling
effort. Study [4] reports that training the ASR model with just $17\%$ of
carefully selected samples can attain the similar level of accuracy as
training with the entire dataset, demonstrating the impact of data quality
over quantity and enhancing the computational efficiency and data usage during
the deep learning workflows.

Active learning (AL) is a technique that employs various aspects of machine
learning models, such as outputs, gradients, and uncertainties, alongside the
intrinsic properties of the unlabeled data samples, to select (query) the most
informative or diverse points for labeling. AL minimizes the need for human
involvement in data labeling, and is especially beneficial in datasets where
duplicates, examples with little informational value, or noisy instances are
common. Beyond failing to contribute to the model’s training process, the
presence of duplicate entries in the training dataset can negatively impact
model performance [5], underscoring the importance of implementing an
intelligent data selection strategy through AL. Additionally, the importance
of AL from the data management perspective for the enhancement of data
preparation pipelines is highlighted in [6]. This work focuses on exploring AL
methods designed for deep learning, known as deep active learning (DAL) [7],
aiming to formulate effective AL algorithms for transformer-based models in
ASR.

Literature review: AL has been employed in ASR research as traditional ASR
methodologies have advanced, utilizing methods based on discrete, symbolic
representations of speech to estimate audio sample informativeness from
statistical models [8, 9]. Recently, deep learning-based models have become
the preferred approach for ASR [10, 11], leveraging continuous, high-
dimensional representations learned from large amounts of labeled data. As a
result, there’s a growing development of DAL techniques for deep learning-
based ASR models, aimed at improving data quality and efficiency.

The most common application of DAL in ASR is found in supervised AL methods
[7]. These methods require an initial set of labeled data to train a deep
learning model. The model then assesses the uncertainty of each unlabeled
sample, selecting those with the highest uncertainty for further labeling.
With these additional labeled samples, a new deep learning model instance is
trained, which then more effectively identifies uncertain samples. This
iterative process is usually repeated a predefined number of times. Supervised
AL enables the model to improve its accuracy in areas of the data distribution
where it initially performs poorly.

In the context of ASR, path probability-based AL methods evaluate the
informativeness of unlabeled data samples by analyzing the decoded sequence
probabilities of each token. These methods follow the principle that lower
probabilities indicate higher informativeness or uncertainty, making such
samples valuable for enhancing model training upon being labeled and added to
the training set. Therefore, in this specific context, ’informativeness,’
’uncertainty,’ and ’confidence’ are often used interchangeably. Studies like
[12, 13, 14] apply these AL approaches in ASR, calculating the confidence
levels of predictions based on path probabilities, or the posterior
probabilities of tokens, derived from the softmax layer during sequence
decoding.

The methods discussed above employ confidence scoring for AL, considering an
unlabeled sample informative either due to its low path probability or high
entropy across all labels, indicating uniform distribution of predictions.
However, deep neural networks (DNNs) often exhibit overconfidence in
predictions from the softmax layer [15], questioning the reliability of using
path probabilities for uncertainty assessment of unlabeled data. A potential
solution to this issue are gradient-based AL methods, which have also been
applied for ASR [16]. These methods evaluate informativeness by measuring the
gradient norm of the model when processing unlabeled samples. The expected
gradient length (EGL) metric estimates how much a model’s parameters might
change if the unlabeled sample was included in the training process. Following
a similar path, the study in [17] employs a distinct DNN to estimate the true
gradient length, using EGL and entropy as inputs. The loss prediction method
[18] operates similarly to gradient-based techniques in that it does not
calculate uncertainty based on path probabilities, but provides an estimate of
how much a model’s parameters might change if the unlabeled sample were
included in the training process. It involves joint training of the ASR model
and a loss prediction DNN, where the uncertainty of an unlabeled sample is
defined by the predicted loss function value.

To enhance the diversity of the methods previously mentioned, i-vectors [19],
which are vector representations capturing speech audio sample variability,
can be used. In [20], the least confidence method is expanded by incorporating
an entropy measure related to clustered i-vectors. By clustering i-vectors
into $K$ groups using the K-means algorithm, an entropy-based regularization
term is added to the least confidence score. This entropy, associated with the
distribution of selected samples across clusters, drives the selection of a
diverse set of samples by encouraging sampling from all clusters. Implementing
this strategy involves the tuning of an additional hyperparameter to balance
the proposed diversity-based regularization term with the least confidence
score.

Committee-based methods introduce an advanced approach by leveraging multiple
models to assess the informativeness of unlabeled data samples, offering
improvements over single-model confidence score methods discussed so far. The
study [21] integrates informativeness scores from both confidence-based and
committee-based methodologies. In this setup, a committee comprising two DNNs,
a recurrent neural network (RNN) and a time delayed neural network, trained
under identical conditions, evaluates unseen samples. Their output
transcriptions are compared to calculate the word matching error rate (WMER),
which, when combined with the confidence score from the RNN, identifies the
most informative unlabeled samples for selection. Further advancements are
proposed in [22, 23], where the committee consists of the base model and
models modified using dropout techniques. The discrepancy in transcriptions
between these models quantifies uncertainty. Notably, [23] focuses on samples
at the intersection of those chosen by the committee approach and those
identified through the lowest confidence measures, to refine sample selection
further.

All the methods discussed so far have been within the scope of supervised AL.
Unsupervised AL, in contrast, does not require an initial set of labeled
data.111In the literature, this concept is often referred to as cold-start
active learning. In our work, however, we will use the term unsupervised AL,
as it has been previously adopted in the ASR domain [24]. These methods may
involve strategies like clustering or density estimation to identify valuable
or representative samples from the unlabeled dataset. The study [24] presents
an unsupervised AL method for ASR that employs acoustic features from a
traditional ASR system and a KL-divergence-based density method to select a
core subset of unlabeled samples that closely reflects the overall dataset’s
distribution. It suggests that an ASR model trained on a dataset chosen
through unsupervised AL can perform similarly to the one trained using
supervised AL, when the latter starts with a low-quality initial dataset.. A
recent work [25] presents an approach that achieves unsupervised AL in a
distinct setup, employing an ASR model trained on separate, unlabeled audio
and text data, i.e., in an unsupervised manner [26]. This methodology diverges
from the diversity-focused strategies typically associated with unsupervised
AL, since it calculates an uncertainty metric for AL using the perplexity of
phone sequences from the mentioned ASR model. This aspect of their approach
shares similarities with the principles of supervised AL, while it leverages
an ASR model trained in an unsupervised way.

In addition to AL, self-supervised learning [27] has emerged as a popular
approach for efficiently utilizing large volumes of unlabeled data. In our
work, we show that these two approaches could be combined for deep learning
models like wav2vec 2.0 [10], which are pre-trained in a self-supervised
manner using unlabeled data to learn rich audio representations, and require
subsequent fine-tuning on audio-transcription pairs. In other words, we will
utilize a wav2vec 2.0 model pretrained without any ASR-specific training and
fine-tune it with labeled data for the ASR task—data that has been effectively
selected using AL techniques.

Contributions: Supervised AL algorithms typically initiate with a labeled
dataset to train an initial deep learning model. This model then helps
identify which subset of data would be the most beneficial for labeling next.
In our research, we explore scenarios where AL starts with a completely
unlabeled dataset. Currently, there are no studies applying supervised AL in
ASR that offer an efficient method for selecting the initial unlabeled data
samples for labeling. We propose a two-stage AL process, presented on a high
level in Fig. 1, where the first stage involves using unsupervised AL to
select the initial dataset for labeling. Our goal is to create an initial ASR
model that is more effective than one trained on a similarly sized, but
randomly chosen dataset. Subsequently, this initial dataset and the ASR model
trained on it serve as inputs to the second stage of AL, which employs the
supervised AL method.

Figure 1: The proposed two-stage active learning pipeline. The first,
unsupervised active learning stage is based on x-vectors clustering. The
second, supervised active learning stage combines x-vector-based batch active
learning with Bayesian active learning via Monte Carlo dropout approximation.

For unsupervised AL, we employ x-vectors [28], extracted from a DNN trained
for speaker classification tasks, offering advantages over the previously
mentioned i-vectors. X-vectors provide a more nuanced representation of speech
data, capturing deeper characteristics beneficial for distinguishing between
different speakers and conditions. This makes them particularly effective for
clustering purposes, allowing us to select diverse data samples in an
unsupervised AL setting. Unlike the existing use of i-vector clusters in a
supervised AL framework to enhance diversity [20], which necessitates tuning
an additional hyperparameter to balance between uncertainty and selection from
diverse clusters, the proposed approach employs direct sampling from x-vectors
clusters to facilitate the selection of varied data samples. This approach
simplifies the process, eliminating the need for additional hyperparameter
tuning, while broadening the scope of data considered for an initial ASR model
training. Moreover, our method ensures that each cluster is represented in the
selection process, regardless of its size, guaranteeing that the ASR model is
exposed to a broad range of data variations during the training process.

Following the initial ASR model training with a dataset selected using
unsupervised AL, the second phase introduces an iterative supervised AL
process. Each AL iteration leverages the current ASR model to identify a new
subset of data for labeling using the proposed batch AL technique, tailored
for ASR. Our adoption of batch AL, which queries a batch of multiple unlabeled
samples in a single AL iteration, is motivated by its ability to select a
diverse set of samples rather than just individually informative ones. This
approach is particularly beneficial when faced with multiple similar uncertain
samples, as it opts for a varied batch, enhancing the overall learning
strategy. In contrast, non-batch AL methods might choose top-ranked samples
based solely on confidence scores, potentially leading to a less diverse
selection.

Our proposed batch AL approach for ASR uniquely integrates x-vectors
clustering for sample diversity with uncertainty assessment through a Bayesian
committee, specifically adjusted for ASR tasks. Theoretically, Bayesian neural
networks [29] can calculate a model’s uncertainty precisely, offering insights
into which samples could most improve model performance. However, due to the
high computational complexity of working with true posterior distributions in
practice, we approximate the Bayesian neural network for ASR using Monte Carlo
(MC) dropout [30]. This approximation creates a distribution over
transcriptions by employing a committee of models with diverse
topologies.222It is important to note here that this is not a typical
committee-based AL approach, since topologies of models are different for each
new data sample. These models are obtained by applying random dropout masks to
simulate diverse predictive outcomes. Each model in the committee outputs a
transcription for a given data sample, collectively forming a distribution.
The variance within this transcription distribution is quantified by
calculating the word error rate (WER) among the transcriptions, providing a
measure of uncertainty for the given unlabeled data sample.

The contributions of this work can be summarized as follows:

  * •

This work introduces, to our knowledge, the first two-stage AL pipeline for
ASR that sequentially combines unsupervised and supervised AL, using
unsupervised AL to establish a strategically selected initial dataset for
enhancing the effectiveness of supervised AL afterward.

  * •

We employ unsupervised AL to sample from groups of unlabeled samples created
through clustering of corresponding x-vectors [28], introducing a novel
application of x-vectors in the context of AL for ASR. This approach shares
some similarity with a method that employs i-vector clusters for
regularization in a supervised AL setting [20]; however, since our method is
applied in an unsupervised AL context, it eliminates the need for an
additional hyperparameter to balance regularization intensity.

  * •

Our batch AL method, designed specifically for ASR and positioned within
supervised AL, innovatively combines Bayesian AL with x-vectors clustering to
ensure sample diversity and enable precise uncertainty estimation.
Importantly, the most computationally intensive part of our method, the
uncertainty calculation, can be parallelized since uncertainties for each
sample are computed independently.

  * •

For uncertainty calculation, we introduce a novel adaptation of Bayesian AL
tailored for ASR. This adaptation includes approximating Bayesian inference
using a Monte Carlo dropout committee and developing a method to calculate the
WER-based variance over the transcription distribution provided by the
committee, marking a unique approach in the field.

  * •

The proposed approach is evaluated on a homogeneous test set designed to
assess performance for groups of speakers underrepresented in the training
data, and on a heterogeneous out-of-distribution (OOD) test set to test
robustness, with our method consistently outperforming two competing
approaches in both scenarios.

The paper is organized as follows: Section 2 provides the theoretical
foundation for our approach, Section 3 details our methodology, Section 4
evaluates the effectiveness of our methods, and Section 5 summarizes our
findings.

##  II Theoretical Preliminaries

In this section, we outline the theoretical background required for
introducing our proposed approach. Specifically, we briefly discuss ASR
utilizing the wav2vec 2.0 transformer neural network [10], x-vectors [28], and
the principles of DAL.

###  II-A Automatic Speech Recognition

Modern ASR methods focus on the goal of transforming spoken language into text
primarily using deep learning. The fundamental aim within this domain is to
develop neural networks that can accurately interpret and transcribe human
speech. In mathematical terms, an ASR system is represented as a trainable
function $f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$, where $\mathcal{X}$
is the set of all possible input audio samples and $\mathcal{Y}$ is the set of
all possible transcriptions, where $\theta$ denotes trainable parameters. The
training process involves adjusting the parameters $\theta$ to minimize the
difference between the predicted output and the actual transcription across a
labeled training dataset
$D_{L}=\\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{N},y_{N})\\}$, where
$x_{i}\in\mathcal{X}$ denotes the $i^{th}$ audio input and
$y_{i}\in\mathcal{Y}$ represents its corresponding transcription.

Wav2vec 2.0, a transformer-based ASR model, processes raw audio waveforms
directly [10]. Its architecture includes an encoder for extracting audio
signal representations, which are then mapped to speech units by a subsequent
linear layer. The model utilizes connectionist temporal classification (CTC)
loss [31] for training, facilitating the alignment of audio sequences with
text transcriptions without the need for explicit segmentation. While the
integration of language models can enhance the transcription quality of
wav2vec 2.0, they will not be considered in this work. Notably, wav2vec’s
design facilitates self-supervised pretraining by using segments of the audio
input to predict adjacent segments, effectively learning the structure of
audio data without labeled examples. In our work, we utilize wav2vec 2.0,
pretrained in a self-supervised manner on unlabeled audio data, and
subsequently fine-tune it using a labeled dataset selected through AL.

###  II-B X-Vectors

X-vectors are embeddings used for speaker recognition, derived from DNNs
trained to classify different speakers [28]. The process involves a DNN that
takes variable-length speech recordings and produces fixed-dimensional
embeddings. This is achieved through a pooling layer that aggregates features
across multiple frames early in the network. The architecture of the DNN
includes several hidden layers after the pooling operation, where each layer
is capable of representing the speaker’s unique features as embeddings. The
objective during training is to use these embeddings for speaker
classification, employing a softmax layer. The softmax layer is discarded
after training, since the focus lies on embeddings themselves, rather than the
classification output. For training, the DNN utilizes a dataset with labeled
speakers and applies multiclass cross-entropy loss to optimize the network’s
parameters. Data augmentation is employed to enhance the model’s ability to
generalize to new, unseen data, which is particularly relevant for our
experiments that utilize datasets not previously encountered by the trained
x-vectors.

Mathematically, the transformation performed by the DNN (excluding the softmax
layer) on an audio data sample can be expressed as
$f_{\theta_{x}}:\mathcal{X}\rightarrow\mathbb{R}^{d}$, where $d$ denotes the
dimensionality of the output x-vector embeddings. This function
$f_{\theta_{x}}$, parameterized by the DNN’s weights $\theta_{x}$,
encapsulates the transformation of speech utterance into a compact, fixed-
dimensional representation, serving as a fundamental component for selecting
diverse samples in our AL approach.

###  II-C Deep Active Learning

Deep active learning optimizes the training of deep learning models by
selectively labeling the most informative samples from an unlabeled dataset
[7]. Given a labeled dataset $D_{L}=\\{(x_{i},y_{i})\\}_{i=1}^{|D_{L}|}$ where
each $x_{i}\in\mathcal{X}$ corresponds to an input sample with its label
$y_{i}\in\mathcal{Y}$, and an unlabeled dataset
$D_{U}=\\{x_{j}\\}_{j=1}^{|D_{U}|}$ where $x_{j}\in\mathcal{X}$ lacks a label,
the goal of DAL is to select a subset $S\subseteq D_{U}$ for labeling that
maximizes the model’s performance improvement, by exploiting $D_{L}$ if
possible.

In supervised AL, the model is initially trained on the existing labeled data
$D_{L}$ and then iteratively updated by incorporating samples from $S$ that,
once labeled and added to $D_{L}$, are expected to most significantly reduce
the model’s error. More formally, supervised AL defines an acquisition
function, depending on the specific AL method or uncertainty metric, to select
the samples with the largest contribution to the model’s performance in each
AL iteration. This iterative process continues until achieving a predefined
performance level or exhausting the labeling budget. On the other hand,
unsupervised AL does not rely on $D_{L}$ for initial training, but instead
focuses on the structure or distribution of $D_{U}$ to select data samples for
labeling. Techniques such as clustering or density estimation might be used to
identify representative samples.

In supervised AL, the uncertainty-based methods selects samples from the
unlabeled dataset $D_{U}$ for which the model’s current predictions are the
least confident. This strategy aims to identify the samples that, if labeled
and added to the training set $D_{L}$, would most improve the model’s
performance by reducing its overall uncertainty. One common uncertainty-based
acquisition function for classification tasks is entropy [32], which for a
given sample $x\in D_{U}$ is defined as:

$H(x)=-\sum_{c=1}^{C}P(y_{c}|x)\log P(y_{c}|x),$ (1)

where $P(y_{c}|x)$ is the model’s predicted probability that the sample $x$
belongs to a class $c$, and $C$ is the total number of classes. The
uncertainty-based query strategy selects a subset $S$ from $D_{U}$ consisting
of a predefined number of samples with the highest entropy values for
labeling.

Committee-based methods in supervised AL use a group of diverse models, named
a committee, to make collective decisions on which samples to label next [33],
mitigating individual model biases that may occur in AL methods that rely
solely on a single model’s prediction probabilities. This approach is based on
the idea that different models may disagree on the predictions for the same
unlabeled data point, and these disagreements can highlight areas where
acquiring labels would most benefit the learning process. Given a committee of
models $\\{f_{\theta_{1}},f_{\theta_{2}},\ldots,f_{\theta_{M}}\\}$, where each
$f_{\theta_{m}}$ represents a model with parameters $\theta_{m}$ and $M$ is
the number of models in the committee, the uncertainty of the committee for a
sample $x\in D_{U}$ can be quantified in several ways. One common acquisition
function for committee-based methods is the vote entropy [33], defined for a
classification task with $C$ classes as:

$V(x)=-\sum_{c=1}^{C}P_{\text{vote}}(y_{c}|x)\log P_{\text{vote}}(y_{c}|x)$
(2)

where $P_{\text{vote}}(y_{c}|x)$ is the proportion of models in the committee
that predict $x$ as belonging to class $c$. This vote entropy increases with
the level of disagreement among the committee members about the class of $x$.
The committee-based query strategy then selects a subset $S$ from $D_{U}$,
consisting of samples with the highest vote entropy values for labeling.

In Bayesian active learning, weights within the DNN model are represented as
probability distributions, rather than deterministic estimates. This approach
allows for the estimation of uncertainty in data samples by examining the
variability in the model’s outputs, as Bayesian DNNs generate output
distributions. A higher variability in these outputs for unlabeled data
samples indicates greater uncertainty, suggesting the prioritization of
samples for labeling [34]. The uncertainty is inferred from the posterior
distribution of the model parameters, $P(\theta|D_{L})$, which is updated upon
observing new data $D_{L}$ in the following way:

$P(\theta|D_{L})=\frac{P(D_{L}|\theta)P(\theta)}{P(D_{L})},$ (3)

where $\theta$ denotes the parameters of the model. The process updates the
posterior distribution $P(\theta|D_{L})$ by combining the initial (often
Gaussian) prior beliefs $P(\theta)$ regarding the parameters with the
likelihood of the data given the parameters $P(D_{L}|\theta)$. Additionally,
it involves the marginal likelihood of observing the data $P(D_{L})$, which is
integrated over all possible parameter values $\theta$, and is computationally
demanding to perform in practice.

In Bayesian AL, the predictive distribution can be used for quantifying the
uncertainty associated with model predictions, providing a probabilistic basis
for making decisions about which data samples to label next. The exact
predictive distribution for output $y$, given an input $x$ and the observed
labeled dataset $D_{L}$, is calculated by integrating the likelihood
$P(y|x,\theta)$ of observing $y$ given the input $x$ and model parameters
$\theta$, with the posterior distribution $P(\theta|D_{L})$ over all possible
values of parameters:

$P(y|x,D_{L})=\int P(y|x,\theta)P(\theta|D_{L})d\theta.$ (4)

Accurately computing the true posterior $P(\theta|D_{L})$ and the predictive
distribution $P(y|x,D_{L})$ in Bayesian neural networks can be computationally
challenging, due to the need for integration across a high-dimensional
parameter space. To overcome this, we can employ stochastic regularization
techniques such as the Monte Carlo dropout as an approximation [30]. This
method uses dropout layers from the standard DNN architecture, with the
assumption that the DNN is trained using dropout, to perform stochastic
forward passes during inference. By performing multiple forward passes and
aggregating the results, we simulate sampling from the posterior distribution
of model parameters. This approximation, while maintaining the original DNN’s
structure unchanged, effectively captures the uncertainty by leveraging a
randomly generated committee of models.

For regression problems, the uncertainty measure for a data sample is
typically the variance across $T$ predicted outputs, which is a direct
approximation of the variance of the predictive distribution $P(y|x,D_{L})$
defined in (4), where $T$ denotes the number of MC dropout forward passes. For
classification problems, there are multiple ways to quantify output
variability, using which various acquisition functions can be defined, such as
the variation ratio, predictive entropy, or mutual information between
predictions and the model posterior [35]. For example, the variation ratio for
a sample $x$, which measures the data dispersion around the mode of
distribution $c^{*}$ obtained using $T$ forward stochastic passes, is defined
as:

$V_{\text{ratio}}(x)=1-\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}[y_{t}=c^{*}].$ (5)

Here, $\mathbb{I}[\cdot]$ is the indicator function, and $y_{t}$ is the
predicted class for the $t^{th}$ forward pass. As $T$ grows, the expression
$\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}[y_{t}=c^{*}]$ begins to approximate
$P(y=c^{*}|x,D_{L})$, which is the maximum predicted probability for class $c$
given the input $x$ and the labeled data $D_{L}$ that the DNN model was
trained on, making it a valuable measure for a model’s uncertainty in the
context of AL.

Batch active learning, referenced also as batch-mode active learning in some
studies, queries multiple unlabeled samples in one AL iteration [36], unlike
traditional supervised AL methods that query one sample at a time. Given the
substantial computational costs associated with retraining large and complex
deep learning models, it is impractical to retrain after adding just a single
labeled sample to the training set, particularly since such an addition
typically results in insignificant improvements. To overcome this issue,
traditional AL methods can be adjusted to select a batch of the most uncertain
samples in every AL iteration. However, the samples are selected based on
their individual uncertainties, without considering which combination would
contribute to the training process the most. The main goal of batch AL is to
select a predefined number $b$ of samples that are both diverse and
informative from the unlabeled dataset in each AL iteration. This approach is
beneficial when the model encounters multiple similar samples with high
uncertainty, avoiding the selection of the top uncertain $b$ samples only.
Focusing both on informativeness and diversity reduces the chance of choosing
duplicate samples, which could hinder learning progress, as indicated in [5].
Additionally, avoiding duplicates is beneficial because it prevents the waste
of training and labeling resources on redundant information and ensures a
broader coverage of samples in a single batch.

Formally, in batch AL, a batch of data samples
$\\{x^{*}_{1},\ldots,x^{*}_{b}\\}$ is selected from the unlabeled dataset
$D_{U}$ based on a generalized acquisition function $A$. This function
evaluates the candidate batches according to criteria that could include
uncertainty, diversity, clustering outcomes, or their combination. The
selection process can be formally described as:

$\\{x^{*}_{1},\ldots,x^{*}_{b}\\}=\underset{\\{x_{1},\ldots,x_{b}\\}\subseteq
D_{U}}{\mathrm{argmax}}\ A(\\{x_{1},\ldots,x_{b}\\},\phi),$ (6)

where parameters $\phi$ can include the current model parameters $\theta$
trained on the current labeled dataset $D_{L}$, as well as any additional
metrics or heuristics relevant to the batch selection strategy.

##  III Proposed Approach

In this section, we introduce our proposed approach, which includes the first
stage of the AL pipeline using unsupervised AL via x-vectors, followed by the
second stage that employs a Bayesian batch AL method, specifically tailored
for ASR. The proposed pipeline starts with unsupervised AL to pick an initial
dataset, setting the stage for more effective supervised AL. This chosen
dataset and the ASR model trained on it initiate the second AL stage, where
supervised AL is applied to select new, informative data samples for labeling.
These labeled samples are then added to the training set, allowing for the
retraining of the ASR model. This retrained model is then used to select
additional data for labeling in the new AL iteration.

###  III-A First Stage: Unsupervised Active Learning

The first AL stage begins with an unlabeled dataset
$D_{U}=\\{x_{i}\\}_{i=1}^{|D_{U}|}$. An x-vector DNN $f_{\theta_{x}}$ [28],
generates a corresponding set of x-vectors, producing one vector
$\mathbf{x}_{i}\in\mathbb{R}^{d}$ for each audio recording $x_{i}$, formalized
as:

$\\{\mathbf{x}_{i}\\}_{i=1}^{|D_{U}|}=f_{\theta_{x}}(D_{U}).$ (7)

These x-vectors are then clustered using the DBSCAN density-based spatial
clustering algorithm [37], chosen for its ability to identify clusters of a
configurable minimal size, thereby avoiding overly small clusters. DBSCAN
identifies core samples for high density areas and extends clusters based on
them. It groups together points that are nearby, while identifying and
labeling as outliers those located in low-density areas. This is beneficial
for isolating groups of audio samples that share common characteristics, such
as being from the same or similar speakers, or similar acoustic conditions.
Unlike approaches such as [20] that employ K-means for clustering i-vectors,
to the best of our knowledge, DBSCAN has not been applied to speech embeddings
in the context of AL. DBSCAN’s advantage lies in its ability to discover
clusters of arbitrary shape without defining the desired number of clusters,
its robustness to noise and outliers, and its insensitivity to the order in
which data points are processed. This grouping of audio samples is performed
based on the spatial relationships within the high-dimensional vector space
generated by x-vector DNN $f_{\theta_{x}}$. The result of DBSCAN clustering
can be represented as a set of clusters $\\{C_{k}\\}_{k=1}^{K}$, where each
cluster $C_{k}$ contains x-vectors that are more similar to each other than to
those in other clusters.

In subsequent steps of the first stage of AL, we enhance diversity among the
data samples selected for labeling by sampling from clusters of corresponding
x-vectors. To select an initial set of samples for labeling $S^{0}\subseteq
D_{U}$, we employ disproportionate cluster sampling, slightly favoring smaller
clusters to ensure diversity. Here, the superscript $0$ indicates the
iteration index within the entire AL process. This approach can be formalized
as:

$|S^{0}_{k}|=\left\lceil\alpha_{k}(|C_{k}|)\cdot\frac{|C_{k}|}{\sum_{i=1}^{K}|C_{i}|}\cdot|S^{0}|\right\rceil,$
(8)

where $|S^{0}_{k}|$ is the number of samples selected from a cluster $C_{k}$
uniformly at random, and $\alpha_{k}$ is an affine function of cluster size
$|C_{k}|$:

$\alpha_{k}(|C_{k}|)=\beta-\gamma\cdot\frac{|C_{k}|}{\sum_{i=1}^{K}|C_{i}|},$
(9)

This function introduces a variable slope to favor smaller clusters by
assigning larger values of $\alpha_{k}$ to them, and vice versa for larger
clusters. Here, $\beta$ and $\gamma$ are parameters that adjust the extent to
which cluster size influences sample selection. Finally, the set of samples
selected for labeling is equal to the union of samples selected from each
cluster: $S^{0}=\bigcup_{k=1}^{K}S^{0}_{k}$, while the remaining unlabeled
samples are given by the following formula:

$D_{U}^{0}=D_{U}\setminus S^{0}.$ (10)

Following the selection, samples from $S^{0}$ are labeled to form the initial
labeled dataset $D_{L}^{0}$, which, along with the ASR model $f_{\theta^{0}}$
trained on it, serves as input for the following supervised AL stage. The
pseudocode encapsulating the entire first stage of AL is presented in
Algorithm 1.

Algorithm 1 First stage: unsupervised AL

1:Input: unlabeled dataset $D_{U}=\\{x_{i}\\}_{i=1}^{|D_{U}|}$

2:for $i=1,2,\ldots,|D_{U}|$ do

3: Calculate x-vector $\mathbf{x}_{i}$ for each $x_{i}$ using (7)

4:end for

5:Cluster the set of x-vectors $\\{\mathbf{x}_{i}\\}_{i=1}^{|D_{U}|}$ using
DBSCAN

6:For each cluster $C_{k}$ from the resulting set $\\{C_{k}\\}_{k=1}^{K}$
determine $|S^{0}_{k}|$ using (8) and (9)

7:Form $S^{0}_{k}$ by selecting $|S^{0}_{k}|$ samples from each cluster
$C_{k}$ randomly

8:$S^{0}=\bigcup_{k=1}^{K}S^{0}_{k}$

9:Label selected samples $S^{0}$ and add them to $D_{L}^{0}$

10:Train an initial ASR model $f_{\theta^{0}}$ using $D_{L}^{0}$

11:Update the unlabeled dataset $D_{U}^{0}$ using (10)

We designed the unsupervised AL stage using x-vectors clustering for initial
data selection, and we reuse these x-vectors clusters to diversify sample
selection during the supervised AL phase. Additionally, this clustering-based
unsupervised AL method represents an improvement over the supervised AL
approach based on i-vector clustering [20], which requires tuning an extra
hyperparameter $\lambda$ to balance between diversity and uncertainty. In
contrast, our approach, applicable in both unsupervised and supervised AL
stages, achieves sample diversity without requiring adjustments to this
specific hyperparameter.

###  III-B Second Stage: Supervised Active Learning

The second stage of AL is an iterative supervised AL approach that integrates
concepts from both batch AL and Bayesian AL, tailored specifically for ASR
applications. In this approach, we enhance sample informativeness by selecting
samples on which the Bayesian committee shows the highest disagreement, and we
promote sample diversity by choosing the most informative samples from each
x-vectors cluster $C_{k}$ defined in Section III-A.

Bayesian AL for ASR: The direct application of Bayesian AL described in
Section II-C to ASR presents challenges due to the sequential nature of
speech. ASR differs from standard classification tasks due to its focus on
making sequential predictions, where each predicted phoneme in a transcription
is effectively a separate classification event. Since metrics designed for
fixed-length predictions, such as the variation ratio, are not suitable for
the sequence-to-sequence problems, we will employ WER metric to align our
Bayesian AL method with the ASR problem.

As outlined in Section II-C, Monte Carlo dropout serves as an approximation of
the exact integration of the model posterior over the parameter space. It can
also be interpreted as a Bayesian committee of models, where random dropout
during multiple forward passes yields a diversity of model topologies. This
ensemble of varied models outputs a range of possible transcriptions for a
given input. For each audio sample $x_{i}$ in the unlabeled dataset, we
conduct a series of $T$ stochastic forward passes using the ASR model
$f_{\theta^{h-1}}$ as a base, derived from the previous supervised AL
iteration. The index $h$ ranges from 1 to $H$, with $H$ denoting the total
number of supervised AL iterations planned within the second stage of AL. Each
stochastic forward pass, indexed by $t$, applies a unique MC dropout mask,
producing a transcription $y_{i}^{t,h}$ as follows:

$y_{i}^{t,h}=f_{\theta_{t}^{h-1}}(x_{i})\quad\text{for }t\in\\{1,\ldots,T\\},$
(11)

where $f_{\theta_{t}^{h-1}}$ indicates the ASR model altered with MC dropout
mask during the $t^{th}$ forward pass. Additionally, a reference transcription
$y_{i}^{r,h}$ from the ASR model without dropout to serve as a stable baseline
is given by:

$y_{i}^{r,h}=f_{\theta^{h-1}}(x_{i}).$ (12)

We then calculate the WER for each transcription $y_{i}^{t,h}$ against the
reference $y_{i}^{r,h}$:

$WER(y_{i}^{t,h},y_{i}^{r,h})=\frac{S_{i}^{r,t,h}+D_{i}^{r,t,h}+I_{i}^{r,t,h}}{N_{i}^{r,h}},$
(13)

where $S_{i}^{r,t,h}$ is the number of substitutions, $D_{i}^{r,t,h}$ is the
number of deletions, $I_{i}^{r,t,h}$ is the number of insertions required to
change the transcription $y_{i}^{t,h}$ into the reference $y_{i}^{r,h}$, and
$N_{i}^{r,h}$ is the total number of words in the reference transcription.
This calculation yields the individual WER for each pair of transcriptions,
which we average across all $T$ stochastic transcriptions to measure the
overall audio sample uncertainty $U^{h}(x_{i})$, which we use for guiding the
selection of samples for labeling in the subsequent iteration of the
supervised AL process:

$U^{h}(x_{i})=\frac{1}{T}\sum_{t=1}^{T}WER(y_{i}^{t,h},y_{i}^{r,h}).$ (14)

This approach was inspired by the Bayesian active summarization strategy in
[38], which uses a pairwise bilingual evaluation understudy (BLEU) score
variance among summary candidates gathered using $T$ stochastic forward passes
as an uncertainty metric. However, our method contrasts in its computational
efficiency, offering a complexity of $\mathcal{O}(T)$ for each input sample
compared to the $\mathcal{O}(T^{2})$ complexity from the pairwise BLEU score
calculations. While the BLEU score is used for evaluating the fluency and
adequacy of language translations, we choose WER as a more suitable metric for
ASR, as it directly measures transcription accuracy at the word level,
aligning with the precision requirements of ASR tasks.

Batch AL for ASR: In the proposed supervised AL framework tailored for ASR,
batch diversity is achieved by using x-vectors clusters
$\\{C_{k}\\}_{k=1}^{K}$ obtained in the initial unsupervised AL stage, while
the sample informativeness is ensured through the uncertainty metric
$U(x_{i})$ as formulated in (14). The number of samples $|S^{h}_{k}|$ to be
chosen from each cluster $k$ in iteration $h$ is calculated by
disproportionate cluster sampling, which favors smaller clusters slightly
more, as described by Equations (8) and (9). For each cluster $C_{k}$, we
select a subset of samples $S^{h}_{k}$ with the highest uncertainty values:

$S^{h}_{k}=\underset{\\{x_{1},\ldots,x_{|S^{h}_{k}|}\\}\subseteq
D_{U}}{\mathrm{argmax}}\ U^{h}(x_{i}).$ (15)

By collecting the top-ranked samples from all clusters, we form the batch
$S^{h}$ targeted for labeling in the current iteration of supervised AL.

$S^{h}=\bigcup_{k=1}^{K}S^{h}_{k}=\bigcup_{k=1}^{K}\underset{\\{x_{1},\ldots,x_{|S^{h}_{k}|}\\}\subseteq
D_{U}}{\mathrm{argmax}}\ U^{h}(x_{i}).$ (16)

Upon labeling selected samples $x_{i}\in S^{h}_{k}$ with the corresponding
transcriptions $y_{i}$, datasets of labeled and unlabeled samples are updated
in the following way:

$\displaystyle D_{L}^{h}=D_{L}^{h-1}\cup\\{(x_{i},y_{i})\mid x_{i}\in
S^{h}\\},$ (17a) $\displaystyle D_{U}^{h}=D_{U}^{h-1}\setminus S^{h}.$ (17b)

The pseudocode of the presented iterative procedure of the second stage of AL
is given in Algorithm 2. This procedure aligns with the general concept of
batch AL as previously defined, where the acquisition function $A$ given in
(6) balances the considerations of uncertainty and diversity within and across
the clusters.

Algorithm 2 Second stage: supervised AL

1:Input: remaining unlabeled data $D_{U}^{0}$, initial labeled data
$D_{L}^{0}$ and the trained ASR model $f_{\theta^{0}}$

2:for $h=1,2,\ldots,H$ do

3: for $i=1,2,\ldots,|D_{U}^{h-1}|$ do

4: Load unlabeled audio sample $x_{i}$

5: Generate Bayesian committee $\\{f_{\theta_{t}^{h-1}}\\}_{t=1}^{T}$ using

6: the base model $f_{\theta^{h-1}}$ and MC dropout

7: Generate Bayesian committee transcriptions $y_{i}^{t,h}$

8: using (11)

9: Generate reference transcription $y_{i}^{r,h}$ using (12)

10: Calculate $WER(y_{i}^{t,h},y_{i}^{r,h})$ using (13)

11: Calculate uncertainty $U^{h}(x_{i})$ using (14)

12: end for

13: For each cluster in $\\{C_{k}\\}_{k=1}^{K}$ determine $|S^{h}_{k}|$

14: using (8) and (9)

15: Form $S^{h}_{k}$ by selecting top $|S^{0}_{k}|$ uncertain samples

16: from each cluster $C_{k}$ using (15)

17: Form the batch $S^{h}=\bigcup_{k=1}^{K}S^{h}_{k}$, as in (16)

18: Label selected samples $S^{h}$

19: Update datasets of labeled and unlabeled samples,

20: $D_{L}^{h}$ and $D_{U}^{h}$, using (17a) and (17b)

21: Train the ASR model $f_{\theta^{h}}$ using $D_{L}^{h}$

22:end for

Performance optimization for large clusters: Although the proposed method for
calculating uncertainty has linear computational complexity with respect to
the number of samples in a cluster, execution time can still be substantial
for clusters with a large number of samples. To address this, we propose two
simple strategies to decrease execution time in such instances:

  * •

The most computationally demanding aspect of our approach, i.e., calculating
uncertainty using a committee, can be efficiently parallelized, since the
predictions of the committee members can be calculated independently. This
stands in contrast to common batch AL methods, such as BatchBALD [39], where
the mutual information between the model’s predictions on a batch of samples
and the model parameters must be calculated jointly. This inherently
sequential process prevents efficient parallelization, leading to increased
computational demands as batch and cluster sizes grow.

  * •

The research paper [40] demonstrates that for large unlabeled datasets, the
performance of AL remains unaffected when it is applied to only $25\%$ of the
unlabeled data, thereby achieving a considerable boost in efficiency. Our
method can leverage this to significantly enhance efficiency by applying it to
large unlabeled data clusters when necessary.

##  IV Results and Discussion

In this section, we first describe the datasets used, and outline the training
hyperparameters employed for the wav2vec 2.0 ASR model. We proceed by
presenting the results of the first, unsupervised AL stage, with the addition
of discussing the advantages of using x-vectors over i-vectors. Next, we
present the results of the second, supervised AL stage, additionally including
the discussion on the quality of the proposed Bayesian-based uncertainty
calculation. Finally, we review the results of the proposed approach on the
out-of-distribution test set.

The English language subsets of the Common Voice 1.0 [41] and the LibriSpeech
[42] ASR datasets were used for conducting experiments in this paper. The
Common Voice english-train subset has a total duration of around 15 hours of
annotated data, while the LibriSpeech train-clean-100 has around 100 hours of
annotated data. The distributions of the datasets are very different, as can
be seen in Table I. Notably, the Common Voice dataset is more heterogeneous,
containing almost five times as many speakers while featuring more than twice
fewer recordings compared to LibriSpeech. Within Common Voice, there are 426
speakers featured in singular audio files, whereas in LibriSpeech each speaker
contributes in multiple audio files, the lowest of which being 26 files. In
Common Voice, the highest occurrence of a single speaker is 687, whereas in
LibriSpeech, this figure is 166. The average duration of audio files is 2.85
times longer in LibriSpeech than in Common Voice.

When evaluating AL methods using heterogeneous test sets, performance can
often appear similar across different strategies, largely due to the high
variability inherent in these test sets. This variability can sometimes mask
differences in method effectiveness, and even random sampling may yield
comparably good performance, as demonstrated by [43]. To primarily assess the
real-world efficacy of the proposed AL pipeline, we have created a more
homogeneous test set containing only LibriSpeech data, hereafter referred to
as the primary test set. Another goal of this test set is to assess the
capability of the proposed AL approach to select a diverse data samples,
particularly focusing on those that are underrepresented in the unlabeled
dataset. This simulates a scenario that involves a specific class of speakers,
representing potential users of the ASR system being developed, who might
differ from the typical user base, aiming to ensure robust performance for
them despite their limited presence. To implement this test setup, we selected
ten distinct speakers from the LibriSpeech dataset, whose x-vector clusters do
not overlap. A portion of data samples from these speakers comprises the
primary test set, while the others are added to the unlabeled dataset. This
dataset also includes the heterogeneous Common Voice dataset, which is larger
by a factor of 20, effectively simulating the previously described scenario
involving underrepresented speakers. While we expect that our AL approach will
mostly select samples from the more represented unlabeled Common Voice data,
we aim to assess whether it promotes diversity by suggesting informative
training data samples from these small speaker groups and achieves strong
performance on the corresponding test samples.

Additionally, we also test our approach using the VoxPopuli dataset [44] to
evaluate its performance on both OOD and more heterogeneous data. In contrast
to the LibriSpeech and Common Voice datasets, which consist of speakers
reading aloud, the VoxPopuli dataset includes utterances from plenary sessions
of the European Parliament, accompanied by aligned transcriptions. The content
of these utterances, as well as the language specific to the European
Parliament domain, contributes to a distribution shift, ideal for testing in
an OOD setting. The OOD test set comprises 1842 audio files from 286 distinct
speakers, making it the most heterogeneous dataset with roughly twice fewer
audio files per speaker than Common Voice and nearly 20 times fewer than
LibriSpeech, making it well-suited for evaluating performance on heterogeneous
data. The maximum duration of each utterance is 20 seconds, with an average
duration of 9.63 seconds, situated between the averages of the Common Voice
and LibriSpeech datasets, while the summed duration over the whole dataset is
about five hours.

TABLE I: Comparison Between Datasets | Common Voice | LibriSpeech
---|---|---
duration [$s$] | 53832.70 | 362127.17
# audio files | 12135 | 28539
# speakers | 1057 | 251
duration mean [$s$] | 4.44 | 12.69
duration variance [$s^{2}$] | 1.79 | 12.78
duration skewness [$s^{3}$] | 1.11 | -1.39
duration kurtosis [$s^{4}$] | 3.17 | 0.98
speaker occurrence mean | 11.48 | 113.70
speaker occurrence variance | 2052.63 | 231.39
speaker occurrence skewness | 10.37 | -0.88
speaker occurrence kurtosis | 129.07 | 6.44

For our experiments, we utilized wav2vec 2.0 XLS-R model [45], which consists
of 300 million trainable parameters and was pretrained on 436 thousand hours
of unlabeled speech data. To ensure uniformity across all ASR training
experiments, the same conditions and hyperparameters were applied, with key
ones presented in Table II. Additionally, in all the experiments we used slope
parameters $\beta=0.095$ and $\gamma=0.0553$ in (9).333The code for
reproducing the experiments will be available following the paper’s
acceptance. Finally, during the implementation of experiments, we utilized the
MC dropout implementation from the Baal library [40] and parallelized
uncertainty calculations using the Ray package [46].

TABLE II: Training Hyperparameters for wav2vec 2.0 Model Hyperparameter | Value
---|---
Number of training epochs | 1000
Learning rate | 4e-5
Batch size | 32
Gradient accumulation steps | 2
Warm-up steps | 100
Dropout rate | 0.1
Loss function | CTC
Freeze encoder weights | Yes

###  IV-A First Stage - Unsupervised Active Learning

First, we will justify our choice of using x-vectors over i-vectors for
clustering unlabeled data samples in the proposed AL approach. This decision
is supported by the observation that x-vector representation, as opposed to
i-vectors, provides a clearer separation of speech recordings from diverse
data sources. To demonstrate this, we calculate Silhouette scores [47] for two
groups of data: one comprising samples from the Common Voice dataset and the
other from the whole LibriSpeech dataset.

The Silhouette score is a measure used to assess the separation between
clusters. It calculates each point’s average distance $a(i)$ to the points in
its own cluster and its distance $b(i)$ to the points in the nearest cluster
not containing it:

$s(i)=\frac{b(i)-a(i)}{\max(a(i),b(i))}.$ (18)

The overall Silhouette score is the mean of $s(i)$ across all points in the
dataset, which provides a clear measure of how distinct the clusters are. A
higher Silhouette score indicates better-defined clusters that are well-
separated from each other.

The Silhouette scores, which measure the separation between the Common Voice
and LibriSpeech datasets, were recorded at $0.0265$ for i-vectors and $0.0616$
for x-vectors, respectively. This considerable difference implies that
x-vectors manage to better distinguish between the speech recordings from the
two datasets. To further demonstrate this, we focus on two speakers, one from
the Common Voice and the other from the LibriSpeech dataset. Fig. 2 presents
the x-vector and i-vector representations of their speech recordings, reduced
to two dimensions using principal component analysis (PCA). This visual
comparison on a smaller example aligns with the conclusion of Silhouette
analysis on the entire datasets, showing a more distinct separation for the
x-vector representations. Thus, we opt for x-vector representations to
facilitate clustering with DBSCAN, which forms a fundamental part of promoting
diversity in our proposed approach. It is noteworthy that x-vectors were not
specifically trained on these two datasets, indicating their robust
generalization ability.

$-50$$0$$50$$100$$150$$-50$$0$$50$First principal componentSecond principal
componentx-vectorsSpeaker 1Speaker 2 (a)

$-4$$-2$$0$$2$$4$$6$$-5$$0$$5$First principal componentSecond principal
componenti-vectorsSpeaker 1Speaker 2 (b)

Figure 2: The upper plot displays x-vectors and the lower plot shows i-vectors
of speech recordings from two speakers, both reduced to two dimensions using
PCA.

To demonstrate the effectiveness of the proposed unsupervised AL approach, we
apply Algorithm 1 to select $|S^{0}|=643$ unlabeled samples (or approximately
one hour of total duration), pair them with the corresponding labels, and use
them as a training set for the wav2vec 2.0 ASR model. For comparison, we also
randomly sampled another $643$ samples and trained the ASR model under the
same conditions. The proposed unsupervised AL approach outperforms the random
sampling, achieving lower WER and CER on the primary test set as shown in
Table III, indicating its superior diversity in sample selection.

TABLE III: Comparison of the Proposed Unsupervised AL Approach With a Random Sampling Strategy Method | WER | CER
---|---|---
Random sampling | 0.2312 | 0.0678
Proposed approach | 0.2119 | 0.0613

###  IV-B Second Stage - Supervised Active Learning

Next, we present the results of the second, supervised AL stage. In all
experiments, we employed $T=20$ dropout models within a Bayesian committee,
following the recommendation of at least ten MC dropout iterations as
suggested by [35]. After the initial unsupervised AL stage, we conducted $H=3$
supervised AL iterations, as described in Algorithm 2. During these
iterations, we repeatedly trained wav2vec 2.0 ASR models, which identified and
selected the most informative unlabeled samples based on the uncertainty
metric. These samples were chosen from each x-vectors cluster to ensure
diversity, resulting in a total of $|S^{h}|=643$ selected samples, which were
then paired with a label and added to the training set.

We compared our supervised AL approach with two alternative methods: the
signal-model committee approach (SMCA) proposed in [22] and an iterative
random sampling strategy. The SMCA includes a committee made up of the base
model and one model modified using dropout, where uncertainty is measured by
the character matching error rate (CMER) between the transcriptions produced
by these two models. In each iteration of both SMCA and the random sampling, a
total of $|S^{h}|=643$ samples are selected, with the SMCA’s selections
comprising the most uncertain samples. All wav2vec 2.0 ASR models used in our
approach, SMCA, and random sampling were trained under the same conditions to
ensure consistency in performance evaluation.

Before presenting the results across all supervised AL iterations, we assess
the quality of uncertainty estimation of our Bayesian AL method by analyzing
the Pearson correlations between the calculated uncertainties $U^{h}(x_{i})$
for each primary test set sample and the WERs from the ASR model
transcriptions relative to the ground-truth labels. These uncertainties are
compared with those obtained using the SMCA method [22] and entropy-based
uncertainties [32]. Entropy-based uncertainty is calculated by averaging the
token entropy derived from the ASR model softmax layer distribution across the
entire transcription. For these comparisons, we utilized the initial ASR model
$f_{\theta^{0}}$ trained on the labeled dataset selected during the first AL
stage, $D_{L}^{0}$. The results are given in Table IV. It is evident that our
approach achieves the highest Pearson correlation between the uncertainties
and the actual WERs, indicating superior sample selection for enhancing model
training. As expected, the entropy-based uncertainty method performed the
least effectively, likely due to DNN overconfidence in predictions generated
from the softmax layer [15], resulting in low entropy for both certain end
uncertain samples. To display the relationship between the WERs and
uncertainties calculated using our approach in a greater detail, these
variables are presented in a scatter plot shown in Fig. 3.

TABLE IV: Pearson Correlations Between Test Set WERs and Different Uncertainty Measures Uncertainty | Pearson correlation
---|---
Proposed method | 0.5578
SMCA | 0.4172
Entropy | 0.3795
$0$$0.2$$0.4$$0.6$$0$$0.2$$0.4$$0.6$WERProposed uncertainty
$U^{\mathit{1}}(x_{i})$ Figure 3: The relationship between WER and the
proposed uncertainty on test set samples. The red line illustrates a linear
fit to the provided relationship.

Fig. 4 illustrates the comparison of WER, expressed as a percentage, on the
primary test set after training ASR models with additional newly labeled
samples suggested by our method, SMCA, and random sampling across successive
AL iterations. Our approach consistently demonstrates lower test set WER in
each AL iteration compared to SMCA and random sampling approaches, indicating
superior performance. SMCA effectively estimates uncertainty, as detailed in
Table IV, but it is outperformed by the iterative random sampling approach.
This occurs because SMCA selects and adds samples with high uncertainty to the
training set, yet these tend to be less varied and are concentrated in
specific areas of the speech data. Random sampling, on the other hand, selects
data to be labeled and added to the training set with greater diversity than
SMCA, confirming the competitiveness of random sampling under certain
conditions as mentioned in [43]. Given that the primary test set includes
underrepresented groups of speakers from the training set, the diversity-
driven approach of our method, especially our use of disproportionate cluster
sampling that favors smaller clusters, is particularly crucial. Our method
combines the precision of uncertainty-based selection with the advantages of
diversity, which together account for its enhanced performance.

Additionally, the horizontal black line in Fig. 4 represents the WER when the
model is trained with the whole dataset. This serves as a benchmark compared
to the much smaller datasets selected through AL iterations, indicating the
performance achievable with the current hyperparameters and available training
data. The proximity of our AL method’s performance to this benchmark
underscores the efficiency of using AL strategies: achieving competitive
performance while training the ASR model with only $19.98\%$ of the whole
training dataset. This comparison emphasizes the benefits of our AL approach,
which leverages a smaller, more targeted subset of the whole dataset, thereby
reducing labeling effort, training time and resource consumption while
maintaining high accuracy.

$0$$1$$2$$3$$8$$10$$12$$14$$16$$18$$20$$22$$24$$26$$28$Iteration ($h$)Test Set
WER (%)Proposed approachSMCARandom samplingThe whole dataset Figure 4: Primary
test set WER (%) for trained ASR models over AL iterations, comparing the
proposed approach, SMCA, random sampling, and the whole dataset baseline.

To better understand how uncertainty is reduced over time using our approach,
we observe the uncertainties calculated by the Bayesian committee after each
AL iteration. This allows us to understand better how the proposed data
selection refines ASR model confidence progressively. Fig. 5 presents whisker
plots that demonstrate the uncertainty distributions of all unlabeled samples
after each AL iteration. The consistent decrease in uncertainty across the
iterations confirms that our approach effectively enhances model certainty
over time. Notably, the diminishing length of the upper whisker over
successive iterations indicates a significant reduction in higher uncertainty
samples.

$1$$2$$3$$0$$0.5$Iteration ($h$)Uncertainty Figure 5: The distribution of
uncertainties calculated using the proposed approach for all unlabeled samples
in each AL iteration.

###  IV-C Results on Out-of-Distribution Test Set

To further evaluate our approach, we tested its performance on the
heterogeneous VoxPopuli test set [44], where the domain-specific content and
language induce a notable shift in data distribution. The evaluation outcome
is presented in Fig. 6, which displays the OOD test set WER for each ASR model
trained in Section IV-B, including models trained with data iteratively
selected by the proposed approach, SMCA, and random sampling, as well as a
model trained on the whole training set. Expectedly, WERs in this test
scenario are higher compared to those in Section IV-B due to the significant
distribution shift in the test data and the relatively brief training duration
paired with relatively small training set sizes. The horizontal black line
denotes the model trained on the whole dataset, serving as a benchmark
compared to the much smaller datasets selected through AL iterations; its high
WER confirms the distribution shift in the OOD dataset. The results indicate
that while the performances of the proposed approach, SMCA, and random
sampling are relatively close, our approach demonstrates superior results on
the OOD test set. The increasing difference in OOD test set WER can be
attributed to the diverse training set samples suggested by our approach, with
the advantage becoming more pronounced as we further add newly labeled
samples.

$0$$1$$2$$3$$24$$26$$28$$30$$32$$34$$36$$38$Iteration ($h$)Test Set WER
(%)Proposed approachSMCARandom samplingThe whole dataset Figure 6: Out-of-
distribution test set WER (%) for ASR models previously trained in each AL
iteration, comparing the proposed approach, SMCA, random sampling, and the
whole dataset baseline.

##  V Conclusion

This study developed and implemented a two-stage AL pipeline for ASR, aimed at
enhancing model training performance while minimizing labeling effort. The
first AL stage utilizes unsupervised AL with x-vectors clustering to select a
diverse initial subset of unlabeled speech samples. Once labeled, these
samples form a high-quality and diverse dataset that establishes a robust
foundation for the subsequent supervised AL stage. Our results show that this
initial selection significantly improves the test set performance of the
initial ASR model compared to the random sampling approach, indicating
effective dataset curation.

In the second AL stage, we adopt a supervised AL strategy that features a
batch AL method customized for ASR. This stage promotes further diversity
through x-vectors clustering and also incorporates a Bayesian AL method with
MC dropout, specifically adapted for uncertainty estimation in ASR. When
evaluated on a test set representing underrepresented speaker groups, our
supervised AL method consistently outperforms competing methods, underscoring
the benefits of promoting diversity through x-vectors clustering.
Additionally, the proposed approach outperforms competing methods on the
heterogeneous and out-of-distribution test set, with the advantage increasing
with each new AL iteration.

## Acknowledgment

The computational resources required for the experiments in this paper were
provided by the Serbian National Platform for Artificial Intelligence, an
initiative of the Office for Information Technologies and e-Government.

## References

  * [1] C. McMullin, “Transcription and qualitative methods: Implications for third sector research,” _Voluntas: Int. J. Volunt. Nonproﬁt Organ._ , vol. 34, no. 1, pp. 140–153, 2023.
  * [2] M. Mazumder _et al._ , “DataPerf: Benchmarks for Data-Centric AI Development,” in _Proc. 36th Int. Conf. Neural Inf. Process. Syst._ , 2023, pp. 5320–5347.
  * [3] S. Kumar, S. Datta, V. Singh, S. K. Singh, and R. Sharma, “Opportunities and Challenges in Data-Centric AI,” _IEEE Access_ , vol. 12, pp. 33 173–33 189, 2024.
  * [4] Y. Wu, R. Zhang, and A. Rudnicky, “Data selection for speech recognition,” in _2007 IEEE Workshop Autom. Speech Recognit. Understand._ , pp. 562–565.
  * [5] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating Training Data Makes Language Models Better,” in _Proc. 60th Ann. Meeting Assoc. Comput. Linguist._ , 2022, pp. 8424–8445.
  * [6] C. Chai, J. Wang, Y. Luo, Z. Niu, and G. Li, “Data Management for Machine Learning: A Survey,” _IEEE Trans. Knowl. Data Eng._ , vol. 35, no. 5, pp. 4646–4667, 2023.
  * [7] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang, “A Survey of Deep Active Learning,” _ACM Comput. Surv._ , vol. 54, no. 9, 2021.
  * [8] G. Riccardi and D. Hakkani-Tur, “Active learning: theory and applications to automatic speech recognition,” _IEEE Trans. Speech Audio Process._ , vol. 13, no. 4, pp. 504–511, 2005.
  * [9] D. Yu, B. Varadarajan, L. Deng, and A. Acero, “Active learning and semi-supervised learning for speech recognition: A unified framework using the global entropy reduction maximization criterion,” _Comput. Speech Lang._ , vol. 24, pp. 433–444, 2010.
  * [10] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representations,” in _Proc. 34th Int. Conf. Neural Inf. Process. Syst._ , 2021, pp. 12 449–12 460.
  * [11] Q. Zhang _et al._ , “Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss,” in _Proc. 2020 IEEE Int. Conf. Acoust. Speech Signal Process._ , pp. 7829–7833.
  * [12] J. Bang, H. Kim, Y. Yoo, and J.-W. Ha, “Boosting active learning for speech recognition with noisy pseudo-labeled samples,” vol. arXiv: 2006.11021, 2020.
  * [13] T. Drugman, J. Pylkkönen, and R. Kneser, “Active and Semi-Supervised Learning in ASR: Benefits on the Acoustic and Language Models,” in _Proc. Interspeech_ , 2016, pp. 2318–2322.
  * [14] M. Chellapriyadharshini, A. Toffy, S. R. K. M., and V. Ramasubramanian, “Semi-supervised and Active-learning Scenarios: Efficient Acoustic Model Refinement for a Low Resource Indian Language,” in _Proc. Interspeech_ , 2018, pp. 1–5.
  * [15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _Proc. 34th Int. Conf. Mach. Learn._ , 2017, p. 1321–1330.
  * [16] J. Huang, R. Child, V. Rao, H. Liu, S. Satheesh, and A. Coates, “Active learning for speech recognition: The power of gradients,” in _Proc. 30th Int. Conf. Neural Inf. Process. Syst._ , 2016, pp. 1–5.
  * [17] Y. Yuan, S.-W. Chung, and H.-G. Kang, “Gradient-based Active Learning Query Strategy for End-to-end Speech Recognition,” in _Proc. 2019 IEEE Int. Conf. Acoust. Speech Signal Process._ , pp. 2832–2836.
  * [18] J. Luo, J. Wang, N. Cheng, and J. Xiao, “Loss Prediction: End-to-End Active Learning Approach For Speech Recognition,” in _2021 Int. Joint Conf. Neural Netw._ , pp. 1–7.
  * [19] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-End Factor Analysis for Speaker Verification,” _IEEE Trans. Audio Speech Lang. Process._ , vol. 19, no. 4, pp. 788–798, 2011.
  * [20] K. Malhotra, S. Bansal, and S. Ganapathy, “Active Learning Methods for Low Resource End-to-End Speech Recognition,” in _Proc. Interspeech_ , 2019, pp. 2215–2219.
  * [21] Y. Long, H. Ye, Y. Li, and J. Liang, “Active Learning for LF-MMI Trained Neural Networks in ASR,” in _Proc. Interspeech_ , 2018, pp. 2898–2902.
  * [22] Sun, Xiusong and Wang, Bo and Liu, Shaohan and Lu, Tingxiang and Shan, Xin and Yang, Qun, “Lmc-smca: A new active learning method in asr,” _IEEE Access_ , vol. 9, pp. 37 011–37 021, 2021.
  * [23] J. Fu and K. Ru, “A Dropout-Based Single Model Committee Approach for Active Learning in ASR,” in _2019 IEEE Workshop Autom. Speech Recognit. Understand._ , pp. 16–22.
  * [24] A. R. Syed, A. Rosenberg, and E. Kislal, “Supervised and unsupervised active learning for automatic speech recognition of low-resource languages,” in _Proc. 2016 IEEE Int. Conf. Acoust. Speech Signal Process._ , pp. 5320–5324.
  * [25] Z. Zheng, Z. Ma, Y. Wang, and X. Chen, “Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for Automatic Speech Recognition,” in _Proc. Interspeech_ , 2023, pp. 3307–3311.
  * [26] A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, “Unsupervised Speech Recognition,” in _Proc. 34th Int. Conf. Neural Inf. Process. Syst._ , 2021, pp. 27 826–27 839.
  * [27] X. Liu _et al._ , “Self-Supervised Learning: Generative or Contrastive,” _IEEE Trans. Knowl. Data Eng._ , vol. 35, no. 1, pp. 857–876, 2023.
  * [28] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-Vectors: Robust DNN Embeddings for Speaker Recognition,” in _Proc. 2018 IEEE Int. Conf. Acoust. Speech Signal Process._ , pp. 5329–5333.
  * [29] R. M. Neal, _Bayesian Learning for Neural Networks_. Berlin, Heidelberg: Springer-Verlag, 1996.
  * [30] Y. Gal and Z. Ghahramani, “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning,” in _Proc. 33rd Int. Conf. Mach. Learn._ , 2016, pp. 1050–1059.
  * [31] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _Proc. 23th Int. Conf. Mach. Learn._ , 2006, p. 369–376.
  * [32] C. Aggarwal, X. Kong, Q. Gu, J. Han, and P. Yu, _Active learning: A survey_. CRC Press, Jan. 2014.
  * [33] I. Dagan and S. P. Engelson, “Committee-Based Sampling For Training Probabilistic Classifiers,” in _Machine Learning Proceedings 1995_ , A. Prieditis and S. Russell, Eds. San Francisco (CA): Morgan Kaufmann, 1995, pp. 150–157.
  * [34] N. Houlsby, F. Huszar, Z. Ghahramani, and M. Lengyel, “Bayesian Active Learning for Classification and Preference Learning,” vol. arXiv: 1112.5745, 2011.
  * [35] Y. Gal, “Uncertainty in deep learning,” Ph.D. dissertation, University of Cambridge, 2016.
  * [36] G. Citovsky, G. DeSalvo, C. Gentile, L. Karydas, A. Rajagopalan, A. Rostamizadeh, and S. Kumar, “Batch Active Learning at Scale,” in _Proc. 34th Int. Conf. Neural Inf. Process. Syst._ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 11 933–11 944.
  * [37] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, “DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN,” _ACM Trans. Database Syst._ , vol. 42, no. 3, jul 2017.
  * [38] A. Gidiotis and G. Tsoumakas, “Bayesian active summarization,” _Comput. Speech Lang._ , vol. 83, p. 101553, 2024.
  * [39] A. Kirsch, J. van Amersfoort, and Y. Gal, “BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning,” in _Proc. 32nd Int. Conf. Neural Inf. Process. Syst._ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., 2019, p. 7024–7035.
  * [40] P. Atighehchian, F. Branchaud-Charron, and A. Lacoste, “Bayesian active learning for production, a systematic study and a reusable library,” vol. arXiv: 2006.09916, 2020.
  * [41] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common Voice: A Massively-Multilingual Speech Corpus,” in _Proc. 12th Lang. Resour. Eval. Conf._ , Marseille, France, May 2020, pp. 4218–4222.
  * [42] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _Proc. 2015 IEEE Int. Conf. Acoust. Speech Signal Process._ , pp. 5206–5210.
  * [43] P. Munjal, N. Hayat, M. Hayat, J. Sourati, and S. Khan, “Towards Robust and Reproducible Active Learning using Neural Networks,” in _2022 Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ Los Alamitos, CA, USA: IEEE Computer Society, jun, pp. 223–232.
  * [44] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation,” in _Proc. 59th Ann. Meeting Assoc. Comput. Linguist._ , 2021.
  * [45] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,” in _Proc. Interspeech_ , 2022, pp. 2278–2282.
  * [46] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: a distributed framework for emerging AI applications,” in _Proc. 13th USENIX Conf. Operat. Syst. Des. Implement._ , 2018, p. 561–577.
  * [47] P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” _J. Comput. Appl. Math._ , vol. 20, pp. 53–65, 1987.