# TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End
ASR

Nagarathna Ravi*, Thishyan Raj T*, Vipul Arora This paper was produced by the
MADHAV Lab ASR research group in IIT Kanpur (e-mail: {rathna, thishyan20,
vipular}@iitk.ac.in). *-Equal ContributionManuscript received …. .., ….;
revised …. .., …..

###### Abstract

Confidence estimation of predictions from an End-to-End (E2E) Automatic Speech
Recognition (ASR) model benefits ASR’s downstream and upstream tasks. Class-
probability-based confidence scores do not accurately represent the quality of
overconfident ASR predictions. An ancillary Confidence Estimation Model (CEM)
calibrates the predictions. State-of-the-art (SOTA) solutions use binary
target scores for CEM training. However, the binary labels do not reveal the
granular information of predicted words, such as temporal alignment between
reference and hypothesis and whether the predicted word is entirely incorrect
or contains spelling errors. Addressing this issue, we propose a novel
Temporal-Lexeme Similarity (TeLeS) confidence score to train CEM. To address
the data imbalance of target scores while training CEM, we use shrinkage loss
to focus on hard-to-learn data points and minimise the impact of easily
learned data points. We conduct experiments with ASR models trained in three
languages, namely Hindi, Tamil, and Kannada, with varying training data sizes.
Experiments show that TeLeS generalises well across domains. To demonstrate
the applicability of the proposed method, we formulate a TeLeS-based
Acquisition (TeLeS-A) function for sampling uncertainty in active learning. We
observe a significant reduction in the Word Error Rate (WER) as compared to
SOTA methods.

###### Index Terms:

Confidence Estimation, Automatic Speech Recognition, Human-in-the-Loop
Annotation, Active Learning, Pseudo-labeling

##  I Introduction

Calibrating an End-to-End (E2E) Automatic Speech Recognition (ASR) model’s
predictions using confidence scores can aid downstream and upstream tasks
[1][2][3] such as domain adaptation, active learning, keyword spotting,
dialogue systems, system combination, machine translation, information
retrieval, semi-supervised learning, smart speakers, automatic error
correction, dis-fluency detection, speaker adaptation, and enhancing the ASR
dataset with diverse set of audio-transcript pairs.

Figure 1: Class-prob. based Scores of Correct and Wrong Words

The Confidence Estimation Model (CEM) can be an algorithm based on statistics
[4] or an auxiliary Machine Learning (ML) model [5]. Statistical confidence
measures include class-probability [6][7] of the output label and entropy [4]
of the class-probabilities. These measures assume that a well-learned model
can accurately project the distance between the estimate and the label
boundaries. However, as shown in Figure 1 (Class-probability-based confidence
scores of ASR model trained using IISc-Mile-Tamil train dataset and tested on
IISc-Mile-Tamil test-set [8][9][10]), there is a substantial overlap between
the SoftMax scores of accurate and incorrect predictions due to the general
overconfidence nature of neural models. Several State-Of-The-Art (SOTA)
[5][11] methods train an auxiliary CEM model using binary labels: score ’1’ is
assigned to the correct label, and score ’0’ is set to the incorrect label as
target values. Levenshtein alignment between the actual and hypothesis
transcripts aids in identifying if the label is correct or wrong. However,
using binary labels does not expose the predictions’ granular information,
such as the temporal alignment between the reference and the hypothesis. It
also does not reveal whether the predicted word is entirely incorrect or
contains spelling errors. Consider the example in which character alignments
determine the target confidence scores of characters. When the input does not
share the domain as the training data, alignments are ambiguous, as seen in
the example (/ denotes the token boundary),
Ref.: (/s/o/n/a/ /bh/ee/ /h/ai/)
Hyp.: (/ba/s/a/ /aa/n/a/ /sa/ch/ /bh/ee/ /h/ai/)
Target Scores: $[0,1,0,0,0,1,1,0,0,0,1,1,1,1,1,1]$
In the preceding example, the reference’s first character  (s) is aligned with
the second character  (s) in the hypothesis. Similarly, the third character
(n) is aligned with the sixth character  (n). However, the aligned characters
are different, which shows that the method does not utilize temporal
information.
Consider yet another example which assigns binary labels to words,
Ref.:  (/hamaare/ /lie/ /prakaashit/ /khushee/ /hai/)
Hyp.:  (/hamaare/ /li/ /aprataashit/ /khushee/ /haiye/)
Target Scores: $[1,0,0,1,0]$
The second word suffers one deletion. However, since the desired score is
zero, the ancillary CEM attempts to penalize the estimation and push the score
to zero.

The need for confidence estimation in ASR for downstream and upstream tasks,
issues with statistical confidence metrics, and insufficient information
exposed by using binary labels as targets in training CEM motivate us to
further probe into building a robust CEM. This article presents a novel CEM
mechanism to estimate the confidence of the words predicted by the ASR model.
The mechanism involves training an auxiliary Word-Level Confidence (WLC)
model. The WLC model is trained with the intermediate outputs from the trained
ASR model and a novel target confidence score $\in[0,1]$. Training a robust
WLC model requires balanced data points covering the target confidence score
range. However, a well-trained ASR model generates limited words with errors,
so the WLC model tends to be biased due to an imbalance in the dataset. SOTA
methods like [11] do not have the methodology to handle an imbalance in the
dataset. A few SOTA [5] methods introduce time and frequency masks in the
audio spectrograms to introduce errors in the inference. However, it is
uncertain whether getting data points across the target score range is
possible. Hence, we propose to use the shrinkage loss [12] function to train
the WLC model. Shrinkage loss uses the sigmoid function to modulate the
squared loss. Modulation aids in learning the infrequent samples and penalizes
the significance of loss from frequent instances, which is our case’s correct
predicted words. We propose an acquisition function based on the estimated
confidence scores from the WLC model for the active learning ASR downstream
task to ascertain the usability of the proposed confidence model. The
acquisition function chooses reliable pseudo labels and gives them to the
annotators for correction, thereby creating a Human-In-The-Loop (HITL)
architecture. The corrected transcripts are added to the training pool, ready
for retraining the ASR model to adapt to the new data domain. In particular,
the key contributions of our work are as follows.

  * •

We propose a novel confidence estimation metric called the Temporal-Lexeme
Similarity score, abbreviated as TeLeS, to train the auxiliary WLC model.
$TeLeS\in[0,1]$. At frame level, the temporal similarity score reveals the
misalignment between timestamps of the words in ground truth and hypothesis.
The lexeme similarity score considers the number of deletions and insertions
in the mismatched words in ground truth and inference. The combination of
temporal and lexeme-level information aids in teaching the TeLeS-WLC model
about the acoustic and decoder shifts that result in incorrect output.

  * •

To account for the data imbalance issue in training TeLeS-WLC model, we
include shrinkage loss in the training architecture to aid the model in
focusing on hard samples and avoid getting biased due to frequently cited
samples. To the best of our knowledge, no works on confidence estimation have
used shrinkage loss to account for the data imbalance issue.

  * •

We propose the TeLeS-based Acquisition, TeLeS-A method to choose reliable
samples for active learning-based HITL training. TeLeS aids in selecting
informative samples for labelling and pseudo-labels for inclusion in the
training loop.

  * •

We provide an intensive evaluation of proposed methods in three different
languages, viz., Hindi, Tamil, and Kannada. We evaluate the performance of the
proposed methods utilizing a range of metrics, and the results are compared
with SOTA methodologies.

  * •

We also release the Hindi dataset prepared by our in-house annotators as an
open-source resource. Codes and sample audios of Hindi dataset are available
online in https://github.com/madhavlab/2023_teles_wlc

The remainder of the paper is structured as follows. Section II gives an
overview of the existing methods in the literature that estimate the
confidence of the outputs from the ASR model. Section III elaborates on the
proposed methodology in detail. Section IV details the experimental design and
outcomes gained while evaluating the TeLeS-WLC model. The results of using
TeLeS-A-based active learning are presented in Section V. The final section
concludes and gives future pointers for advancing the research further.

##  II Related Work

###  II-A Confidence Estimation

Confidence estimation in ASR has been actively explored for a few years.
Class-probability score [6][7] and its scaled version [13] are naive
confidence estimation metrics. Entropy over prediction probabilities is also a
statistical confidence measure [13][4].

Huang et al. [14] propose training a deep neural network to predict whether or
not the words or utterances are in-domain. Word or utterance identification
data derived from the training corpus’s frequency statistics is used to train
the network. However, a train data corpus cannot always represent real-time
statistics.

Authors in [15] predict confidence using Recurrent Neural Networks (RNN). The
acoustic and Language Models’ (LM) input features are extracted and fed to RNN
layers. Del-Agua et al. [16] train Deep Bidirectional RNN (DBRNN) using word-
level predictor features and adapt the trained model to new speakers using a
conservation strategy. Li et al. [3] propose estimating confidence using a
BRNN that takes inputs like duration, embeddings, and word posterior
probabilities from the ASR model. [17] suggests extracting acoustic and
hypothesis embedding to train a confidence estimator. The input Log Filter
Bank Energy (LFBE) features are given to the Long Short Term Memory (LSTM)
layer to get the acoustic embedding of each token. The authors extract each
token’s final LSTM decoder layer output as hypothesis embedding. The authors
in [18] also employ a variant of the RNN to assess the accuracy of black-box
ASR predictions. Jeon et al. [19] combine a heterogeneous word confusion
network with a bidirectional lattice RNN to determine the most confident
sequence. [20] proposes integrating ELECTRA with ASR. The proposed model
leverages the notion of masked LM to learn the context and detect whether the
output tokens from ASR are correct or have been substituted with another
token. Ogawa et al. [11] extract multiple features from the ASR model,
including the Connectionist Temporal Classification (CTC) score [21],
attention vectors, LM scores, and the token’s SoftMax probability score. A
bidirectional-LSTM network gets the extracted features to estimate confidence.
The authors use class-balanced cross-entropy loss to handle data imbalance in
CEM train data. The method in [22] extracts acoustic and lexical information
from the ASR model and parses them into a sequence layer to calibrate the
predicted tokens. The word level score is calculated by averaging the token
level scores. The proposed methodologies in [5][23] extract various features
from the ASR model, which include attention, embedding, SoftMax scores, and
top-k SoftMax probabilities. The features are concatenated to serve as input
to fully connected and residual-energy-based models in [5] and [23],
respectively. To assess the confidence of predictions from out-of-domain
speech signals as well, the methodology in [24] integrates the LM with both
methods described in [5][23]. To handle the CEM data imbalance, [5], [23], and
[24] use spectrogram augmention masking techniques to introduce errors in the
ASR predictions during CEM training. The approach in [25] employs a
methodology similar to that used in [5][23]. CEM in [25] collects features
from the ASR model and calculates cross and self-attention of the extracted
features. Attention vectors serve as inputs to the neural network for
confidence estimation.

The techniques mentioned above take some features from the ASR model and parse
them to auxiliary confidence estimation networks. Typically, the methods
construct alignments at the token or word level between the ground truth
transcript and the hypothesis. If the alignment between two words/tokens is
correct, a score of ’1’ is assigned to the hypothesis index; otherwise, a
score of ’0’ is assigned. The sequence of binary labels generated serves as
the target confidence score for the auxiliary models. During training, the
model attempts to reduce the gap between the output of the final layer and the
target binary labels.

###  II-B HITL-Active Learning ASR

This section discusses the HITL active learning approaches in the ASR domain.
The LM Certainty (LMC) approach [26] uses LM perplexity to choose informative
samples. The authors train an n-gram LM and feed the ASR predictions to the LM
to obtain a perplexity score. Finally, the samples above a threshold are
marked as informative data. Signal-Model Committee Approach (SMCA) [26] varies
the dropout in the trained model and computes the character-matching error
rate between the inferences generated by models with different dropouts. The
dataset tuples with error rates above a threshold are marked as informative
samples. Methods in [27] and [28] use length-normalized path probability of
the ASR predictions as the active learning acquisition metric. The
methodologies choose the samples with low path probability to annotate as they
are more informative.

###  II-C Advantages over SOTA

Next, we highlight some key advantages the proposed method provides over the
SOTA methodologies.

  * •

ASR model attempts to overfit the data during training and tends to be
overconfident in predicting unobserved data [29][30]. So, the scores based on
prediction probabilities in [6], [7], [13], and [4] are an unreliable measure.
The TeLeS score uses the ground truth to obtain the target scores for training
WLC model; hence, the overconfident ASR predictions do not impact the
confidence estimation.

  * •

[5], [11], [15]-[25] use binary target scores. TeLeS score gives additional
information when compared to binary targets. Hence, TeLeS target scores reveal
the confidence of the predictions more effectively than the binary target
scores.

  * •

[15], [16], [3], [17], [18], [22] and [25] do not have methodologies to handle
data imbalance in training CEM. Whereas, we use shrinkage loss that focuses on
infrequent samples while training the WLC model.

  * •

The active learning LMC approach in [26] uses LM trained on the source
dataset. The LMC approach may not capture the informative samples if there is
a domain mismatch. SMCA [26] and methods in [27] and [28] depend on the ASR
prediction probabilities. Our approach uses estimated confidence, computed
based on intermediate states from the ASR model and TeLeS target score.

##  III Proposed Methodology

This section begins by introducing the task of speech recognition and
estimating the confidence of recognition. Further, we introduce the confidence
metric TeLeS. Then, we describe our TeLeS-WLC training approach that learns
the confidence measures. Finally, we present the TeLeS-A function that can be
utilised for active learning tasks in ASR.

###  III-A Problem Formulation

Consider the set of audio-transcript pairs
$\mathcal{D}=\\{(\mathbf{\tilde{x}}_{i},\mathbf{z}_{i})\\}_{i=1}^{K}$, where
$\mathbf{\tilde{x}}_{i}$ is a raw speech audio file and $\mathbf{z}_{i}$ is
the ground-truth transcript of $\mathbf{\tilde{x}}_{i}$. Speech-to-text
conversion comprises taking $\mathbf{\tilde{x}}_{i}$ or its spectral
representation and extracting from it basic sound units ($L$), such as
phonemes, word-pieces, graphemes, characters and so on. Due to the quasi-
stationary nature of speech signals, it is preferable to predict the sound
units present in small overlapping time-frames, over which the characteristics
of the speech signal remains stationary. In speech processing, MelSpectrogram
feature representation is one of the most popular choices, as this represents
the energies of the frequencies present in $\mathbf{\tilde{x}}_{i}$, in
accordance with the Mel frequency scale.

The MelSpectrogram representation of the audio signals and labels form the
train dataset
$\mathcal{D^{\prime}}=\\{(\text{X}_{i},\mathbf{z}_{i})\\}_{i=1}^{K}$. From
this point onward, we consider each
$(\text{X},\mathbf{z})\in\mathcal{D}^{\prime}$ as a general data point used
for training our model.
$\text{X}=\left[\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}\right]$
with each $\mathbf{x}_{t}\in\mathbb{R}^{D}$ ($\mathbf{x}_{t}$ is the
$D$-dimensional MelSpectrogram representation for the $t^{\text{th}}$ frame,
$T$ is the number of frames for each X). There is a corresponding
$\mathbf{z}=\left[z_{1},z_{2},\ldots,z_{U}\right]$ which is the label sequence
for X and $U\leq T$ is the length of the label sequence. Each $\mathbf{z}$ is
from the set of all possible sequences made up of $L$ and each $z_{u}\in L$.

Models can be trained to classify the frames of X as $z_{u}$. In our case, we
train an Attention-based Deep Neural Network (A-DNN) model $\mathcal{F}$ with
parameters $\theta$ using CTC loss [21] to extract frame-wise context
information and predict the utterance. $\mathcal{F}$ is a combination of two
cascaded networks, namely, encoder $e$ and decoder $d$.

The network $e$ extracts attentions from a sub-sampled version of the X as
$\text{A}=e(\text{X})=[\mathbf{a}_{1},\mathbf{a}_{2},\ldots,\mathbf{a}_{T^{\prime}}]$
where $T^{\prime}$ is the number of frames after sub-sampling and
$\mathbf{a}_{t^{\prime}}$ is the attention state vector at the $t^{\prime}$
sub-sampled frame. The reduction in number of time frames depends on the
number of sub-sampling layers used in the attention network. If $k$ sub-
sampling layers are used then $kT^{\prime}=T$, i.e., each frame of the sub-
sampled representation contains information from $k$ frames of the original
representation. Attention matrix A is then given to a decoder network to
obtain,

$\displaystyle\text{Y},\text{H}$ $\displaystyle=d(\text{A})$ (1) S
$\displaystyle=\text{SoftMax}(\text{Y})$ (2)

where
$\text{Y}=[\mathbf{y}_{1},\mathbf{y}_{2},\ldots,\mathbf{y}_{T^{\prime}}]$ is
the output sequence of the decoder network,
$\text{H}=[\mathbf{h}_{1},\mathbf{h}_{2},\ldots,\mathbf{h}_{T^{\prime}}]$ is
the output of the decoder layer of the decoder network and
$\text{S}=[\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{T^{\prime}}]$ are
the frame-wise probability distributions over $L^{\prime}=L\cup\\{blank\\}$.
We use S to determine the frame-wise sound units by greedy decoding as,
$\mathbf{\hat{z}}=\left[\hat{z}_{1},\hat{z}_{2},\ldots,\hat{z}_{T^{\prime}}\right]$
where

$\displaystyle\hat{z}_{t^{\prime}}=\underset{L^{\prime}}{\arg\max}\
s_{t^{\prime}}$ (3)

$\mathbf{\hat{z}}$ is transformed to words that make up the sentence as
$\mathbf{\hat{w}}=\left[\hat{w}_{1},\hat{w}_{2}\ldots,\hat{w}_{N^{\prime}}\right]$,
where each $\hat{w}_{n}=\left\\{\hat{z}_{i},\ldots,\hat{z}_{j}\right\\}$ with
$1\leq i\leq j\leq T^{\prime}$, using a many-to-one map $\mathcal{B}$ required
for models trained with CTC [21]. $\mathcal{B}$ removes the blank tokens and
repeated tokens in $\mathbf{\hat{z}}$.

To make the predictions from $\mathcal{F}$ more reliable, we define an
auxiliary WLC neural network $\mathcal{K}_{\mathcal{F}}$ with parameters
$\hat{\theta}$ to estimate confidence $c\in[0,1]$ of elements in
$\mathbf{\hat{w}}$. To satisfy our objective that $c$ should be correlated
with level of correctness of words in $\mathbf{\hat{w}}$, we train
$\mathcal{K}_{\mathcal{F}}$ using TeLeS score.

###  III-B TeLeS Score as a Confidence Measure

TeLeS score matches the temporal alignment of the predicted words with the
temporal alignment of the reference words, and the similarity between the
aligned reference and predicted words.

We use Kaldi speech recognition toolkit to determine the temporal alignment of
the words, $\mathbf{w}=\left[w_{1},w_{2},\ldots,w_{N}\right]$ in ground truth.
The dataset $\mathcal{D^{\prime}}$ is used to train a Gaussian Mixture Model-
Hidden Markov Model (GMM-HMM) based acoustic model. The trained GMM-HMM model
is used to perform forced alignment to obtain the start and stop times of the
$\mathbf{w}$,
$\left[(w_{1}^{ST},w_{1}^{ET}),(w_{2}^{ST},w_{2}^{ET}),\ldots,(w_{N}^{ST},w_{N}^{ET})\right]$.

We approximate the start and end times of words in
$\mathbf{\hat{w}}=\left[(\hat{w}_{1}^{ST},\hat{w}_{1}^{ET}),(\hat{w}_{2}^{ST},\hat{w}_{2}^{ET}),\ldots,(\hat{w}_{N^{\prime}}^{ST},\hat{w}_{N^{\prime}}^{ET})\right]$
from the frame-level indices of the predicted sequence, $\mathbf{\hat{z}}$.

We perform word level alignment between the reference and the prediction using
Levenshtein’s alignment to ensure that the temporal and lexical similarity
scores are being computed between corresponding words. For the reference word
sequence $\mathbf{w}$, and the predicted sequence $\mathbf{\hat{w}}$, we
define the function AliGen to generate the Levenshtein alignments as,

$\displaystyle G=\text{{AliGen}}(\mathbf{w},\mathbf{\hat{w}})$ (4)

where $G=[g_{1},g_{2},\ldots g_{P}]$ with $g_{p}\in\left\\{C,S,I,D\right\\}$,
which denotes correct, substitution, insertion and deletion respectively. We
now define the temporal agreement score $c^{T}$ as

$\displaystyle
c^{T}=\begin{cases}\max\left(0,1-\frac{\left|w_{i}^{ST}-\hat{w}_{j}^{ST}\right|+\left|w_{i}^{ET}-\hat{w}_{j}^{ET}\right|}{\left|w_{i}^{ET}-w_{j}^{ST}\right|}\right)&\text{if
}g_{p}=C,S\\\ 0&\text{if }g_{p}=I,D\end{cases}$ (5)

When the words are substituted, the words are not necessarily entirely wrong
but they may be wrong due to spelling mistakes. We compare the lexical
similarity score for these cases as it is essential to calibrate the model’s
confidence. We define the lexical similarity score $c^{L}$ as

$\displaystyle
c^{L}=\begin{cases}\frac{\left|w_{i}\cap\hat{w}_{j}\right|}{\left|w_{i}\cup\hat{w}_{j}\right|}&\text{if
}g_{p}=C,S\\\ 0&\text{if }g_{p}=I,D\end{cases}$ (6)

With these two components, we define the TeLeS score for every word as

$\displaystyle c=\begin{cases}\alpha\times c^{L}+(1-\alpha)\times
c^{T}&\text{if }g_{p}=C\\\ \beta\times c^{L}+(1-\beta)\times c^{T}&\text{if
}g_{p}=S\\\ 0&\text{if }g_{p}=I,D\end{cases}$ (7)

where, $\alpha\in[0,1]$ and $\beta\in[0,1]$ are weights for temporal and
lexeme similarity scores and they are tunable hyper-parameters.

###  III-C Learning to Predict Confidence with TeLeS-WLC

Figure 2: TeLeS-WLC Train Architecture

Figure 2 gives an overview of the training architecture of TeLes-WLC model.
TeLeS-WLC model is trained using intermediate states of ASR model and target
TeLeS score. We fetch attention states A, decoder states H, and class
probabilities S from ASR model.

Using $\mathbf{\hat{z}}$ across $T^{\prime}$ frames, for each word
$\hat{w}_{n}$, we find start and end indices of $\hat{w}_{n}$,
$I_{n}=(b_{\hat{w}_{n}},l_{\hat{w}_{n}})$, where $b_{\hat{w}_{n}}$ is the
start index and $l_{\hat{w}_{n}}$ is the end index, satisfying the conditions
$1\leq b_{\hat{w}_{n}}\leq T^{\prime}$ and $b_{\hat{w}_{n}}\leq
l_{\hat{w}_{n}}\leq T^{\prime}$. For each $\hat{w}_{n}$, we can now compute,

$\displaystyle\mathbf{\bar{a}}_{\hat{w}_{n}}$
$\displaystyle=\text{mean}(\mathbf{a}_{b_{\hat{w}_{n}}},\ldots,\mathbf{a}_{l_{\hat{w}_{n}}})$
(8) $\displaystyle\mathbf{\bar{h}}_{\hat{w}_{n}}$
$\displaystyle=\text{mean}(\mathbf{h}_{b_{\hat{w}_{n}}},\ldots,\mathbf{h}_{l_{\hat{w}_{n}}})$
(9) $\displaystyle\mathbf{\bar{s}}_{\hat{w}_{n}}$
$\displaystyle=\text{mean}(\mathbf{s}_{b_{\hat{w}_{n}}},\ldots,\mathbf{s}_{l_{\hat{w}_{n}}})$
(10)

We train $\mathcal{K}_{\mathcal{F}}$ with the following pairs
$([\mathbf{\bar{a}}_{\hat{w}_{n}},\mathbf{\bar{h}}_{\hat{w}_{n}},\mathbf{\bar{s}}_{\hat{w}_{n}}],c_{\hat{w}_{n}})$,
where $c_{\hat{w}_{n}}$ is the TeLeS score of word $\hat{w}_{n}$. We use
shrinkage loss [12] to train the WLC model. Shrinkage loss is defined as,

$\displaystyle\mathcal{L}=\left[\frac{\frac{1}{N^{\prime}}\sum_{\hat{w}_{n}\in\mathbf{\hat{w}}}\left(\hat{c}_{\hat{w}_{n}}-c_{\hat{w}_{n}}\right)^{2}\cdot
e^{{\hat{c}}_{\hat{w}_{n}}}}{1+e^{\gamma\cdot\left(\kappa-\frac{1}{N^{\prime}}\sum_{\hat{w}_{n}\in\mathbf{\hat{w}}}\left|\hat{c}_{\hat{w}_{n}}-c_{\hat{w}_{n}}\right|\right)}}\right]$
(11)

where,
$\hat{c}_{\hat{w}_{n}}=\mathcal{K}_{\mathcal{F}}([\mathbf{\bar{a}}_{\hat{w}_{n}},\mathbf{\bar{h}}_{\hat{w}_{n}},\mathbf{\bar{s}}_{\hat{w}_{n}}])$

$\gamma$ and $\kappa$ are hyper-parameters. $\kappa\in[0,1]$ penalizes easy-
to-learn data points by controlling the importance of frequently seen samples
and $\gamma\in[0,\infty)$ controls the shrinkage rate of the $L2$ loss between
$\hat{c}_{\hat{w}_{n}}$ and $c_{\hat{w}_{n}}$. $e^{{\hat{c}}_{\hat{w}_{n}}}$
is weight factor to accentuate the importance of hard-to-learn data point.

###  III-D TeLeS-A

Consider an unlabeled set of audio $\text{U}=\\{\text{U}_{i}\\}_{i=1}^{N}$.
The cost of manually labelling U may be higher than the budget. Active
learning entails discovering a set of uncertain samples within the budget and
transcribing it with a human annotator. We propose an acquisition function
TeLeS-A that uses confidence estimation from the TeLeS-WLC model. TeLeS-A
acquires uncertain samples and samples that the ASR model is confident about.
The uncertain samples are chosen within the labelling budget and given to the
annotator. A certain sample set is selected with the help of a threshold
$\delta$ and the audio-pseudolabel pairs are included in the train set.

$\displaystyle\mathbf{\hat{w}}$ $\displaystyle=\mathcal{F}(\text{U}_{i})$ (12)
$\displaystyle\hat{c}_{\hat{w}_{n}}$
$\displaystyle=\mathcal{K}_{\mathcal{F}}([\mathbf{\bar{a}}_{\hat{w}_{n}},\mathbf{\bar{h}}_{\hat{w}_{n}},\mathbf{\bar{s}}_{\hat{w}_{n}}])$
(13) $\displaystyle A_{\text{U}_{i}}$
$\displaystyle=\frac{\sum_{n=1}^{|\mathbf{\hat{w}}|}\hat{c}_{\hat{w}_{n}}}{|\mathbf{\hat{w}}|}$
(14) $\displaystyle\hat{\text{U}}$
$\displaystyle=sort([A_{\text{U}_{i}}]_{i=1}^{N})$ (15)

The top samples from $\hat{\text{U}}$ within the labelling budget are picked,
annotated and added to the training pool. The audio-pseudo-label pairs of the
samples with $A_{\text{U}_{i}}\geq\delta$ are added to the train set.

##  IV CEM Evaluation

###  IV-A Dataset

The efficiency of the TeLeS-WLC model is evaluated using speech datasets of
three different Indian languages: Hindi, Tamil, and Kannada. The following
details the dataset used to train the base ASR and TeLeS-WLC models.

  * •

Hindi- The dataset is a combination of speech data from different domains. The
size of the dataset is $\approx 100$ hours. It has $\approx 17.6$ hours of
speech data from the Common Voice Hindi (CVH) dataset [31]. Our annotators
made minor corrections to CVH, such as replacing numeric digits with the
uttered text, transliterating English words into Hindi, inserting missing
words, and correcting spelling errors. The rest of the speech data ($\approx
78.21$ hours) are taken from Prasar Bharati archive and manually annotated.
The manually annotated datasets, collectively called the PB Hindi dataset, are
speech recordings from different scenarios, including read news, the public
addresses of eminent personalities, and recordings during programmes by
various people. Hence, the dataset combines a few hours of recordings from
diverse domains.

  * •

Tamil- We use the benchmark IISc-MILE Tamil speech corpus, which comprises
$\approx 150$ hours of train and test speech dataset [8][9].

  * •

Kannada- Benchmark IISc-MILE Kannada dataset [8][9][32] is used for the
Kannada language, which comprises $\approx 350$ hours of training and test
speech dataset.

Verifying if the TeLeS-WLC model performs better in mismatched domain datasets
is imperative. Hence, we use the benchmark KathBath (KB) [33][34] test-known
dataset of the three languages to evaluate the respective trained TeLeS-WLC
models.

ASR models are trained using speech conformer architecture [35] and CTC loss.
We train the Hindi ASR model with two Nvidia V100 16GB GPUs [36]. Four 40GB
Nvidia A100 GPUs are used to train Tamil and Kannada ASR models [37]. The
“small” version of the speech conformer is configured in the encoder and
decoder layers of the architecture, resulting in 10M parameters. If necessary,
the speech audios are downsampled at 16000 Hz, and two channels are converted
into one. We extract 80-dimensional Melspectrogram features from the processed
speech audios and give them as input to the training module. AdamW optimizer
with a learning rate 5e-4 and transformer scheduler [38] are configured during
the training phase. The output of the ASR models is space and CTC blank symbol
added with the smallest grapheme unit of the languages: 64 Hindi characters,
51 Tamil characters, and 65 Kannada characters. Table I gives the Word Error
Rate (WER) and Character Error Rate (CER) observed on the PB Hindi, IISc-Mile
Tamil and Kannada testsets using a greedy decoder and beam-search decoder with
3gram KenLM [39]. To confirm if the KB datasets are from a mismatched domain,
we check the error rates of the inferences from the ASR model and present them
in Table II. It is evident from Table II that the KB datasets have domain
shift with respect to the trained ASR model.

TABLE I: Accuracy of ASR models on Testsets Testset | Without LM | With LM
---|---|---
WER(%) |  CER(%) |  WER(%) |  CER(%)
PB- Hindi |  29.80 |  12.86 |  25.15 |  11.89
IISc-Mile- Tamil |  25.72 |  4.27 |  22.26 |  3.88
IISc-Mile- Kannada |  23.66 |  4.27 |  18.52 |  3.48
TABLE II: Accuracy of ASR models on KB Test-known sets Testset | Without LM | With LM
---|---|---
WER(%) |  CER(%) |  WER(%) |  CER(%)
KB-Hindi |  37.76 |  14.44 |  32.26 |  13.23
KB-Tamil |  58.71 |  14.68 |  50.23 |  13.48
KB-Kannada |  56.67 |  15.76 |  48.48 |  14.50

We train the TeLeS-WLC architecture on Nvidia 3070Ti GPU. The TeLeS-WLC neural
network comprises three fully connected layers and a sigmoid layer. The three
layers have 512, 256 and 128 neurons, respectively. Activation function
Rectified Linear Unit is added in between the neural layers. The model
training is done with the Adam optimizer and a learning rate of 0.0001 for 50
epochs. We employ grid search over a range of $\left[0,1\right]$ and run 10
epochs during each run to tune the TeLeS score hyperparameters $\alpha$ and
$\beta$. We find that the ($\alpha,\beta$) = (0.75, 0.5) combination provides
better results than other combinations and use them in subsequent studies.

###  IV-B Baseline Methodologies

We use three baseline approaches. The confidence score generated from ASR
class probabilities is an approach [6][7]. The class probabilities of the
grapheme tokens are averaged to get the word-level confidence scores. The
second baseline approach involves finding the Tsallis entropy with exponential
normalization of the class probabilities of the tokens [13][4]. The minimum
function is the aggregation function to obtain the word-level confidence
measure [4]. The final baseline approach [5], [11], [15]-[25] involves
training the confidence neural network with binary target scores extracted
from the alignment between ground truth and hypothesis using Levenshtein
alignment.

###  IV-C Evaluation Metrics

Following standard evaluation metrics are used to evaluate TeLeS-WLC model
performance:
Mean Absolute Error (MAE $\downarrow$)- The MAE $\in[0,1]$ measures the
disparity between the estimated score $\hat{y}_{i}$ and the actual score
$y_{i}$ for the $i^{\text{th}}$ word. MAE is defined as

$\displaystyle\text{MAE}(Y,\hat{Y})=\frac{1}{N}\sum_{i=1}^{N}\left|y_{i}-\hat{y}_{i}\right|$
(16)

Kullback-Leibler Divergence score (KLD $\downarrow$)- KLD $\in[0,\infty]$
evaluates the difference between the estimated and actual distributions of
scores. For $N$ words with the actual score as $y_{i}$ and estimated score as
$\hat{y}_{i}$ for the $i^{\text{th}}$ word, KLD is defined as

$\displaystyle
D_{\text{KL}}(Y\parallel\hat{Y})=\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\log\frac{y_{i}}{\hat{y}_{i}}+\left(1-y_{i}\right)\log\frac{1-y_{i}}{1-\hat{y}_{i}}\right]$
(17)

Jensen–Shannon divergence (JSD $\downarrow$)- JSD $\in[0,1]$ is a smoothed
version of KLD.

$\displaystyle\text{JSD}(Y\parallel\hat{Y})=\frac{\left(D_{\text{KL}}(Y\parallel
M)+D_{\text{KL}}(\hat{Y}\parallel M)\right)}{2}$ (18)

where $M=\dfrac{Y+\hat{Y}}{2}$.
Normalized Cross Entropy (NCE $\uparrow$)- NCE $\in[-\infty,1]$ [40][41]
evaluates the correlation between the estimated confidence score and base ASR
model accuracy.

$\displaystyle H(p)$ $\displaystyle=-p\log p-(1-p)log(1-p)$ (19)
$\displaystyle H(p,\hat{Y})$
$\displaystyle=\frac{-1}{N}\left[\sum_{i|g_{i}=C}\log\hat{y}_{i}\ +\
\sum_{i|g_{i}\neq C}\log(1-\hat{y}_{i})\right]$ (20) NCE
$\displaystyle=\frac{H(p)-H(p,\hat{Y})}{H(p)}$ (21)

where $p$ is the accuracy of the ASR model and $g_{i}$ indicates the word
status obtained from the Levenshtein alignment described in section III.B.
Calibration Error (CE $\downarrow$) The purpose of using confidence scores is
to minimise the worst-case gap between estimated confidence and measured word
accuracy [42]. The computation of CE requires binning. Expected Calibration
Error (ECE) $\in[0,1]$ and Maximum Calibration Error (MCE) $\in[0,1]$ are two
distinct CE metrics. ECE is the weighted mean of all confidence and accuracy
gaps. In contrast, MCE is the most significant gap among all bins.

ECE
$\displaystyle=\sum_{m=1}^{M}\frac{B_{m}}{n}\left|acc(B_{m})-conf(B_{m})\right|$
(22) MCE
$\displaystyle=\underset{m\in{1,\ldots,M}}{\max}\left|acc(B_{m})-conf(B_{m})\right|$
(23)

where $B_{m}$s are equally spaced ‘$M$’ bins, $n$ is the total number of
words, $conf(B_{m})=\frac{1}{m_{n}}\sum_{n=1}^{m_{n}}\hat{y}_{n}$ for
$\hat{y}_{n}\in B_{m}$ and $acc(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in
B_{m}}\mathds{1}(y_{i}=\hat{y}_{i})$.
Root Mean Square Error- Word Correctness Ratio (RMSE-WCR $\downarrow$)- RMSE-
WCR $\in[0,1]$ evaluates the accuracy of the estimated confidence with respect
to WCR [25].

$\displaystyle\text{A}_{i}$
$\displaystyle=\frac{1}{N^{\prime}}\sum_{j=1}^{N^{\prime}}\hat{y}_{j}$ (24)
$\displaystyle\text{WCR}_{i}$ $\displaystyle=\frac{\text{Number of correctly
predicted words}}{\text{Total number of predicted words}}$ (25) RMSE WCR
$\displaystyle=\sqrt{\frac{1}{K^{\prime}}\sum_{i=1}^{K^{\prime}}(\text{A}_{i}-\text{WCR}_{i})^{2}}$
(26)

NCE and CE are not ideal evaluation metrics for our case. They require the
target labels to be binary. SOTA [5], [11], [15]-[25] methods use binary
target labels, and hence, their evaluation metrics include NCE and CE. For
comparison, we compute NCE and CE between the predicted scores from TeLeS-WLC
model and the binary labels from Levenshtein alignment. Hence, better measures
for comparing continuous TeLeS targets and estimated scores are MAE, KLD, JSD
and RMSE-WCR.

###  IV-D Training Modes

Our first step of the experiment is to verify which training mode yields
better performance. Four different training combinations are coded as follows.

  * •

Train with MAE loss (MAE-L).

  * •

Transform the audio on-the-fly by applying time and frequency masking on the
respective spectrograms [5][23][24], before generating ASR inferences and
employ MAE loss for training (Mask-MAE-L).

  * •

Train with shrinkage loss (Shrink-L).

  * •

Apply time and frequency masking on-the-fly on the audio spectrograms before
generating ASR inferences and use shrinkage loss for training (Mask-Shrink-L).

Spectrogram augmentation involves transformation using ten time masks with
mask lengths of 20 and 0.05 as the time-steps proportion that can be masked,
followed by frequency mask with a mask length of 27. The shrinkage loss
parameters are configured as $\gamma=5$, $\kappa=0.2$, which we identify
through grid search over ranges $[2,20]$ for $\gamma$ and $[0.1,1.0]$ for
$\kappa$.

Table III shows the results observed in the Hindi ASR (PB Hindi test) case for
different training modes. It is evident from the ASR model’s WER in Table I
that the WLC model training sees a significant percentage of correct words.
Hence, MAE-L and Shrink-L are influenced by correct word scores. To boost the
rate of incorrect words, [5][23][24] transform the audio with time and
frequency mask techniques. However, it does not produce a balanced set of
words across the $[0,1]$ TeLes score range, as evident from Figure 3 (Score
distribution taken from PB dataset). We can find a slight overlap between
scores of correct and incorrect words. It is because some predicted words have
minor spelling mistakes and temporal alignment with the ground truth. For
instance, the words  (/n/au/ja/v/aa/n/) and  (/n/au/j/a/v/aa/l/) by one
character and are temporally aligned with each other. Hence, we observe its
TeLeS score as $0.816$.

Mask-Shrink-L train combination yields better results, as seen in Table III.
MAE-L has better MAE than the Mask-Shrink-L. However, other metrics show
better results in Mask-Shrink-L case, as shrinkage loss handles the score
imbalance. CE curves in Figures 4 and 5 also pictorially depict the efficacy
of the same. Hence, in further experiments, we use Mask-Shrink-L for training.

Figure 3: Distribution of Words across TeLeS Score Range TABLE III: Evaluation of TeLeS-WLC Training modes - Hindi ASR model Metrics |  MAE-L |  Mask-MAE-L |  Shrink-L |  Mask-Shrink-L
---|---|---|---|---
MAE $\downarrow$ |  0.2043 |  0.1783 |  0.1966 |  0.1817
KLD $\downarrow$ |  0.3680 |  0.2583 |  0.2809 |  0.1785
JSD $\downarrow$ |  0.0783 |  0.0595 |  0.0659 |  0.0467
RMSE-WCR $\downarrow$ |  0.2557 |  0.1987 |  0.2229 |  0.1457
NCE $\uparrow$ |  -0.2948 |  -0.0238 |  -0.1272 |  0.1363
ECE $\downarrow$ |  0.1908 |  0.1345 |  0.1483 |  0.0260
MCE $\downarrow$ |  0.4265 |  0.3575 |  0.2671 |  0.1011
Figure 4: Calibration Curve - Mask-MAE-L Figure 5: Calibration Curve - Mask-
Shrink-L

###  IV-E Ablation

To verify if the combination of lexeme and temporal similarity is required for
computing the TeLeS score, we study the performance of models trained using
only lexeme similarity (i.e., $\alpha=1$; $\beta=1$) and temporal similarity
($\alpha=0$; $\beta=0$). Table IV gives the results of the ablation study done
with the Tamil language dataset (IISc-Mile Tamil testset). We see that MAE is
better with lexeme-similarity only case. It can be partly due to the training
of the model, wherein the training procedure tries to bring down the
difference between predictions and targets. However, RMSE-WCR and NCE metrics
that correlate the estimated confidence values with ASR model accuracy show
that combining lexeme and temporal similarity via TeLeS score is more
effective than using either.

TABLE IV: Ablation Study on Components of TeLeS \- Tamil Metrics |  Lexeme only |  Temporal only |  TeLeS
---|---|---|---
MAE $\downarrow$ |  0.1296 |  0.1917 |  0.1489
KLD $\downarrow$ |  0.1942 |  0.2091 |  0.1432
JSD $\downarrow$ |  0.0596 |  0.0584 |  0.0373
RMSE-WCR $\downarrow$ |  0.2258 |  0.2405 |  0.1955
NCE $\uparrow$ |  -0.1277 |  -0.1741 |  0.0971
ECE $\downarrow$ |  0.1244 |  0.1312 |  0.0456
MCE $\downarrow$ |  0.2146 |  0.6528 |  0.1514

###  IV-F Comparison with SOTA

Tables V, VI, and VII compare the results of various evaluation metrics
observed during the experiment on the three languages with SOTA methods (PB
and IISc-Mile test sets). Using class-probability-based [6][7] and entropy-
based [13][4] confidence scores showcases degraded performance resulting from
the overconfidence behaviour of ASR. Overconfidence behaviour fails to
delineate a boundary between the scores of correct and incorrect words. TeLeS-
WLC method shows better performance than using binary targets [5], [11],
[15]-[25]. Binary targets try to penalize the predictions of words with minor
mistakes and bring them down to 0. However, the input features of the words
with minor errors will be close to those of the correct words. Hence, the
model with binary targets could not learn the subtle variations in the
patterns of the input features.

TABLE V: TeLeS-WLC and SOTA \- Hindi PB Testset Metrics |  Class-Prob |  Entropy |  Binary |  TeLeS
---|---|---|---|---
MAE $\downarrow$ |  0.5026 |  0.3998 |  0.2650 |  0.1817
KLD $\downarrow$ |  0.7912 |  0.6333 |  1.1799 |  0.1785
JSD $\downarrow$ |  0.2313 |  0.1818 |  0.1920 |  0.0467
RMSE-WCR $\downarrow$ |  0.2883 |  0.2684 |  0.2593 |  0.1457
NCE $\uparrow$ |  -0.2641 |  -0.0100 |  -0.0055 |  0.1363
ECE $\downarrow$ |  0.2472 |  0.2253 |  0.2627 |  0.0260
MCE $\downarrow$ |  0.3904 |  0.3663 |  0.4294 |  0.1011
TABLE VI: TeLeS-WLC and SOTA \- Tamil IISc-MILE Testset Metrics |  Class-Prob |  Entropy |  Binary |  TeLeS
---|---|---|---|---
MAE $\downarrow$ |  0.5957 |  0.5618 |  0.2158 |  0.1489
KLD $\downarrow$ |  1.0214 |  0.9849 |  0.9579 |  0.1432
JSD $\downarrow$ |  0.2871 |  0.2726 |  0.1544 |  0.0373
RMSE-WCR $\downarrow$ |  0.4993 |  0.4956 |  0.2441 |  0.1955
NCE $\uparrow$ |  -0.9016 |  -0.8337 |  -0.2912 |  0.0971
ECE $\downarrow$ |  0.4495 |  0.4579 |  0.2084 |  0.0456
MCE $\downarrow$ |  0.8958 |  0.5437 |  0.4330 |  0.1514
TABLE VII: TeLeS-WLC and SOTA \- Kannada IISc-MILE Testset Metrics |  Class-Prob |  Entropy |  Binary |  TeLeS
---|---|---|---|---
MAE $\downarrow$ |  0.5350 |  0.6539 |  0.1845 |  0.1261
KLD $\downarrow$ |  0.8632 |  1.4422 |  0.8297 |  0.0927
JSD $\downarrow$ |  0.2502 |  0.3518 |  0.1332 |  0.0246
RMSE-WCR $\downarrow$ |  0.3946 |  0.6289 |  0.1780 |  0.1413
NCE $\uparrow$ |  -0.6703 |  -1.7907 |  -0.4396 |  0.1584
ECE $\downarrow$ |  0.3537 |  0.6126 |  0.1821 |  0.0699
MCE $\downarrow$ |  0.6122 |  0.6935 |  0.3597 |  0.1100

###  IV-G Evaluation of TeLeS-WLC on Mismatched Domain

The TeLeS-WLC model shows better results for in-domain speech data. Table IX
shows the performance of the TeLeS-WLC model on the mismatched KB test-known
datasets for Hindi, Tamil and Kannada. We find that the performance is better
for all three languages, which indicates that TeLeS generalizes well. Table
VIII gives examples of reference, hypothesis and confidence scores given by
TeLeS-WLC (Rough transliteration is given).

TABLE VIII: Estimated TeLeS Confidence Score Language |  Reference |  Prediction |  Alignment |  Estimated TeLeS
---|---|---|---|---
Hindi |  (/jarman/ /aakramanakaariyon/ /ne/ /thietar/ /ke/ /inteeriyar/ /aur/ /sajaavat/ /ke/ /jarmanee/ /tatvon/ /ko/ /le/ /liya/) |  (/jarman/ /aapar/ /maran/ /kaaryon/ /ne/ /phyetar/ /ke/ /inateereee/ /aur/ /sajaavat/ /ke/ /jarmanee/ /tathvon/ /ko/ /le/ /liya/) |  [’C’, ’I’, ’I’, ’S’, ’C’, ’S’, ’C’, ’S’, ’C’, ’C’, ’C’, ’C’, ’S’, ’C’, ’C’, ’C’] |  [0.67, 0.37, 0.43, 0.42, 0.81, 0.50, 0.78, 0.18, 0.69, 0.82, 0.81, 0.77, 0.42, 0.86, 0.72, 0.69]
Tamil |  (/whatsapp/ /kuzhumam/ /ceydhu/ /kondirukkiradhu/ /adharku/ /valla/ /rahman/ /kooli/ /kodukka/ /podhumaanavan/) |  (/Maarshak/ /kuzhumam/ /teydhu/ /kondirukkiradhu/ /adharku/ /valla/ /ratman/ /kali/ /kodukka/ /podhum/ /aanavan/) |  [’S’, ’C’, ’S’, ’C’, ’C’, ’C’, ’S’, ’S’, ’C’, ’I’, ’S’] |  [0.48, 0.82, 0.51, 0.86, 0.81, 0.74, 0.58, 0.62, 0.75, 0.52, 0.53]
Kannada |  (/Ondhu/ /chalanasheela/ /vidyutkaantavu/ /balavannu/ /anvayisutthadhey/ /matthu/ /enannaadaru/ /chalisutthadey/) |  (/Ondu/ /chalanasheela/ /vidyut/ /naankavu/ /balavannu/ /anvayisutthadhey/ /matthu/ /enannadaru/ /chalisutthadey/) |  [’C’, ’C’, ’I’, ’S’, ’C’, ’C’, ’C’, ’C’, ’C’] |  [0.77, 0.73, 0.55, 0.32, 0.83, 0.77, 0.78, 0.79, 0.70]
TABLE IX: Performance of TeLeS-WLC on KB Test-Known Datasets Metrics |  Hindi |  Tamil |  Kannada
---|---|---|---
MAE $\downarrow$ |  0.1869 |  0.2280 |  0.1925
KLD $\downarrow$ |  0.1804 |  0.2217 |  0.1621
JSD $\downarrow$ |  0.0478 |  0.0588 |  0.0437
RMSE-WCR $\downarrow$ |  0.1670 |  0.2700 |  0.2181
NCE $\uparrow$ |  0.1590 |  0.1151 |  0.2156
ECE $\downarrow$ |  0.0593 |  0.2116 |  0.1460
MCE $\downarrow$ |  0.1552 |  0.2643 |  0.2745

##  V TeLeS-A Evaluation

###  V-A Data and Baseline

For active learning experiments, we use the base ASR models trained with PB-
Hindi, IISc-MILE Tamil and IISc-MILE Kannada datasets and consider the KB-
Hindi ($\approx 150\ hours$), Tamil ($\approx 185\ hours$) and Kannada
($\approx 65\ hours$ train dataset as unlabeled audio sets. We set $\delta\geq
0.8$ for acquiring pseudo-labels in TeLeS-A experiment.

Baseline Approaches- We compare with three baselines. The first approach uses
the length-normalized path probability approach given in [27][28]. The second
approach is SMCA, presented in [26]. The final method is the LMC methodology
proposed in [26], where we build a 3-gram KenLM [39] using the transcripts to
train the base ASR model and compute the perplexity score for acquiring data
for annotation.

###  V-B Evaluation Metrics

We use the standard WER (%) $\downarrow$ and CER (%) $\downarrow$ metrics to
evaluate the active learning approaches. We observe how much WER and CER
reduce after finetuning the base ASR models with the acquired data. Each
experiment is run for 30 epochs with same parameters used to train the base
ASR models.

###  V-C Comparison with SOTA

We fix the labelling budget based on audio hours. We choose the experimental
label budget as a proportion $(1/10,1/7)$ of the training dataset.

Tables X, XI, and XII report the WER and CER statistics on KB test-known sets.
As seen in the statistics, TeLeS-A has better performance than SOTA methods.
LMC depends on the LM, trained on the source ASR dataset and has no statistics
about the target dataset transcripts. In a real-time scenario, there is no
guarantee that obtaining transcripts from the target domain is possible. Thus,
LMC is unreliable for real-time implementation, as evidenced by the observed
WER in the tables. SMCA method performs better than LMC as it varies the
dropout of the base ASR model and sees if the inference is similar even after
model perturbation. For the SMCA experiment, we configure ten seeds to prepare
ten different perturbed models, which may be a significant run-time overhead
in cases of huge unlabeled datasets. However, the base ASR model’s encoder
weights are the same, and we cannot expect the perturbed model to expose
uncertainty in all the circumstances. The TeLeS-A method is also better than
the SOTA method as it acquires the set of confident samples, which aids in
convergence.

TABLE X: Active Learning - KB Hindi Test-Known Dataset Acquired Data Proportion |  Path Probability (WER/CER $\downarrow$) |  SMCA (WER/CER $\downarrow$) |  LMC (WER/CER $\downarrow$) |  TeLeS (WER/CER $\downarrow$)
---|---|---|---|---
$1/10$ |  43.53/15.22 |  59.97/23.01 |  92.55/53.16 |  32.96/11.09
$1/7$ |  31.71/10.59 |  33.09/11.00 |  36.93/12.42 |  28.78/9.51
TABLE XI: Active Learning - KB Tamil Test-Known Dataset Acquired Data Proportion |  Path Probability (WER/CER $\downarrow$) |  SMCA (WER/CER $\downarrow$) |  LMC (WER/CER $\downarrow$) |  TeLeS (WER/CER $\downarrow$)
---|---|---|---|---
$1/10$ |  50.05/10.52 |  49.63/10.43 |  50.64/10.71 |  47.38/9.83
$1/7$ |  45.62/9.28 |  45.11/9.13 |  45.49/9.19 |  44.35/8.87
TABLE XII: Active Learning - KB Kannada Test-Known Dataset Acquired Data Proportion |  Path Probability (WER/CER $\downarrow$) |  SMCA (WER/CER $\downarrow$) |  LMC (WER/CER $\downarrow$) |  TeLeS (WER/CER $\downarrow$)
---|---|---|---|---
$1/10$ |  54.29/12.22 |  99.97/85.58 |  99.95/93.97 |  47.98/10.71
$1/7$ |  43.75/9.57 |  60.87/13.67 |  74.47/17.69 |  41.91/9.15

##  VI Conclusion and Future Work

The manuscript examines the cons of class-probability-based confidence scores,
entropy-based confidence scores, and CEM trained with binary target labels. We
propose a better confidence target score that includes temporal and lexeme
similarity. In addition to the score proposal, our work also gives a solution
to handle the data imbalance issue in the confidence neural net training data.
We test the efficacy of the score in three different language ASR models, each
trained with varying hours of speech data. Through several evaluation metrics
and graph-based investigations, we find that the proposed score is better than
the SOTA approaches. The usability of TeLeS-WLC in practical ASR downstream
active learning task further illustrates that TeLeS is a reliable score.
Current work can be extended to estimate confidence of deletion of words in
the utterances. Our next plan is to design an appropriate module to detect the
deletions. Also, one could experiment with the usability of the proposed
method for other downstream and upstream tasks. Future research direction
could be to adapt a model trained in one language to another language.

## Acknowledgments

Prasar Bharati and MeitY, India funded the project. We acknowledge the
assistance of the project’s stakeholders for their consistent support and
Param Siddhi for providing compute infrastructure. Contribution of IIT Kanpur
student interns Raghav Karan and Mridul Pandey towards the project are
noteworthy. We also recognise the data annotation team members- Divyanshu
Tripathi, Anika Kumari, Ankita Bhattacharya, Shruthi Mishra and Sachindananda
Prajapathi for their meticulous effort in preparing the data.

## References

  * [1] Chang, H. J., Lee, H. Y., & Lee, L. S. (2021). Towards lifelong learning of end-to-end ASR. arXiv preprint arXiv:2104.01616.
  * [2] Qiu, D., He, Y., Li, Q., Zhang, Y., Cao, L., & McGraw, I. (2021). Multi-task learning for end-to-end ASR word and utterance confidence with deletion prediction. arXiv preprint arXiv:2104.12870.
  * [3] Li, Qiujia, P. M. Ness, Anton Ragni, and Mark JF Gales. ”Bi-directional lattice recurrent neural networks for confidence estimation.” In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6755-6759. IEEE, 2019.
  * [4] A. Laptev and B. Ginsburg, ”Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-to-End Automatic Speech Recognition,” 2022 IEEE Spoken Language Technology Workshop (SLT), Doha, Qatar, 2023, pp. 152-159.
  * [5] Q. Li et al., ”Confidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition,” ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 6388-6392.
  * [6] D. S. Park, Y. Zhang, Y. Jia, W. Han, C. C. Chiu, B. Li, Y. Wu, and Q. V. Le. ”Improved noisy student training for automatic speech recognition.” arXiv preprint arXiv:2005.09629 (2020).
  * [7] Y. Chen, W. Wang, and C. Wang. ”Semi-supervised ASR by end-to-end self-training.” arXiv preprint arXiv:2001.09128 (2020).
  * [8] Pilar, Bharathi. ”Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada.” arXiv preprint arXiv:2207.13331 (2022).
  * [9] Pilar, Bharathi. ”Knowledge-driven Subword Grammar Modeling for Automatic Speech Recognition in Tamil and Kannada.” arXiv preprint arXiv:2207.13333 (2022).
  * [10] https://www.openslr.org/127/ Accessed online: July 30, 2023.
  * [11] A. Ogawa, N. Tawara, T. Kano and M. Delcroix, ”BLSTM-Based Confidence Estimation for End-to-End Speech Recognition,” ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 6383-6387.
  * [12] X. Lu, C. Ma, J. Shen, X. Yang, I. Reid and M. -H. Yang, ”Deep Object Tracking With Shrinkage Loss,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2386-2401, 1 May 2022.
  * [13] D. Oneaţa, A. Caranica, A. Stan and H. Cucu, ”An Evaluation of Word-Level Confidence Estimation for End-to-End Automatic Speech Recognition,” 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China, 2021, pp. 258-265.
  * [14] P. -S. Huang, K. Kumar, C. Liu, Y. Gong and L. Deng, ”Predicting speech recognition confidence using deep learning with word identity and score features,” 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 2013, pp. 7413-7417.
  * [15] K. Kalgaonkar, C. Liu, Y. Gong and K. Yao, ”Estimating confidence scores on ASR results using recurrent neural networks,” 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia, 2015.
  * [16] M. Á. Del-Agua, A. Giménez, A. Sanchis, J. Civera and A. Juan, ”Speaker-Adapted Confidence Measures for ASR Using Deep Bidirectional Recurrent Neural Networks,” in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 7, pp. 1198-1206, July 2018.
  * [17] Swarup, P., Maas, R., Garimella, S., Mallidi, S. H., & Hoffmeister, B. (2019). Improving ASR confidence scores for Alexa using acoustic and hypothesis embeddings.
  * [18] Kastanos, Alexandros, Anton Ragni, and Mark JF Gales. ”Confidence estimation for black box automatic speech recognition systems using lattice recurrent neural networks.” In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6329-6333. IEEE, 2020.
  * [19] Jeon, Woojay, Maxwell Jordan, and Mahesh Krishnamoorthy. ”On Modeling ASR Word Confidence.” In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6324-6328. IEEE, 2020.
  * [20] Futami, Hayato, Hirofumi Inaguma, Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara. ”ASR rescoring and confidence estimation with ELECTRA.” In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 380-387. IEEE, 2021.
  * [21] Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006, June). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning (pp. 369-376).
  * [22] Wang, Mingqiu, Hagen Soltau, Laurent El Shafey, and Izhak Shafran. ”Word-level confidence estimation for RNN transducers.” In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1170-1177. IEEE, 2021.
  * [23] Li, Qiujia, Yu Zhang, Bo Li, Liangliang Cao, and Philip C. Woodland. ”Residual energy-based models for end-to-end speech recognition.” arXiv preprint arXiv:2103.14152 (2021).
  * [24] Li, Qiujia, Yu Zhang, David Qiu, Yanzhang He, Liangliang Cao, and Philip C. Woodland. ”Improving confidence estimation on out-of-domain data for end-to-end speech recognition.” In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6537-6541. IEEE, 2022.
  * [25] Qiu, David, Qiujia Li, Yanzhang He, Yu Zhang, Bo Li, Liangliang Cao, Rohit Prabhavalkar et al. ”Learning word-level confidence for subword end-to-end ASR.” In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6393-6397. IEEE, 2021.
  * [26] X. Sun, B. Wang, S. Liu, T. Lu, X. Shan and Q. Yang, ”LMC-SMCA: A New Active Learning Method in ASR,” in IEEE Access, vol. 9, pp. 37011-37021, 2021.
  * [27] Malhotra, Karan, Shubham Bansal, and Sriram Ganapathy. ”Active Learning Methods for Low Resource End-to-End Speech Recognition.” INTERSPEECH. 2019.
  * [28] Bang, Jihwan, et al. ”Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples.” arXiv preprint arXiv:2006.11021 (2020).
  * [29] Hendrycks, Dan, and Kevin Gimpel. ”A baseline for detecting misclassified and out-of-distribution examples in neural networks.” arXiv preprint arXiv:1610.02136 (2016).
  * [30] Vasudevan, Vishal Thanvantri, Abhinav Sethy, and Alireza Roshan Ghias. ”Towards better confidence estimation for neural models.” In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7335-7339. IEEE, 2019.
  * [31] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. ”Common voice: A massively-multilingual speech corpus.” arXiv preprint arXiv:1912.06670 (2019).
  * [32] https://www.openslr.org/126/ Accessed online: July 30, 2023.
  * [33] https://ai4bharat.iitm.ac.in/indic-superb Accessed online: July 28 2023.
  * [34] Javed, Tahir, et al. ”Indicsuperb: A speech processing universal performance benchmark for indian languages.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.
  * [35] Gulati, Anmol, et al. ”Conformer: Convolution-augmented transformer for speech recognition.” arXiv preprint arXiv:2005.08100 (2020).
  * [36] https://www.iitk.ac.in/new/param-sanganak Accessed online: July 28 2023.
  * [37] M. Modani, A. Banerjee and A. Das, ”Performance analysis of quantum algorithms on Param Siddhi-AI system,” 2022 International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT), Pune, India, 2022, pp. 1-5.
  * [38] Vaswani, Ashish, et al. ”Attention is all you need.” Advances in neural information processing systems 30 (2017).
  * [39] Heafield, Kenneth. ”KenLM: Faster and smaller language model queries.” Proceedings of the sixth workshop on statistical machine translation. 2011.
  * [40] Siu, Man-hung, Herbert Gish, and Fred Richardson. ”Improved estimation, evaluation and applications of confidence measures for speech recognition.” In Fifth European Conference on Speech Communication and Technology. 1997.
  * [41] https://www.nist.gov/system/files/documents/2017/11/30/nce.pdf (Accessed Online 21.02.2023).
  * [42] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. ”On calibration of modern neural networks.” In International conference on machine learning, pp. 1321-1330. PMLR, 2017.

|  Nagarathna Ravi received the B.Tech. degree in information technology from
the Madras Institute of Technology, Anna University, India, and the M.E.
degree in computer science engineering from the Thiagarajar College of
Engineering, India. She received the Ph.D. degree from Anna University, India.
She is currently a Postdoctoral Researcher at the Indian Institute of
Technology (IIT) Kanpur, India. Her research interests are ASR, SDN and IoT.
---|---
|  Thishyan Raj T received the B.E. degree in Electronics and Communications
Engineering from the PESIT Bangalore South Campus. He is currently pursuing
M.S. degree in Department of Electrical Engineering, IIT Kanpur. His research
interests are ASR, audio processing, and machine learning.
---|---
|  Vipul Arora received the B.Tech. and Ph.D. degrees in electrical
engineering from the Indian Institute of Technology (IIT) Kanpur, India, in
2009 and 2015, respectively. He has been a Postdoctoral Researcher at the
University of Oxford and a Research Scientist at Amazon Alexa, Boston, MA,
USA. He is currently working as an Associate Professor with the Department of
Electrical Engineering, IIT Kanpur. His research interests include machine
learning, audio processing, machine learning for physics, and time series
analysis.
---|---