# CONSISTENCY BASED UNSUPERVISED SELF-TRAINING FOR ASR PERSONALISATION

###### Abstract

On-device Automatic Speech Recognition (ASR) models trained on speech data of
a large population might underperform for individuals unseen during training.
This is due to a domain shift between user data and the original training
data, differed by user’s speaking characteristics and environmental acoustic
conditions. ASR personalisation is a solution that aims to exploit user data
to improve model robustness. The majority of ASR personalisation methods
assume labelled user data for supervision. Personalisation without any
labelled data is challenging due to limited data size and poor quality of
recorded audio samples. This work addresses unsupervised personalisation by
developing a novel consistency based training method via pseudo-labelling. Our
method achieves a relative Word Error Rate Reduction (WERR) of 17.3% on
unlabelled training data and 8.1% on held-out data compared to a pre-trained
model, and outperforms the current state-of-the art methods.

**footnotetext: Equal contribution

Index Terms—  speech recognition, unsupervised, speaker adaptation,
personalisation

##  1 Introduction

End-to-end ASR models are known to underperform when deployed in the wild as
they face voice characteristics (e.g. accent, tone) and background acoustics
(e.g. noise, reverberation) unseen during training [1, 2, 3, 4]. A promising
approach to remedy this issue consists of fine-tuning a pre-trained ASR model
on device with collected user data [5, 6, 7]. This type of approach is also
known as “personalisation” or “adaptation” as it tailors the ASR model to the
single user of the device.

While supervised personalisation methods [5, 6, 7] substantially improve
performance for end-users, they require labelled data which is impractical in
many use cases [8]. This paper focuses on unsupervised self-training, which
aims to improve the robustness of an ASR model using only unlabelled user
data. Recently, unsupervised ASR personalisation has made progress by
incorporating auxiliary speaker features into an ASR model [9], exploring
self-training methods [10, 11], and training based on entropy minimisation [8,
12].

A common pipeline for the unsupervised self-training method contains data
filtering, pseudo-labelling, and training [10, 11]. In [11], a confidence
estimation module uses ASR output probabilities to select a less erroneous
subset of utterances from the entire set of unlabelled samples. The filtered
samples are processed by the pre-trained model to generate pseudo-labels,
which are subsequently used during training. However, the training process can
be unstable without accessing labelled data for supervision and the model
drifts away due to erroneous pseudo-labels [13].

Consistency Constraint (CC) forces a model to predict the same results on the
same input with various versions of perturbation and has been shown effective
for exploring unlabelled data [14, 15, 16, 17]. Applying various perturbations
introduces randomisation to regularise a model, leading to more stable model
generalisation [14]. CC has been successfully combined with supervised loss
function for semi-supervised learning in computer vision [15] and speech
applications [16, 17]. However, it has not yet been explored for speech
recognition in a fully unsupervised self-training setting.

In this work, we exploit CC to improve the robustness of training process for
unsupervised ASR personalisation. We introduce CC to the common unsupervised
self-training pipeline, however, perturbations are applied to both pseudo-
labelling and the training process, forcing the model to output a consistent
label in the vicinity of the training sample after filtering. To the best of
our knowledge, our work is the first to apply CC in the context of a fully
unsupervised setting for ASR. We compare against a state-of-the-art (SOTA)
unsupervised self-training method that combines a data filtering strategy with
an adapter based training mechanism [10] and a separate unsupervised
adaptation method based on entropy minimisation [8]. Our proposed method
achieves a 17.3% relative WERR compared to a pre-trained model, and
outperforms the recently developed techniques in literature, leading to a new
SOTA performance. The main contributions in this paper are summarised as
follows:

  * •

We propose a novel consistency based training method for unsupervised
personalisation of ASR models which outperforms current SOTA methods.

  * •

We empirically show that our proposed method is agnostic to the choice of
data-filtering methods and thus can be used in conjunction with any of them.

  * •

We evaluate the robustness of our method on a broad range of English accents
with pseudo-label WER ranging from 10% (high-quality) to 45% (low-quality).

The rest of the paper is organised as follows. Section 2 reviews related work
in the literature. Section 3 provides background details of an ASR model and a
confidence-score based filtering method used in the proposed framework.
Section 4 describes the proposed training method based on consistency. Section
5 presents implementation details and the experiment setup. Results and
analysis are presented in Section 6. Finally, the paper is concluded in
Section 7.

##  2 Related work

Data filtering is a commonly used pre-processing step when exploring
unlabelled data [18, 10]. Confidence based filtering methods have been
developed to select samples with good quality labels that yield lower Word
Error Rate (WER) among the whole unlabelled dataset [19, 18, 20, 21]. For
example, in DUST [18, 21], multiple ASR hypotheses are created using dropout,
and edit distances from reference (w/o dropout) are used to estimate the
confidence of ASR model on the generated transcript. Another class of methods
use a confidence estimation system that trains a separate neural network using
intermediate features derived from the ASR model [19, 20, 10]. For example,
both [20] and [10] use light-weight binary classifier models to predict
whether a given set of ASR features correspond to an error-free pseudo label
(WER=0) or not.

Model adaptation methods focus on learning highly compact speaker-dependent
parameter representations [22, 23, 24, 10]. Given a pre-trained model, a
diagonal linear transformation of the input features is learned to match the
distribution of test data to the training data [22]. An ASR model’s batch
normalization layer parameters can be learned from the target speaker data for
adaptation [23]. Additional speaker-dependent parameters can be introduced to
an ASR model for adaptation [24, 10]. Specifically, learning hidden unit
contributions (LHUC) trains embedding vectors with fixed dimensions to modify
the amplitudes of hidden unit activations [24]. Recently, the LHUC method has
been successfully applied to a Conformer based end-to-end ASR model for
speaker adaptation [10]. However, the major drawbacks of these techniques are
that the selection of adaptation layers in the ASR model is non-trivial and
adding new parameters changes the model architecture.

Recently, entropy minimisation based approaches have been shown effective for
unsupervised ASR domain adaptation [8, 12]. It aims to reduce uncertainty for
samples in a target domain by minimising an entropy loss based on output
probabilities from a pre-trained model given the target samples. However, one
issue of this method is that the model tends to drift away when the initial
predictions are incorrect.

##  3 Background

This section provides the background details of a streaming ASR system and the
Neural Confidence Measure (NCM) based data filtering method, which are
required for our proposed unsupervised personalisation pipeline.

###  3.1 ASR model

Fig. 1: Streaming two-pass end-to-end ASR model architecture. The first pass
model is a conformer based transducer. The second pass model is an attention-
based encoder-decoder model (LAS). NCM classifier is a confidence estimation
module that uses intermediate ASR features for WER based data filtering.

A state-of-the-art, two-pass Conformer-T model [25] is used as a pre-trained
ASR model for both filtering unlabelled data and adaptation to a target
speaker. As shown in Fig. 1, it consists of two sub-models, namely, a parent
model and a second-pass model. The parent model is a conformer transducer
[26], which consists of a transcription network, a prediction network and a
joint network. The transcription network contains a convolution subsampling
module followed by stacked convolution-augmented Transformer (Conformer)
blocks. The prediction network contains two LSTM layers. The second-pass model
is an LAS rescorer, consisting of an LSTM based encoder-decoder architecture.
The LAS encoder takes as input the parent’s transcription output, and the
first-pass prediction is refined by attending to the second-pass encoder
outputs. The model is trained using both a transducer [27] loss and a cross-
entropy (CE) loss calculated based on the output from the parent model and the
second-pass model, respectively.

###  3.2 Data filtering methods

Inspired by recent works that employ WER prediction models for data filtering
[28, 29, 20, 10, 11], we use one such existing model that was exclusively
developed for our custom ASR model described previously. The NCM binary
classification model [20] is used for filtering the transcripts generated by
our two-pass ASR model (see Fig. 1). The NCM model is made up of dense layers
and self-attention mechanism and takes two types of intermediate features from
the ASR model as input, namely, second pass decoder output and beam scores.
The second pass decoder outputs are logits from the LAS second pass decoder.
We use top-K logits corresponding to each decoded token for this decoder. Beam
scores are the log-probability scores for each beam assigned by the 2nd pass
decoder. Input features are obtained by running the ASR model with beam search
and the model is trained using binary cross-entropy loss. The output consists
of two classes, WER=0 and WER$>$0 and only the predicted WER=0 samples are
selected as the filtered set. Note that different to the original NCM model
that used 6 types of features, we use only two based on the relevance of
features as shown in the Table 1 of [20]. Once the training is complete, the
saved NCM model is used to perform pseudo-label filtering on the on-device
recorded personal data.

Additionally, we also experiment with two other techniques from literature for
data filtering, namely, DUST [18] and Confidence Thresholding (CT) [30]. DUST
is based on the intuition that for confident predictions, the ASR output would
remain unchanged even if some amount of uncertainty is introduced into the
model in the form of dropout. In practice, dropout layers are enabled in the
ASR model during evaluation and each utterance is forward propagated through
the network for multiple times to generate different hypotheses. Levenshtein
edit distances between a reference (hypothesis with no dropout) and each of
the hypotheses are calculated. If any of the distances corresponding to an
utterance is above a predefined threshold, that utterance is perceived as
having a lower confidence by the ASR model and hence rejected.

Confidence threshold for each utterance is obtained by taking sum of log
softmax scores across all tokens. WER values are binarized by taking WER=0 as
positive class and WER$>$0 as negative class. The threshold is then found by
taking the geometric mean of sensitivity and specificity using an ROC curve.

##  4 Proposed method

Fig. 2: Unsupervised personalisation pipeline based on data filtering and
consistency constraint

The proposed unsupervised personalisation pipeline starts from filtering
unlabelled data to using the filtered data to adapt a pre-trained ASR model
based on the Consistency Constraint (CC). The novelty of our adaptation method
is that it is the first work to apply CC in the context of fully unsupervised
ASR personalisation. The personalisation pipeline (see Fig. 2) based on the
proposed CC training is summarised in Algorithm 1. We first apply data
filtering DataFilter to the entire unlabelled set $\mathcal{X}$ to obtain
filtered set $\mathcal{\hat{X}}$. Subsequently, the model is trained on the
filtered set, involving $N$ rounds of pseudo-labelling and training, and in
each round the model $f$ is trained for $M$ epochs with the paired audio
samples and pseudo-labels $\mathcal{\hat{D}}$.

Algorithm 1 Proposed consistency based unsupervised personalisation pipeline

1:Input: ASR model $f$, weights $\theta$, unlabelled data $\mathcal{X}$

2:$\mathcal{\hat{X}}=\textsc{DataFilter}(\mathcal{X}),\;\theta_{0}=\theta$

3:for $i=0$, …, $N-1$ do // $N$ rounds

4:
$\mathcal{\hat{D}}=\\{(\hat{x},f(\textsc{SpecAug}(\hat{x}),\theta_{i})),\,\ldots\\}\;\;\forall\;\hat{x}\in\mathcal{\hat{X}}$

5: $\theta_{i+1}=\textsc{Train}(f,\theta_{i},\mathcal{\hat{D}},M)$ // $M$
epochs

6:end for

7:Output: $\theta_{N}$

The consistency is realised via pseudo-labelling on utterances augmented with
random data perturbations, after which the generated pseudo-labels are used to
train the model that are perturbed in a different way. The pseudo-label for
each utterance is updated frequently with the latest ASR model during the
personalisation process. CC has been commonly applied as a regularisation loss
to semi-supervised learning approaches [15, 16, 17, 31], in which a main loss
is calculated based on the available labelled source domain data. This work
uses CC as the only loss for unsupervised personalisation instead of using it
as an auxiliary loss.

Pseudo-labels are generated by decoding the output of the second-pass model
via beam search to produce hard labels, a transcription sequence $\hat{y}$.
During the training phase, only the parameters of the first-pass model are
adapted. Specifically, the first-pass transducer takes augmented input
features $\tilde{x}$ and outputs the posterior probabilities of all possible
alignments, which are used to calculate the loss against the pseudo-label
$\hat{y}$. The loss is then used to train the first-pass transducer comprised
of transcription network, prediction network, and joint network. Since the
first-pass model is a conformer transducer model, the loss function is
implemented as incorporating the CC within the standard RNN-T loss:

$\cal{L}=-\ln\Pr(\hat{\mathit{y}}|\tilde{\mathit{x}})$ (1)

SpecAugment [32, 33] is used as the data perturbation for both the pseudo-
labelling and the training. Due to randomisation, the SpecAugment applied to a
sample during training is different to the pseudo-labelling for the same
sample. During training, besides the data perturbation, the model is perturbed
by a dropout strategy [34]. The combination of data as well as model
perturbation forms a stronger augmentation for training than that of pseudo-
labelling.

##  5 Experiment setup

###  5.1 Data

The ASR model is pre-trained on 20K hours of English speech data. This
includes data from public speech datasets such as LibriSpeech [35] as well as
in-house data from a variety of domains such as search, telephony, far-field,
etc. The performance of this model on an in-house validation set of 6 hours
(5K utterances) is 15.69% WER.

For personalisation experiments, the proposed method is evaluated on an in-
house synthetic user data for a mobile phone use case. There are 12 speakers,
each containing three styles of speech: (i) application launch/download
commands (Apps), contact call/text commands (Contacts) and common real-world
voice assistant commands (Dictations). Table 1 provides examples for the three
types of data. The average audio length of Apps and Contacts samples is two
seconds. The filtered Apps and Contacts data are used to personalise a pre-
trained acoustic model. Then, the personalised model is evaluated on the
entire (filtered and unfiltered) Apps and Contacts data, and the held-out
Dictations data that is unseen during the training. On average, for each
speaker, the duration of entire Apps, Contacts and Dictation sets are 8.37,
16.93 and 6.47 minutes respectively.

###  5.2 ASR model configuration

The transcription network in the two-pass ASR model consists of 16 conformer
blocks. Each conformer block consists of one feed-forward layer block, one
Convolution Block, one multi-head self-attention block, another one feed-
forward layer block, followed by a layer normalisation layer. The prediction
network is constructed by stacking two LSTM layers with a dimension of 640.
The joint network is a dense layer. In the second-pass model, the encoder
contains one LSTM with dimension of 680, and the decoder contains two LSTMs
with dimension of 680. When decoding, the beam sizes for the first-pass model
and the second-pass LAS re-scoring are set as 4 and 1, respectively. No
language model is used for decoding.

Batch size of 16 and Adam optimizer [36] are used for all the adaptation
experiments. Learning rates for ASR model fine-tuning and LHUC based training
are set as 5e-6 and 1e-3, respectively. The SpecAugment used for both pseudo-
labelling and training has one frequency mask with size of 13 and two time
masks with size of 12. During training, the model is perturbed with dropout of
0.1.

Table 1: Adaptation data examples: Apps, Contacts, and Dictations Split | Examples
---|---
Apps | Open Messenger
| Install Snapchat
Contacts | Send a message to Anne Hathaway
| Call Emma Stone
Dictations | When does summer start
| Set an alarm for seven thirty

###  5.3 Data filtering method setup

In the NCM, the first fully connected block consists of two dense layers each
with 64 neurons followed by Tanh activation. This is followed by a self-
attention layer whose output is summed across all tokens for each utterance
and concatenated with the beam scores. This is then passed through another
fully connected block made up of two dense layers of 64 neurons each and an
output dense layer for the binary class prediction. Training the NCM model
uses a batch size of 32 and Adam optimizer [36] with an initial learning rate
of 1e-3. An exponential decay scheduler with decay rate 0.5 for every 500
training steps was also used. The K value is set as 4. The NCM model is
trained using an in-house 6 hours data, where it is split in the ratio 80:20
for training and validation.

The CT method uses the NCM training data (in-house dataset of 6 hours) for
finding the threshold, which is used to identify correct pseudo labels from
the Apps and Contacts sets of personal data. For DUST, the dropout of 0.2
(which is the same value used during the original ASR training) is enabled for
the transformers in ASR model and 5 hypotheses are generated for each
utterance. We tried different thresholds for edit distance from {0.1, 0.3,
0.5, 0.7} as recommended in [18] and found that the best results are obtained
with 0.1 threshold.

##  6 Results and analysis

This section presents the main results of the proposed method and shows the
comparison against recently developed existing unsupervised adaptation
methods. The second part of this section conducts a detailed ablation study on
the proposed method with respect to data filtering strategies and individual
user performance.

###  6.1 ASR personalisation results

Methods used as baselines in this work include noisy student training (NST)
[37] without accessing labelled source data, confidence score based LHUC [10],
and a recently proposed unsupervised test-time adaptation approach based on
entropy minimization (EM) [8]. The additional LHUC modules are inserted into
the streaming two-pass model used in this work. Different to the strategy used
in [10] that applies only one LHUC layer to the hidden output of the
convolution subsampling module, we apply multiple LHUC layers to the
transcription network, which yields better results than applying a single
layer. Specifically, one LHUC layer is applied to the output of the
convolution subsampling module and another to the output of each of the
Conformer block. In total, there are 17 LHUC layers. During adaptation, only
the LHUC layers are updated and the parameters of the main ASR model are
unchanged. The NST method employs data augmentation (SpecAugment) during
training, but not for pseudo-labelling. For the EM approach, we calculate and
minimise the entropy of the token-wise posterior probabilities output from the
second-pass LAS decoder. The NCM data filtering is applied before the EM
training. Both NST and EM also update only the first-pass model parameters to
make fair comparison with the proposed method.

Table 2: Word Error Rate (WER) of the proposed method and existing methods for unsupervised personalisation Methods | Apps | Contacts | Dictation
---|---|---|---
Pre-trained | 22.66 | 23.49 | 9.43
NST | 21.94 | 23.07 | 9.36
EM [8] | 20.26 | 23.23 | 9.53
NCM+EM | 19.12 | 22.22 | 8.86
NCM+LHUC [10] | 20.30 | 22.70 | 9.10
NCM+CC+LHUC | 19.30 | 21.99 | 8.64
NCM+CC | 18.73 | 21.79 | 8.67

Table 2 summarises the ASR performance of the baseline and proposed methods.
The proposed method achieves 17.3%, 7.2%, and 8.1% relative WERR on Apps,
Contacts, and Dictation, respectively, compared to the pre-trained model. It
outperforms both entropy minimisation and the confidence score based LHUC,
achieving a new SOTA result. The results show that the proposed method not
only improves the ASR performance on unlabelled data used for training (Apps &
Contacts), but also generalises well to unseen data (Dictation) spoken by the
target speaker. The second half of Table 2 shows the performance of the
proposed method (NCM+CC) by adaptation with and without LHUC and the results
indicate that addition of LHUC does not provide any improvement.

NCM+CC outperforms the combination of data filtering and entropy minimization
(NCM+EM). EM aims to reduce the output uncertainty when a model processes a
test sample. When the model prediction is incorrect, the EM approach trains
the model to be more confident about its incorrect decision. For the
consistency based approach, a model may output different predictions for a
highly uncertain sample with various perturbations. Since the model is trained
to map this same sample to different predictions, this sample will have lesser
impact on the model weight change compared to samples that provide consistent
pseudo-labels.

###  6.2 Ablation study

Table 3: The effect of three data filtering methods (CT, DUST, NCM) on the unsupervised personalisation performance Methods | Apps | Contacts | Dictation
---|---|---|---
CC | 21.25 | 22.71 | 9.04
CC (WER=0) | 20.38 | 22.40 | 8.87
CT+CC | 18.91 | 22.10 | 8.75
DUST+CC | 18.93 | 21.87 | 8.69
NCM+CC | 18.73 | 21.79 | 8.67

We first investigate the effect of quality of audio samples on the ASR
personalisation performance. The Consistency Constraint (CC) based method is
tested either on the unfiltered whole data set or the filtered data set based
on three filtering strategies, namely Confidence Thresholding (CT), DUST, and
the NCM. The method is also tested on training with only audio samples that
the ASR system recognises correctly (WER=0), selected based on the ground-
truth transcriptions.

$5$$10$$15$$20$$0$$5$$10$$15$$20$RoundsWERR (%)AppsEpochs = 1Epochs = 3Epochs
= 5NSTCC$5$$10$$15$$20$$0$$2$$4$$6$$8$RoundsWERR
(%)Contacts$5$$10$$15$$20$$0$$2$$4$$6$$8$$10$$12$RoundsWERR (%)Dictation

Fig. 3: Word Error Rate Reduction (WERR) compared to the pre-trained model for
Apps, Contacts & Dictation using consistency training (CC) and unsupervised
NST for 20 rounds with a choice of 1, 3 or 5 epochs per round. Higher values
are better. Plot values are smoothed using an exponential moving average with
weight of 0.6. Best viewed in colour.

Fig. 4: ASR personalisation results for each of the 12 individual users. The
pre-trained model, NST trained on unfiltered data, and the proposed method are
compared in the plot. (Top: Apps data, Middle: Contacts data, Bottom:
Dictation data.)

In Table 3, we first observe that the CC trained with audio samples that the
model recognises correctly (WER=0) performs better compared with the CC
trained with the whole data (CC). This demonstrates the importance of adapting
a given model on samples that yield low WER. The second half of Table 3 shows
the results of CC with different filtering strategies. Remarkably, though
trained on filtered data that contains samples with erroneous labels, the CC
based training outperforms that trained with WER=0 samples. It suggests that
the CC based training is able to explore samples with erroneous labels to
adapt the model.

All three data filtering strategies achieve similar performance improvement.
Out of the three, NCM is favored for on-device personalisation. NCM is a
lightweight model that requires only one-time training and can be easily
deployed on device. The CT method requires manual tuning of the threshold
value, which is challenging for an end-to-end ASR model because of the well
known overconfidence issue that exists in the end-to-end ASR models [29].
However, the NCM can exploit large amounts of training data on server to
automatically learn the boundary between reliable and unreliable samples by
exploiting multiple intermediate ASR features. Compared to DUST that requires
multiple forward passes, which is undesirable due to time and resource
constraints, the NCM obtains the result with a single forward pass.

We perform ablation experiments to study the effect of increasing the number
of rounds and epochs per round on the overall WERR. We use epochs
$\in\\{1,3,5\\}$ for up to 20 rounds and compare our method against
unsupervised NST. Fig. 3 shows that training with five epochs per round can
lead to divergence for Dictation which is a classic example of overfitting due
to increased model updates. Conversely, training for a single epoch per round
leads to sub-optimal convergence due to the increased stochasticity in
regenerating pseudo-labels with input augmentation every round. Our method
performs up to 40% better than unsupervised NST which is more susceptible to
overfitting due to being easily stuck in a local minima.

We investigate the performance of our proposed method on each individual user,
and the analysis is shown in Fig. 4. In comparison to the pre-trained model
and the baseline NST trained on unfiltered data, our method achieves better or
equivalent recognition accuracy for most users on both held-in data (Apps &
Contacts) and held-out data (Dictation). There is a wide range of speech
recognition accuracy among the test users, whose WERs range from 10% to 45%.
The results demonstrate that the proposed method improves the robustness of
the training process to erroneous labels.

##  7 Conclusions

This work introduces a novel unsupervised personalisation training method to
address a domain shift issue when an ASR model is deployed in the wild. The
proposed method performs data filtering of unlabelled user data and applies a
consistency constraint to the training process. A neural confidence measure
approach has been employed for data filtering and has demonstrated to
effectively discard unreliable audio samples that yield erroneous pseudo-
labels. To apply consistency constraint, data perturbation is introduced to
the iterative pseudo-labelling process, forcing the ASR model to predict the
same labels on the same sample with various versions of perturbations.
Experiments show that the proposed method reduces the negative impact of low
quality samples during training and improves model generalisation to test
domain data. We further separately evaluate the consistency based training in
combination with three existing filtering methods and all filtering methods
achieve similar results. This suggests that the consistency based training can
be used in conjunction with a wide range of data filtering strategies.

## References

  * [1] Yong Zhao, Jinyu Li, Shixiong Zhang, Liping Chen, and Yifan Gong,  “Domain and speaker adaptation for cortana speech recognition,”  in ICASSP, 2018.
  * [2] Zhouyuan Huo, Dongseong Hwang, Khe Chai Sim, Shefali Garg, Ananya Misra, Nikhil Siddhartha, Trevor Strohman, and Françoise Beaufays,  “Incremental layer-wise self-supervised learning for efficient unsupervised speech domain adaptation on device,”  in Interspeech, 2022.
  * [3] Khe Chai Sim, Petr Zadrazil, and Françoise Beaufays,  “An investigation into on-device personalization of end-to-end automatic speech recognition models,”  in Interspeech, 2019.
  * [4] Peter Bell, Joachim Fainberg, Ondrej Klejch, Jinyu Li, Steve Renals, and Pawel Swietojanski,  “Adaptation algorithms for neural network-based speech recognition: An overview,”  IEEE Open Journal of Signal Processing, vol. 2, pp. 33–66, 2020\.
  * [5] Khe Chai Sim, Petr Zadražil, and Françoise Beaufays,  “An investigation into on-device personalization of end-to-end automatic speech recognition models,”  in Interspeech, 2019.
  * [6] Zhong Meng, Jinyu Li, Yashesh Gaur, and Yifan Gong,  “Domain adaptation via teacher-student learning for end-to-end speech recognition,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 268–275.
  * [7] Yan Huang, Guoli Ye, Jinyu Li, and Yifan Gong,  “Rapid speaker adaptation for conformer transducer: Attention and bias are all you need,”  in Interspeech, 2021.
  * [8] Guan-Ting Lin, Shang-Wen Li, and Hung yi Lee,  “Listen, adapt, better WER: Source-free single-utterance test-time adaptation for automatic speech recognition,”  in Interspeech, 2022.
  * [9] Marc Delcroix, Shinji Watanabe, Atsunori Ogawa, Shigeki Karita, and Tomohiro Nakatani,  “Auxiliary feature based adaptation of end-to-end ASR systems,”  in Interspeech, 2018.
  * [10] Jiajun Deng, Xurong Xie, Tianzi Wang, Mingyu Cui, Boyang Xue, Zengrui Jin, Mengzhe Geng, Guinan Li, Xunying Liu, and Helen M. Meng,  “Confidence score based conformer speaker adaptation for speech recognition,”  in Interspeech, 2022.
  * [11] Jiajun Deng, Xurong Xie, Tianzi Wang, Mingyu Cui, Boyang Xue, Zengrui Jin, Guinan Li, Shujie Hu, and Xunying Liu,  “Confidence score based speaker adaptation of conformer speech recognition systems,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1175–1190, 2023.
  * [12] Changhun Kim, Joonhyung Park, Hajin Shim, and Eunho Yang,  “SGEM: Test-time adaptation for automatic speech recognition via sequential-level generalized entropy minimization,”  in Interspeech, 2023.
  * [13] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu,  “Model adaptation: Unsupervised domain adaptation without source data,”  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [14] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen,  “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,”  in NIPS, 2016.
  * [15] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li,  “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,”  Advances in neural information processing systems, vol. 33, pp. 596–608, 2020.
  * [16] Felix Weninger, Franco Mana, Roberto Gemello, Jes’us Andr’es-Ferrer, and Puming Zhan,  “Semi-supervised learning with data augmentation for end-to-end ASR,”  in Interspeech, 2020.
  * [17] Ashtosh Sapru,  “Using data augmentation and consistency regularization to improve semi-supervised speech recognition,”  in Interspeech, 2022.
  * [18] Sameer Khurana, Niko Moritz, Takaaki Hori, and Jonathan Le Roux,  “Unsupervised domain adaptation for speech recognition via uncertainty driven self-training,”  ICASSP, 2020.
  * [19] Kaustubh Kalgaonkar, Chaojun Liu, Yifan Gong, and Kaisheng Yao,  “Estimating confidence scores on ASR results using recurrent neural networks,”  in ICASSP, 2015.
  * [20] Ashutosh Gupta, Ankur Kumar, Dhananjaya N. Gowda, Kwangyoun Kim, Sachin Singh, Shatrughan Singh, and Chanwoo Kim,  “Neural utterance confidence measure for RNN-Transducers and two pass models,”  ICASSP, 2021.
  * [21] Nauman Dawalatabad, Sameer Khurana, Antoine Laurent, and James Glass,  “On unsupervised uncertainty-driven speech pseudo-label filtering and model calibration,”  in ICASSP, 2023.
  * [22] Zhong-Qiu Wang and Deliang Wang,  “A joint training framework for robust automatic speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, pp. 796–806, 2016.
  * [23] Zhongqiu Wang and Deliang Wang,  “Unsupervised speaker adaptation of batch normalized acoustic models for robust ASR,”  ICASSP, 2017.
  * [24] Pawel Swietojanski, Jinyu Li, and Steve Renals,  “Learning hidden unit contributions for unsupervised acoustic model adaptation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 8, pp. 1450–1463, 2016.
  * [25] Jinhwan Park, Sichen Jin, Junmo Park, Sungsoo Kim, Dhairya Sandhyana, Chang heon Lee, Myoungji Han, Jungin Lee, Seokyeong Jung, Chang Woo Han, and Chanwoo Kim,  “Conformer-based on-device streaming speech recognition with KD compression and two-pass architecture,”  2022 IEEE Spoken Language Technology Workshop (SLT), pp. 92–99, 2023.
  * [26] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Interspeech, 2020.
  * [27] Alex Graves,  “Sequence transduction with recurrent neural networks,”  ArXiv, vol. abs/1211.3711, 2012.
  * [28] Ankur Kumar, Sachin Singh, Dhananjaya Gowda, Abhinav Garg, Shatrughan Singh, and Chanwoo Kim,  “Utterance confidence measure for end-to-end speech recognition with applications to distributed speech recognition scenarios.,”  in Interspeech, 2020.
  * [29] Qiujia Li, David Qiu, Yu Zhang, Bo Li, Yanzhang He, Philip C. Woodland, Liangliang Cao, and Trevor Strohman,  “Confidence estimation for attention-based sequence-to-sequence models for speech recognition,”  ICASSP, 2020.
  * [30] Jacob Kahn, Ann Lee, and Awni Y. Hannun,  “Self-training for end-to-end speech recognition,”  ICASSP, 2019.
  * [31] Marvin Zhang, Sergey Levine, and Chelsea Finn,  “MEMO: test time robustness via adaptation and augmentation,”  in NeurIPS, 2022.
  * [32] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  in Interspeech, 2019.
  * [33] Daniel S. Park, Yu Zhang, Chung-Cheng Chiu, Youzheng Chen, Bo Li, William Chan, Quoc V. Le, and Yonghui Wu,  “Specaugment on large scale datasets,”  ICASSP, 2019.
  * [34] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov,  “Improving neural networks by preventing co-adaptation of feature detectors,”  ArXiv, vol. abs/1207.0580, 2012.
  * [35] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an ASR corpus based on public domain audio books,”  in ICASSP, 2015.
  * [36] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  International Conference for Learning Repreentations (ICLR), 2015\.
  * [37] Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le,  “Improved noisy student training for automatic speech recognition,”  in Interspeech, 2020.