# Knowledge Distillation from Multiple Foundation Models for End-to-End Speech
Recognition

Xiaoyu Yang, , Qiujia Li,
Chao Zhang, Philip C. Woodland This paper was produced by the IEEE Publication
Technology Group. They are in Piscataway, NJ.

###### Abstract

Although large foundation models pre-trained by self-supervised learning have
achieved state-of-the-art performance in many tasks including automatic speech
recognition (ASR), knowledge distillation (KD) is often required in practice
to transfer the knowledge learned by large teacher models into much smaller
student models with affordable computation and memory costs. This paper
proposes a novel two-stage KD framework to distil the knowledge from multiple
speech foundation models as teachers into a single student neural transducer
model for ASR. In the first stage, the student model encoder is pre-trained
using the embeddings extracted from multiple teacher models. In the second
stage, the student encoder is fine-tuned with the audio-text pairs based on
the ASR task. Experiments on the LibriSpeech 100-hour subset show that the
proposed KD framework improves the performance of both streaming and non-
streaming student models when using only one teacher. The performance of the
student model can be further enhanced when multiple teachers are used jointly,
achieving word error rate reductions (WERRs) of 17.5% and 10.6%. Our proposed
framework can be combined with other existing KD methods to achieve further
improvements. Further WERRs were obtained by incorporating extra unlabelled
data during encoder pre-training, leading to a total relative WERR of 55.0% on
the non-streaming student model.

###### Index Terms:

Foundation model, teacher-student training, multi-teacher knowledge
distillation, neural transducer, ASR

##  I Introduction

Automatic speech recognition (ASR) is the task of mapping an input speech
signal to its corresponding text sequence. An end-to-end (E2E) trainable ASR
model implements this using a single trainable neural network such as neural
transducers [1, 2] and attention-based encoder-decoder models [3, 4, 5, 6]. As
a result, E2E trainable models have become the most prevalent ASR approach in
both industry and academia.

In real-world applications, the size of an E2E trainable ASR system is usually
limited by computation and memory budgets. Knowledge distillation (KD) [7],
also known as teacher-student training, is a commonly used approach for model
compression, from the original large model (”teacher”) to the compressed small
model (”student”). Instead of training the student purely using ground truth
labels, KD also enables the use of the outputs of the teacher model, namely
teacher labels. Different KD loss functions and training targets for KD to
improve the performance of the student ASR model have been previously
explored. For example, Takashima et al [8] uses the final output distributions
as the teacher labels and the Kullback-Leibler (KL) divergence as the KD loss
together with the connectionist temporal classification (CTC) loss for ASR. In
[9] hidden layer representations or embeddings were extracted as the teacher
labels for training the student ASR model. Unlike the parameter quantisation-
based methods [10, 11] that usually require the teacher and student to share
the same model structure, KD does not impose this constraint, making it a more
flexible approach. Meanwhile, as ground-truth labels are not required, KD can
be used as a semi-supervised training method to leverage a large amount of
unlabelled data to improve the student [12, 13]. Furthermore, the student
model can also be jointly trained using multiple teachers. Since the ensemble
of multiple models can usually outperform a single model [14], it can serve as
a more reliable teacher during KD training. The weighted sum of the
distributions output from multiple teachers is used as distillation targets in
[15], leading to a larger performance gain compared to using only a single
teacher. Wong et al [16] ensembled the teachers with different output nodes
obtained by decision tree clustering, and sequence-level KL divergence was
also developed as an improved KD loss [17]. In [18], one teacher from a
teacher pool was selected in every mini-batch to generate the teacher labels.

Recently, pre-trained foundation models [19] obtained by self-supervised
learning (SSL) [20] have achieved state-of-the-art performance on many speech
processing tasks [21, 22, 23, 24, 25, 26]. After being pre-trained on a large
amount of unlabelled speech data, foundation models can be fine-tuned with the
labelled data for specific downstream tasks, such as ASR [21, 22], speaker
diarisation [23, 27], and emotion recognition [28, 29] etc. However,
foundation models tend to be very large (e.g. from hundreds of millions [21,
22] to even billions [22] of model parameters) in order to fully leverage the
richness and diversity of the unlabelled data during pre-training. This
prevents their uses in resource-constrained scenarios, such as on-device
streaming ASR, due to the large latency, computational footprint and inference
cost. Therefore, it is of great interest to compress these large foundation
models with as little performance degradation as possible. To this end, Peng
et al [30] projected the embeddings generated by a wav2vec 2.0 model [21] to
distributions and performed KD based on the KL divergence. Chang et al [31]
proposed a multi-task KD framework by predicting embeddings extracted from
different hidden layers of a HuBERT [22] layer with multiple heads.

Due to use of various loss functions, model structures, and training data,
embeddings generated by different foundation models can be very different and
complementary. Therefore, further potential performance improvement could be
achieved if a student model can learn from multiple foundation models. In this
paper, we propose an efficient 2-stage distillation method to train student
neural transducers for E2E ASR using the knowledge distilled from multiple
foundation models as teachers. KD is carried out first at the embedding level
and then at the hypothesis level. At the embedding level, the student jointly
learns the embeddings from multiple teacher models using a regression loss. At
the hypothesis level, the top two ASR hypotheses obtained by beam search
(termed as 1-best and 2-best) can be used as the supervision in the transducer
loss. The main contributions of this paper are summarised as follows:

  1. 1.

Proposes a two-stage KD framework which is effective both for streaming and
non-streaming transducer models;

  2. 2.

Demonstrates that using multiple teachers in the proposed KD framework leads
to lower student word error rates (WERs) compared to a single teacher;

  3. 3.

Shows that the proposed KD framework is complementary with existing KD methods
for neural transducers, and lower student WERs can be achieved when
incorporating more unlabelled data.

In the rest of this paper, Sec. II briefly reviews the background and related
work. In Sec. III, details of the proposed two-stage KD framework are
presented. The experimental setups and results are given in Sec. V and VI.
Finally, conclusions are drawn in Sec. VII.

##  II Related work

###  II-A Neural transducer models

The neural transducer [1, 2] is a prevalent E2E trainable ASR approach. It
consists of an encoder, a predictor and a joint network (see Fig. 1). Given a
pair of input feature sequence $\bm{X}=\bm{x}_{1},\ldots,\bm{x}_{M}$ of length
$M$ and its transcription $\bm{y}=y_{1},\ldots,y_{U}$ of length $U$, the
neural transducer is trained to maximise the conditional probability
$P(\bm{y}|\bm{X})$. The encoder receives $\bm{X}$ and generates an acoustic
embedding sequence $\bm{F}=\bm{f}_{1},\ldots,\bm{f}_{T}$ of length $T$. In
practice, $T$ is often set to $M$, or a value smaller than $M$ by a constant
factor (e.g. when $\bm{X}$ is a raw waveform instead of acoustic feature
sequence). The predictor produces a text embedding $\bm{g}_{u}$ for each text
token $y_{u}$, and $\bm{G}=\bm{g}_{1}\ldots,\bm{g}_{U}$. The joint network
takes each pair of $\bm{f}_{t}$ and $\bm{g}_{u}$ as input and generates a
$V$-dimensional (-d) distribution, where $V$ is the size of the output layer
(i.e the token vocabulary) including an extra blank symbol which is removed
from the final output sequence.

A (transducer) lattice refers to the collection of all valid alignments
between $\bm{F}$ and $\bm{y}$ (and thus $\bm{X}$ and $\bm{y}$) traversing from
$(t=0,u=0)$ to $(t=T,u=U)$. The training objective $P(\bm{y}|\bm{X})$ is then
the sum of the probability of all valid alignments:

$\displaystyle
P(\bm{y}|\bm{X})=\sum\nolimits_{\alpha\in\mathcal{A}}P(\alpha|\bm{X})$ (1)

where $\mathcal{A}$ is the set containing all valid alignments between
$\bm{X}$ and $\bm{y}$. Enumerating all valid alignments is computationally
prohibitive and the forward-backward procedure can be used [1, 2] to
efficiently compute the summation (the RNNT loss). During decoding, Viterbi
beam search can be applied to generate output tokens in a time-synchronous
manner. Compared to other E2E ASR approaches, including connectionist temporal
classification (CTC) [32] attention-based encoder-decoder (AED) models [3, 5,
33], the neural transducer generally achieves lower WERs than CTC and more
naturally handles streaming input speech than AED.

Figure 1: A neural transducer model.

In addition, we use the term distribution lattice to refer to the $(T,U,V)$-d
tensor composed by $\bm{F}$, $\bm{G}$ and all output distributions, which is
denoted as $\bm{Z}$. Each node in the distribution lattice, $\bm{Z}(t,u,k)$,
represents the probability of generating $k$-th token in the vocabulary at
time $t$ after generating the partial text sequence $y_{1},\ldots,y_{u}$.

###  II-B Knowledge distillation for neural transducers

KD [7] is a widely used technique that compresses a large teacher model to a
small student model by training the small model to match the output of the
large model. Depending on the output of the teacher model, different loss
functions can be used. For example, the KL-divergence is commonly used as the
distillation loss to match probability distributions whereas $L_{1}$ or
$L_{2}$ loss is more appropriate for matching hidden representations. Since
the output of a neural transducer for an input acoustic sequence is a
distribution lattice, applying KD to neural transducers is inherently
expensive. A straightforward way of applying KD to neural transducers is to
use the KL-divergence between the teacher distribution lattice
$\bm{Z}_{\mathcal{T}}(t,u,k)$ and the student distribution lattice
$\bm{Z}_{\mathcal{S}}(t,u,k)$ as the distillation loss:

$\displaystyle\mathcal{L}_{\text{KD}}=-\sum_{t=1}^{T}\sum_{u=1}^{U}\sum_{k=1}^{V}\bm{Z}_{\mathcal{T}}(t,u,k)\log\bm{Z}_{\mathcal{S}}(t,u,k).$
(2)

In practice, however, directly applying the KL-divergence loss defined in Eqn.
(2) is inefficient, due to the high computational and storage costs for the
distribution lattice. To reduce cost, [34] kept only three elements in each
node of the original distribution lattice: the probability of the correct
symbol $y_{u}$, the probability of the blank symbol and the sum of the
probabilities of all other symbols, and used this collapsed distribution
lattice for KD:

$\displaystyle\mathcal{L}_{\text{KD}}\approx-\sum_{t=1}^{T}\sum_{u=1}^{U}\sum_{k=1}^{3}\bm{Z}^{\prime}_{\mathcal{T}}(t,u,k)\log\bm{Z}^{\prime}_{\mathcal{S}}(t,u,k)$
(3)

where $\bm{Z}^{\prime}_{\mathcal{T}}(t,u,k)$ and
$\bm{Z}^{\prime}_{\mathcal{S}}(t,u,k)$ are the collapsed versions of the
teacher and student posterior lattice. As a result, the computational
complexity is reduced from $\mathcal{O}(TUV)$ to $\mathcal{O}(TU)$. However, a
large amount of information is discarded and this is sub-optimal for KD
training. To keep the full distribution while being efficient, [35]
approximated the distribution lattice with its one-best alignment, which was
obtained while generating the transducer lattice and saved prior to training.
KD then uses only on the one-best alignment instead of the whole lattice,
leading to another approximation of the KD loss:

$\displaystyle\mathcal{L}_{\text{KD}}\approx-\sum_{(t,u)\in\textsc{1Best}}\sum_{k=1}^{V}\mathbf{Z}_{\mathcal{T}}(t,u,k)\log\mathbf{Z}_{\mathcal{S}}(t,u,k).$
(4)

As the length of an alignment is $T+U$, this approximation reduces the
computation from $\mathcal{O}(TUV)$ to $\mathcal{O}((T+U)V)$. When $V$ is
large, a potential drawback of this method is that it would still use a large
amount of memory.

As a workaround, [36] used the encoder features from an intermediate layer of
a pre-trained teacher model for KD. Since the encoder output dimension of a
teacher model is fixed, the computational cost is no longer related to $V$. To
reduce the amount of data received from the teacher in on-the-fly KD, a multi-
codebook vector quantisation method was used to compress 32-bit “float”
features to 8-bit integers. During KD, the student model tries to predict the
codebook indexes of teacher embeddings. This comes at a cost of slight
performance degradation when compared to $L_{1}$ and $L_{2}$ loss [36].

Natural support for streaming is one reason why neural transducers are gaining
more popularity. Streaming models have less future context and tend to emit
non-blank symbols later than their non-streaming counterparts. Therefore,
performing knowledge distillation on streaming transducers is more challenging
when its teacher model is non-streaming. A 3-step distillation process is
proposed in [37] to delay the emission of the non-streaming teacher model
using a streaming model before performing KD. Yang et al [35] modified the
1-best alignment KD loss to improve the performance of the streaming student
model. Directly applying Eqn. (4) to a streaming student could be problematic
if the pre-trained teacher model is non-streaming as this would force the
student to “guess” the future. Therefore, a time-shift variable $\tau$ was
introduced to allow the student streaming model to emit symbols later than the
teacher model. This leads to a modified version of Eqn. (4):

$\displaystyle\mathcal{L}_{\text{KD}}\approx-\sum_{(t,u)\in\textsc{1Best}}\sum_{k=1}^{K}\mathbf{Z}_{\mathcal{T}}(t,u,k)\log\mathbf{Z}_{\mathcal{S}}(t+\tau,u,k).$
(5)

###  II-C Self-supervised learning for speech foundation models

A foundation model refers to a large model pre-trained on a vast quantity of
unlabelled data at scale based on SSL [19]. It has become a new paradigm that
an increasing number of research works are conducted by fine-tuning pre-
trained foundation models using labelled data. Recently, many speech
foundation models were developed and achieved state-of-the-art performance for
various speech processing tasks [21, 22, 23]. Wav2vec 2.0 [21] used product
quantisation to quantise acoustic features encoded by a stack of convolutional
layers. The Gumbel softmax [38] is applied to make the codebook selection
process differentiable. SSL pre-training is carried out using a contrastive
loss to encourage the Transformer encoder to discriminate each ground truth
quantised vector from a set of distractors. HuBERT [22] applies k-means
clustering to the speech representations and generates pseudo-class labels
using the clustering results. During pre-training, the model is trained to
predict the ground truth class labels of both masked and unmasked timestamps.
To refine the clustering results, the features from an intermediate
Transformer block are extracted for the second round of k-means clustering.
WavLM [23] adopted the same idea as HuBERT for generating pseudo labels during
pre-training, but used a more diverse pre-training dataset to increase the
model’s generalisation capability. It also performs speech denoising modelling
during pre-training by adding noises and overlapping speech to the input. This
not only enhances the robustness of the learned features, but also makes it
more suitable for non-ASR tasks.

###  II-D Multi-teacher knowledge distillation

In teacher-student training, the quality of the teacher model is crucial to
the performance of the student model. Ensemble [39, 40, 41] is a common
approach for constructing stronger models by combining the output of different
models. Lindqvist et al [15] used an ensemble as the teacher model to perform
KD training, where a student model learns from the ensemble distribution
obtained by averaging the distribution. However, the individual teacher
information cannot be recovered from the weighted mean distribution, limiting
the knowledge to be transferred from each information source. To address this
information loss, [18] proposed to randomly sample a teacher from a teacher
pool for each training batch to so that the student model is exposed to
individual teachers. In [42], embeddings extracted from multiple teacher
models are used as teacher targets and $L_{2}$ loss between the teacher
embeddings and student embeddings was added as an auxiliary loss for training.
Regarding ASR, [16] proposed an ensemble of hybrid teacher acoustic models
constructed based on different tied-state triphones or tri-characters derived
using the decision tree clustering approach.

##  III Multi-Teacher Knowledge Distillation for Neural Transducers

###  III-A Motivation

The quality of the teacher model is a critical factor in KD training, since
the resulting student model tends to have better performance with labels
generated by better teacher models. Using an ensemble of multiple models is a
common strategy to improve the teacher quality [39, 43] and various attempts
have been made to distil the knowledge from an ensemble of teacher models [44,
45].

SSL pre-training utilises the richness of unlabelled data and helps foundation
models learn better feature representations. Good performance can be achieved
when fine-tuning with even a small amount of labelled data. The data
representations learned through different pre-training tasks could be very
different, and maybe even complementary to each other. Therefore, it is
desirable to use multiple SSL pre-trained foundation models to leverage their
complementarity. In practice, it is reasonable to have multiple foundation
models with similar performance, which can have different model structures or
be trained on different datasets. In this work, we propose a two-stage multi-
teacher KD framework, where a student transducer model learns from multiple
teacher foundation models at the same time. In the first stage, the encoder of
the student transducer model is learned to regress the teacher models’
embeddings. Extra unlabelled speech can be incorporated at this stage as no
ground truth transcriptions are required. In the second stage, a neural
transducer is initialised from the encoder in stage one, which will be then
fine-tuned with audio-text pairs using the RNNT loss. Existing KD methods can
be combined during the fine-tuning stage for performance improvement.

Figure 2: The proposed 2-stage KD framework. The left part describes the first
stage: encoder pre-training. The teacher model receives raw waveform as input
and generates embeddings. The right part is the second stage of training,
where the student model is initialised from the well-trained encoder in the
first stage and fine-tuned with audio-text pair.

###  III-B Encoder pre-training

The encoder of a neural transducer often has the majority of the total number
of parameters. Therefore, we propose to carry out KD on encoder embeddings.
The proposed KD training framework consists of two stages. In the first stage,
only the encoder of the student model is considered while the decoder and
joint network remain frozen. Assuming the number of teacher models to be $N$
and the encoder output dimension of each teacher to be $D^{\mathcal{T}_{n}}$.
Given the same input acoustic features $\bm{X}$, the encoder of $n$-th teacher
model generates
$\mathcal{E}^{\mathcal{T}_{n}}=\bm{e}^{\mathcal{T}_{n}}_{1},...,\bm{e}^{\mathcal{T}_{n}}_{T}$
as output. The encoder of the student model also generates embeddings
$\mathcal{E}^{\mathcal{S}}=\bm{e}^{\mathcal{S}}_{1},...,\bm{e}^{\mathcal{S}}_{T}$
of dimension $D^{\mathcal{S}}$. During training, the student attempts to
regress the teacher embeddings $\mathcal{E}_{n},n=1,...,N$ by sampling one
teacher embedding for each training utterance in the mini-batch. Assuming the
$n$-th teacher model is selected, the loss function for a training sample is
as follows:

$\displaystyle\mathcal{L}_{\text{KD}}=\frac{1}{T}\sum_{t=1}^{T}\textsc{Dist}(\textsc{LossNet}(\bm{e}^{\mathcal{S}_{t}}),\bm{e}^{\mathcal{T}_{n}}_{t}),$
(6)

where $\textsc{LossNet}(\cdot)$ is a linear transformation that maps the
student vector from $D^{\mathcal{S}}$ to $D^{\mathcal{T}}_{n}$ and
$\textsc{Dist}(\cdot)$ is any function that measures the distance between two
vectors of the same dimension, such as the commonly used $L_{1}$ and $L_{2}$
distances. Since KD is carried out at the encoder level, the computational and
storage costs for encoder embedding level pre-training are independent of the
output vocabulary size $V$, which alleviates the computation issue in
distribution-based KD approaches for neural transducers[34, 35].

Streaming models tend to emit symbols later due to the lack of future context.
Therefore, directly applying Eqn. (6) to streaming student models can be
problematic. Inspired by [35], a time-delay factor $\tau$ is adopted during
the first stage of training for streaming student models. The $t$-th frame of
$n$-th teacher model’s embedding will be aligned to the
$\left(t+\tau\right)$-th frame of the student model. As a consequence, the
last $\tau$ frames in the teacher model are discarded as no corresponding
frames exist in the student model. This leads to a modified version of the
distillation loss:

$\displaystyle\mathcal{L}_{\text{KD}}=\frac{1}{T}\sum_{t=1}^{T-\tau}\textsc{Dist}(\textsc{LossNet}\left(\bm{e}^{\mathcal{S}}_{t}\right),\bm{e}^{\mathcal{T}_{n}}_{t+\tau}).$
(7)

Note that due to the introduction of the time-delay factor, the last $\tau$
frames of the student embedding have to be discarded during KD loss
computation as no teacher embeddings exist for them. Intuitively, $\tau$
controls the trade-off between emission latency and model performance: a
larger $\tau$ allows the student model to have more future contexts but
increases emission delay and vice versa. However, a recent method applying
this idea claims that the latency of the student model also decreases while
the recognition accuracy also improves [36]. With a carefully tuned $\tau$,
this could be because the fixed time delay not only provides good supervision
but also enforces the student model to emit earlier.

###  III-C Supervised fine-tuning

In the second stage, the encoder pre-trained with embeddings from different
teacher models in stage 1 is adopted as the initialisation of a student
transducer model’s encoder. The decoder and the joint network are randomly
initialised. If the encoder is sufficiently trained, it should be able to
accurately reproduce the output of the teacher model given the same input
features. Therefore, this encoder can be treated as a weaker approximation to
the self-supervised pre-trained model. The encoder-initialised student model
is then fine-tuned with labelled speech to perform ASR tasks. To avoid
catastrophic forgetting, a tri-stage learning rate schedule is used so that
the model does not quickly depart from the initialisation. During the first
warm-up stage, the learning rate increases linearly from a very small value to
a warm-up learning rate. The learning rate keeps the same fixed value during
the second stage and then decreases linearly to the final learning rate. The
RNNT loss [1] is used to train the whole student model. Note that there is a
fundamental difference between the proposed KD framework with existing
embedding-level KD methods. Our framework splits the embedding learning and
ASR training into two distinct processes whilst other methods perform KD in a
multi-task fashion by treating KD as an auxiliary task during ASR training. An
illustration of the proposed 2-stage KD framework is shown in Fig. 2

##  IV Combination with other KD methods

The proposed KD method performs distillation only using the output of the
encoder, which does not involve the predictor and the joint network in the
student neural transducer. Therefore, it is likely to be compatible with
distribution-based KD methods as they focus on a different part of a
transducer model. Here, we use the 1-best alignment KD [35] in addition to the
proposed encoder embedding level KD. Under the single-teacher scenario, the
1-best loss can be added during supervised fine-tuning as an auxiliary loss to
the normal RNNT loss. However, modifications need to be made when integrating
the 1-best alignment-based KD under a multi-teacher KD setup. In this paper,
we extend the 1-best alignment KD to n-best alignment KD by using the 1-best
alignment from each individual teacher as the teacher label. By doing so, the
loss function of n-best alignment KD becomes a weighted sum of $N$ 1-best KD
loss defined in Eqn. (4):

$\displaystyle\mathcal{L}_{\text{nBest}}=\sum_{n=1}^{N}\omega_{n}\mathcal{L}^{n}_{\text{1Best}}$
(8)

where $\omega_{n}$ is the weight of the alignment decoded from the $n$-th
teacher model and it holds $\sum_{n=1}^{N}\omega_{n}=1$.

This auxiliary loss is interpolated with the original RNNT loss during the
second stage of training, resulting in the final loss function:

$\displaystyle\mathcal{L}_{\text{final}}=\mathcal{L}_{\text{RNNT}}+\lambda\mathcal{L}_{\text{nBest}},$
(9)

where $\lambda$ is a tunable coefficient controlling the contribution of the
auxiliary loss.

##  V Experimental Setup

###  V-A Model

The Base version of Wav2vec 2.0 [21], HuBERT [22] and WavLM [23] were selected
as teacher models. For convenience, they will be referred to as “W2v2”, “HU”
and “WL” in the following experiments. Note that there are two Base versions
of WavLM, with different amounts of unlabelled data used during pre-training.
Here, the one pre-trained on MIX-94k hour is chosen as it is expected to have
better performance. All three pre-trained models take the raw waveform as
input and generate 50 encoder frames per second, whereas the most commonly
used filter bank features for E2E trainable ASR usually operates at 25Hz. An
extra subsampling module is appended to the pre-trained model, which
concatenates two subsequent frames and applies a linear transform followed by
a non-linear activation which eliminates the frame mismatch problem and the
memory and computation cost is also reduced. The student model architecture is
the small version of the Conformer transducer model (Cfm-S) [46] with 16
encoder layers of a hidden dimension of 144. To ensure the teacher model and
student model have the same encoder feature dimension, an extra linear
projection layer is added to the end of the encoder stack of the student
model. All teacher models have a single-layer 640-d long short-term memory
(LSTM) predictor network, and all student models have a single-layer 320-d
LSTM predictor network. The streaming student model has zero future context. A
triangular attention mask and causal convolution [47] are used to ensure the
model only attends to previous frames. The details of the teacher and student
model and WERs of three teacher models are shown in Tables I and II. WavLM
transducer achieves the lowest WERs among the three teachers because of the
larger pretraining set. Note that the teacher models are about 10 times the
size of the student model. All models are trained using the ESPnet [48] ASR
toolkit.

Teacher Transducer Student Encoder Wav2vec 2.0
111https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt
HuBERT222https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt
WavLM333https://github.com/microsoft/unilm/tree/master/wavlm Cfm-S Encoder
size 768-d 768-d 768-d 144-d Pretraining data LS-960 LS-960 MIX-94k None Num.
params 99.2M 99.5M 99.2M 10.3M

TABLE I: Details of the teacher models. “LS-960” and “MIX-94k” denote the full LibriSpeech dataset and 94k mix dataset. Teacher Transducer | dev | test
---|---|---
clean | other | clean | other
Wav2vec 2.0 (W2v2) | 5.1 | 12.2 | 5.2 | 11.8
Hubert (Hu) | 5.2 | 11.0 | 5.3 | 11.2
WavLM (WL) | 3.9 | 8.4 | 3.9 | 8.3
TABLE II: WERs of teacher models fine-tuned on LibriSpeech “train-clean-100”.

###  V-B Dataset

The two popular datasets LibriSpeech [49] and Libri-Light [50] were used for
the experiments. The full LibriSpeech training set contains 960 hours of
audiobook recordings with corresponding transcriptions and the LibriLight
dataset contains 60+k hours of unlabelled speech. To verify the effectiveness
of the proposed multi-teacher KD framework, experiments are first carried out
on the “train-clean-100” subset. All teacher models are also fine-tuned on the
“train-clean-100” subset using the raw waveform. During KD training, extra
unlabelled speech is used to augment the training set for potential
performance improvement. To investigate the effectiveness of this, the teacher
model’s embeddings of the remaining 860h of LibriSpeech and 1000 hours of
speech randomly sampled from Libri-Light were extracted as teacher labels,
resulting in a total pre-training dataset of around 2k hours. Pseudo
transcriptions were also used to improve student models’ performance. The
pseudo transcriptions of 860 hours of remaining speech from LibriSpeech were
obtained by performing beam search decoding of size 8 on individual teacher
model. The transcriptions were not improved by language model fusion.

During training, SpecAugment [51] was applied for data augmentation. Speed
perturbation was not used in order to reduce the number of teacher labels that
need to be stored. The output vocabulary has 256 sub-word units generated by
SentencePiece [52]. During inference, the model with the lowest WER on the
dev-other development set was chosen and the WERs on two test sets (test-clean
and test-other) are reported.

###  V-C KD Training

In the encoder pre-training stage, the raw waveform is fed to the fine-tuned
teacher models to generate encoder embeddings. The masking in the teacher
models was disabled and the embeddings were stored on disk prior to training.
Various teacher model selection strategies were explored including uniform
sampling and WER-related or similarity-related sampling. It is found that
uniform sampling yields the best performance after fine-tuning and this
sampling strategy is adopted in the rest of this paper. During training, the
student model was given the 82-d filter bank features concatenated with 1-d
pitch features corresponding to the waveform that is used to generate
embeddings. SpecAugment is applied during student encoder pre-training to
improve the robustness of features learned by the student encoder and Noam
[53] optimiser with warmup was used to update student encoder parameters. To
evaluate the student model’s encoder, the teacher embeddings of the “dev-
other” development set were also collected. The model with the lowest $L_{1}$
error on the “dev-other” was selected as the initialisation for the second
stage of training. When multiple teachers are used during KD, the averaged
$L_{1}$ error over all teachers embeddings on “dev-other” was used as the
model selection criterion.

During the second stage of KD training, the model was fine-tuned with audio-
text pairs. The texts are either ground truth transcriptions (for “train-
clean-100”) or pseudo transcriptions decoded on unlabelled speech (the
remaining 860-hour LibriSpeech). A tri-stage fine-tuning learning rate
schedule was adopted, where the warm-up stage, constant stage and decay stage
takes 10%, 40% and 50% of the total training steps. The learning rate schedule
for 100-hour and 960-hour experiments are shown in Table III.

Fine-tuning data | Initial | Warmup | Decay
---|---|---|---
100 hour | 1e-6 | 1e-4 | 5e-6
960 hour | 1e-6 | 5e-4 | 1e-5
TABLE III: The learning rate schedule of fine-tuning in 100-hour and 960-hour
experiments.

##  VI Experimental Results

###  VI-A Single Teacher vs Multi teacher

Teacher model KD dev test Embedding nBest clean other clean other Non-
streaming baseline, no KD - - - 7.5 20.9 8.0 21.8 Single teacher KD W2v2 ✓ 6.8
20.9 7.2 21.7 W2v2 ✓ ✓ 6.6 20.7 6.8 20.9 Hu ✓ 6.7 20.5 7.0 21.2 Hu ✓ ✓ 6.6
20.3 6.8 20.7 WL ✓ 6.5 20.2 6.7 20.7 WL ✓ ✓ 6.5 20.1 6.6 20.6 Multi teacher KD
W2v2+Hu ✓ 6.6 19.0 6.9 19.4 W2v2+Hu ✓ 6.4 19.6 6.6 19.8 W2v2+Hu ✓ ✓ 6.2 19.2
6.5 19.4 WL+Hu ✓ 6.6 19.0 6.8 19.3 WL+Hu ✓ 6.2 19.2 6.6 19.5 WL+Hu ✓ ✓ 6.1
18.9 6.5 19.2

TABLE IV: %WERs for non-streaming student transducer models. A tick under
embedding KD or n-best KD means that embedding-level encoder pretraininng or
1-best (or n-best if multiple teachers are used) is adopted. “W2v2”, “Hu” and
“WL” stand for Wav2vec 2.0, HuBERT and WavLM respectively.

Experiments were first carried out on the ”train-clean-100” subset. Results on
non-streaming transducers are reported in Table IV. The following observations
can be made. First, using encoder embeddings from self-supervised pre-trained
models for KD successfully improves the performance of the student model.
Second, the performance of the student model after embedding-level KD training
is better when the teacher model has lower WERs. The best performing student
model under single teacher setup is pre-trained using embeddings extracted
from WavLM, which is also the best among the three teacher models, achieving
16.2% and 5.0% relative WERR on the test sets compared to the baseline model.
Third, increasing the number of teachers leads to a further WER reduction. By
using a teacher combination of HuBERT and WavLM during encoder pre-training,
further relative WERRs of 1.5% and 4.6% is achieved compared to the best
single-teacher setup. Fourth, embedding level KD is compatible with n-best
alignment-based KD for both the single-teacher and multi-teacher setups.
Larger WER reductions are observed after adding n-best alignment KD loss to
the HuBERT+WavLM setup, leading to final relative WERRs of 18.8% and 11.9%
compared to the baseline. Note that the performance improvement on test-clean
is bigger than on the more challenging test-other set. This could be the
reason that the student model is only pre-trained on the train-clean-100
subset, which is more acoustically similar to test-clean.

Experiments are then carried out for streaming student transducer models.
Results in the 100-hour setup are shown in Table V. To verify the
effectiveness of $\tau$, student models trained with different $\tau$ values
were compared under the single-teacher setup. It can be seen that $\tau=7$
consistently outperforms $\tau=0$ (which is equivalent to allowing no delay in
the streaming student model), suggesting the necessity of choosing a sensible
$\tau$ value during embedding level KD for streaming student model. The model
trained with $\tau=0$ even performs worse than the baseline model trained
without distillation. Therefore, $\tau$ is set to 7 in the following
experiments. When combined with 1-best KD in the second stage, the same
$\tau=7$ used during encoder pre-training was adopted when using Eqn. (5) for
one-best loss computation. Similar trends as in Table IV can be observed for
streaming student models. Using embedding-level KD consistently improves the
student model under the single-teacher setup and a further gain is observed
when two teachers are used for encoder pre-training or adding n-best alignment
KD during fine-tuning. The best streaming student model is obtained under the
two-teacher embedding KD setup with HuBERT and WavLM combined with 2-best KD
during self-supervised fine-tuning, resulting in relative WERRs of 19.7% and
7.8% compared to the baseline model trained without KD.

Teacher model KD dev test Embedding nBest clean other clean other Streaming
baseline, no KD - - - 11.2 28.4 12.2 29.6 Single teacher KD W2v2, $\tau=0$ ✓
10.4 28.8 11.1 30.0 W2v2 ✓ 9.9 27.4 10.2 28.5 W2v2 ✓ ✓ 9.9 27.4 10.1 28.4 Hu,
$\tau=0$ ✓ 10.4 28.6 11.0 29.9 Hu ✓ 9.9 27.4 10.2 28.5 Hu ✓ ✓ 9.9 27.1 10.1
28.3 WL, $\tau=0$ ✓ 10.3 28.3 10.8 29.4 WL ✓ 9.8 27.2 10.1 28.2 WL ✓ ✓ 9.7
26.9 10.1 28.0 Multi teacher KD W2v2+Hu ✓ 9.7 26.9 10.0 27.9 W2v2+Hu ✓ ✓ 9.6
26.8 9.9 27.7 WL+Hu ✓ 9.6 26.6 9.9 27.6 WL+Hu ✓ ✓ 9.5 26.4 9.8 27.3

TABLE V: %WERs for streaming student transducer models. $\tau$ is the time-
shift factor used during embedding KD and is set to 7 unless explicitly
specified.

###  VI-B Effect of Unlabelled Data

The effect of incorporating extra unlabelled data during encoder pretraining
was investigated here. Experiments were first scaled up to include the
remaining 860h audio from LibriSpeech as unlabelled data. The 960h audio
features are fed to the teacher models for teacher embedding generation. The
extracted features are utilised during encoder pre-training. Erroneous pseudo
transcriptions obtained by decoding the teacher models on the 860 hour audio
were used alongside the 100 hour ground truth transcriptions. This is expected
to further improve the performance of the student model as reported in [54,
55]. The WERs of different teacher models decoded on the 860h data are shown
in Table VI.

Teacher model | clean-360 | other-500
---|---|---
W2v2 | 4.9 | 6.4
Hu | 5.0 | 6.2
WL | 4.0 | 5.2
TABLE VI: %WERs of teacher models on train-clean-360 and train-other-500
subset.

The WERs of non-streaming student models trained on 960h data are shown in
Table VII. The WERs of models trained using only pseudo transcriptions
(alongside with 100h ground truth) without KD are also listed for comparison
(see section “No KD” in Table VII). The following points could be drawn.
First, the student model’s performance is significantly improved by using only
the pseudo transcriptions. The better the pseudo transcriptions are, the lower
WERs the student model achieves. Second, the student model consistently
benefits from single teacher embedding-level KD, resulting in an averaged
relative WERR of 9.6% and 10.9% on three teacher models. Third, the
performance of the student model is further improved under the multi-teacher
encoder pre-training setup, achieving further relative WERR of 4.7% and 2.2%
with WavLM and HuBERT compared to the best single-teacher setup. Finally,
slight performance improvement is achieved when n-best alignment KD is also
used under a multi-teacher setup.

Teacher model KD dev test Embedding nBest clean other clean other No KD
Baseline - - 7.5 20.9 8.0 21.8 W2v2 trans - - 5.3 11.3 5.4 11.6 Hu trans - -
5.1 10.6 5.3 10.8 WL trans - - 4.4 10.2 4.6 10.2 Single teacher KD W2v2 ✓ 4.7
9.8 4.7 9.9 Hu ✓ 4.7 9.6 4.8 9.8 WL ✓ 4.2 9.2 4.3 9.3 Multi teacher KD W2v2+Hu
✓ 4.5 9.4 4.5 9.6 W2v2+Hu ✓ ✓ 4.4 9.2 4.4 9.5 WL+Hu ✓ 3.9 9.0 4.1 9.1 WL+Hu ✓
✓ 3.9 8.9 4.0 9.0

TABLE VII: %WERs for non-streaming student transducer models trained on 960h
data. “trans” denotes the 860 hour pseudo transcription obtained from teacher
model using beam search decoding.

Teacher model KD dev test Embedding nBest clean other clean other No KD
Baseline - - 11.2 28.4 12.2 29.6 W2v2 trans - - 9.5 18.9 10.6 19.4 Hu trans -
- 9.3 17.5 10.1 17.8 WL trans - - 8.4 15.9 9.5 16.2 Single teacher KD W2v2 ✓
7.6 16.1 8.1 17.0 Hu ✓ 7.3 15.5 7.3 15.7 WL ✓ 6.8 15.3 7.2 15.3 Multi teacher
KD W2v2+Hu ✓ 7.1 15.3 7.1 15.6 W2v2+Hu ✓ ✓ 7.0 15.1 7.1 15.4 WL+Hu ✓ 6.7 15.0
7.0 15.2 WL+Hu ✓ ✓ 6.6 14.9 6.8 14.9

TABLE VIII: %WERs for streaming student transducer models trained on 960h
data. “trans” denotes the pseudo transcription obtained from teacher model
using beam search decoding. $\tau$ is set to 7 in all KD experiments.

The WERs of streaming models are shown in Table VIII. Here, $\tau$ is fixed to
7 as it produces the best results in [35]. Again, the models trained using
only pseudo transcriptions from the teacher models are shown in the first
section in Table VIII for comparison. The same trend of performance
improvement can be observed for streaming models when multi-teacher KD was
applied. When using HuBERT and WavLM jointly during encoder pre-training, a
WERR of 40.1% and 48.6% was achieved compared to the baseline student model
trained without KD.

1009602k$12$$13$$14$$15$$16$$17$$18$$19$$20$$21$$22$$23$Encoder pre-training
data (hours)WER (%)HuWLHu+WL (a) WERs of student models.

1009602k$0$$5$$10$$15$$20$$25$$30$$35$$40$Encoder pre-training data
(hours)relative WERR (%)HuWLHu+WL (b) Relative WERRs of student models.

1009602k$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0.35$$0.4$$0.45$Encoder
pre-training data (hours)averaged $L_{1}$ errorHuWLHu+WL (c) $L_{1}$ error
after pre-training.

Figure 3: %WERs and relative WERRs of student models pre-trained with different amount of acoustic data. All models were fine-tuned with the 100 hour ground truth data. Teacher model | dev | test
---|---|---
clean | other | clean | other
100h fine-tune
Hu | 7.3 | 15.5 | 7.3 | 15.7
WL | 6.8 | 15.3 | 7.2 | 15.3
WL+Hu | 6.8 | 15.0 | 7.0 | 15.2
960h fine-tune
Hu | 4.6 | 9.4 | 4.6 | 9.7
WL | 4.1 | 9.2 | 4.2 | 9.2
WL+Hu | 3.8 | 8.8 | 3.9 | 9.0
TABLE IX: %WERs for non-streaming student transducer models pre-trained on 2k
hours. Only embedding KD is adopted.

To investigate if extra unlabelled data from another domain helps embedding-
level KD, the extra 1000 hours of unlabelled data from Libri-Light was
incorporated during encoder KD training for non-streaming student models.
Fine-tuning with train-clean-100 ground truth data and 960 hours mix data are
carried out and the WERs are shown in Table IX. Compared to the result in
Table VII, the WERs were further improved after increasing the pre-training
dataset from 960 to 2k hours.

###  VI-C Pretraining Error and Finetuning Accuracy

The previous experiments demonstrated that the student model achieves better
performance with the increase of pre-training data, indicating that a better
encoder initialisation is learned in the first stage. Apart from the WER after
fine-tuning, the KD training in the first stage can also be evaluated with the
$L_{1}$ error on test sets. Here, the relationship between the $L_{1}$ error
after pre-training and the final WER is investigated. The fine-tuning results
of the student models pre-trained with 100 hours, 960 hours and 2k hours data
are shown in Fig. 3(a) and Fig. 3(b). Note that only train-clean-100 is used
during fine-tuning.

The normalised $L_{1}$ error on the development set of student models pre-
trained with different amounts of unlabelled data is illustrated in Fig. 3(c).
The $L_{1}$ error decreases with extra unlabelled data used during encoder
pre-training for both the single-teacher and multi-teacher setups. This is in-
line with the improved student model performance after fine-tuning. However, a
lower $L_{1}$ error does not guarantee a better student model after fine-
tuning when comparing single-teacher and multi-teacher setups (see Fig. 3(b)).
The student model trained with two teachers (HuBERT and WavLM) has a much
higher $L_{1}$ error than the single-teacher setup, but yields lower WERs
after fine-tuning. This could be the reason that the random teacher model
selection strategy drives the encoder initialisation to a manifold that is in
between the teacher models and can be quickly adapted to the fine-tuning data.
This also effectively prevents the student model from overfitting to a single
teacher, which is beneficial for multi-teacher training.

##  VII Conclusions

In this paper, a two-stage teacher-student framework has been proposed, where
a student neural transducer ASR model distils the knowledge either from one or
multiple complementary SSL pre-trained speech foundation models. In the first
stage, the student ASR encoder is trained to approximate the embeddings
generated by one or multiple teacher encoders without using the ground truth
labels. In the second stage, the entire student model is fine-tuned with
paired audio-text data, where the paired texts can be generated either by
human annotation or by existing teacher ASR models. On LibriSpeech 100h, the
averaged WER of a non-streaming student model trained with a single teacher is
11% relative lower than that trained from scratch and using the multi-teacher
setup further increases the relative WER reduction to 14%. The proposed KD
framework is also effective for streaming neural transducers using an
additional time-delay factor, which is to resolve the emission mismatch
between the non-streaming teacher and the streaming student. The proposed KD
framework is also complementary to existing KD methods, leading to further
performance improvement in combination. Further WER reductions can be achieved
when scaling up the amount of unlabelled data used in the first stage. The
best-performing student is obtained under a multi-teacher setup with extra
unlabelled data, resulting relative WERR of 55%.

## References

  * [1] A. Graves, “Sequence transduction with recurrent neural networks,” in _Proc. ICML Workshop on Representation Learning_ , Edinburgh, 2012.
  * [2] A. M. A. Graves and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _Proc. ICASSP_ , Vancouver, 2013.
  * [3] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in _Proc. ICASSP_ , Shanghai, 2016.
  * [4] K. C. L. Lu, X. Zhang and S. Renals, “A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition,” in _Proc. Interspeech_ , Dresden, 2015.
  * [5] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell,” in _Proc. ICASSP_ , Shanghai, 2016.
  * [6] T. H. S. Kim and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in _Proc. ICASSP_ , New Orleans, 2017\.
  * [7] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _Proc. NIPS Deep Learning Workshop_ , Montreal, 2014.
  * [8] R. Takashima, S. Li, and H. Kawai, “An investigation of a knowledge distillation method for CTC acoustic models,” in _Proc. ICASSP_ , Calgary, 2018.
  * [9] R. V. Swaminathan, B. King, G. P. Strimel, J. Droppo, and A. Mouchtaris, “CoDERT: Distilling encoder representations with co-learning for transducer-based speech recognition,” in _Proc. Interspeech_ , Brno, 2021\.
  * [10] K. Hwang and W. Sung, “Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1,” in _Proc. SiPS_ , Belfast, 2014.
  * [11] M. Courbariaux, Y. Bengio, and J.-P. David, “Binaryconnect: Training deep neural networks with binary weights during propagations,” in _Proc. NIPS_ , Montreal, 2015.
  * [12] G. Menghani and S. Ravi, “Learning from a teacher using unlabeled data,” _arXiv preprint arXiv:1911.05275_ , 2019.
  * [13] J. Li, M. L. Seltzer, X. Wang, R. Zhao, and Y. Gong, “Large-scale domain adaptation via teacher-student learning,” in _Proc. Interspeech_ , Stockholm, 2017.
  * [14] Z.-H. Zhou, “Ensemble learning,” in _Machine learning_. Springer, 2021, pp. 181–210.
  * [15] J. Lindqvist, A. Olmin, Lindsten F., and Svensson L., “A general framework for ensemble distribution distillation,” in _Proc. MLSP_ , Espoo, 2020.
  * [16] J. Wong and M. Gales, “Student-teacher training with diverse decision tree ensembles,” in _Proc. Interspeech_ , Stockholm, 2017.
  * [17] J. H. M. Wong and M. J. F. Gales, “Sequence student-teacher training of deep neural networks,” in _Proc. Interspeech_ , San Francisco, 2016.
  * [18] T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran, “Efficient knowledge distillation from an ensemble of teachers,” in _Proc. Interspeech_ , Shanghai, 2017.
  * [19] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [20] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,” _Technologies_ , vol. 9, no. 1, 2021\.
  * [21] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _NeurIPS_ , Vancouver, 2020.
  * [22] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai _et al._ , “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, Language Processing_ , pp. 3451–3460, 2021.
  * [23] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, 2022.
  * [24] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, “Big self-supervised models are strong semi-supervised learners,” in _Proc. NeurIPS_ , Vancouver, 2020.
  * [25] C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech representation learning with labeled and unlabeled data,” in _Proc. ICML_ , Baltimore, 2021.
  * [26] F. L. Kreyssig, Y. Shi, J. Guo, L. Sari, A. Mohamed, and P. C. Woodland, “Biased self-supervised learning for ASR,” in _arXiv preprint arXiv:2211.02536_ , 2022.
  * [27] X. Zheng, C. Zhang, and P. C. Woodland, “Tandem multitask training of speaker diarisation and speech recognition for meeting transcription,” in _Proc. Interspeech_ , Incheon, 2022.
  * [28] L. Pepino, P. Riera, and L. Ferrer, “Emotion recognition from speech using wav2vec 2.0 embeddings,” in _Proc. Interspeech_ , Brno, 2021.
  * [29] G. Shen, R. Lai, R. Chen, Y. Zhang, K. Zhang, Q. Han, and H. Song, “WISE: Word-level interaction-based multimodal fusion for speech emotion recognition,” in _Proc. Interspeech_ , Shanghai, 2020.
  * [30] Z. Peng, A. Budhkar, I. Tuil, J. Levy, P. Sobhani, R. Cohen, and J. Nassour, “Shrinking bigfoot: Reducing wav2vec 2.0 footprint,” _arXiv preprint arXiv:2103.15760_ , 2021.
  * [31] H.-J. Chang, S.-w. Yang, and H.-y. Lee, “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert,” in _Proc. ICASSP_ , Toronto, 2022.
  * [32] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proc. ICML_ , Pittsburgh, 2006.
  * [33] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid CTC/attention architecture for end-to-end speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, pp. 1240–1253, 2017\.
  * [34] S. Panchapagesan, D. S. Park, C.-C. Chiu, Y. Shangguan, Q. Liang, and A. Gruenstein, “Efficient knowledge distillation for RNN-transducer models,” in _Proc. ICASSP_ , Toronto, 2021.
  * [35] X. Yang, Q. Li, and P. C. Woodland, “Knowledge distillation for neural transducers from large self-supervised pre-trained models,” in _Proc. ICASSP_ , Singapore, 2022.
  * [36] L. Guo, X. Yang, Q. Wang, Y. Kong, Z. Yao, F. Cui, F. Kuang, W. Kang, L. Lin, M. Luo _et al._ , “Predicting multi-codebook vector quantization indexes for knowledge distillation,” _arXiv preprint arXiv:2211.00508_ , 2022.
  * [37] G. Kurata and G. Saon, “Knowledge distillation from offline to streaming RNN transducer for end-to-end speech recognition,” in _Proc. Interspeech_ , Shanghai, 2020.
  * [38] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in _Proc. ICLR_ , Toulon, 2017.
  * [39] T. G. Dietterich, “Ensemble methods in machine learning,” in _proc. MCS_ , Cagliari, 2000.
  * [40] L. Rokach, “Ensemble-based classifiers,” _Artificial Intelligence Review_ , vol. 33, p. 1–39, 2010.
  * [41] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Proc. NIPS_ , Long Beach, 2017.
  * [42] Y. Liu, W. Zhang, and J. Wang, “Adaptive multi-teacher multi-level knowledge distillation,” _Neurocomputing_ , vol. 415, pp. 106–113, 2020.
  * [43] R. Sahraeian and D. Van Compernolle, “Cross-entropy training of DNN ensemble acoustic models for low-resource ASR,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, pp. 1991–2001, 2018.
  * [44] B. B. Sau and V. N. Balasubramanian, “Deep model compression: Distilling knowledge from noisy teachers,” _arXiv preprint arXiv:1610.09650_ , 2016\.
  * [45] H. Zhao, X. Sun, J. Dong, C. Chen, and Z. Dong, “Highlight every step: Knowledge distillation via collaborative teaching,” _IEEE Transactions on Cybernetics_ , vol. 52, pp. 2070–2081, 2020.
  * [46] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” in _Proc. Interspeech_ , Shanghai, 2020.
  * [47] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals _et al._ , “WaveNet: A generative model for raw audio,” in _Proc. SSW_ , Sunnyvale, 2016.
  * [48] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba _et al._ , “ESPnet: End-to-end speech processing toolkit,” in _Proc. Interspeech_ , Hyderabad, 2018.
  * [49] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in _Proc. ICASSP_ , Brisbane, 2015.
  * [50] J. Kahn, M. Rivière, W. Zheng, Kharitonov _et al._ , “Libri-light: A benchmark for ASR with limited or no supervision,” in _Proc. ICASSP_ , Barcelona, 2020.
  * [51] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph _et al._ , “SpecAugment: A simple data augmentation method for automatic speech recognition,” in _Proc. Interspeech_ , Graz, 2019.
  * [52] T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in _Proc. EMNLP_ , Brussels, 2018.
  * [53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. NIPS_ , Long Beach, 2017.
  * [54] Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, “Iterative pseudo-labeling for speech recognition,” in _Proc. Interspeech_ , Shanghai, 2020.
  * [55] Y. Higuchi, N. Moritz, J. L. Roux, and T. Hori, “Momentum pseudo-labeling for semi-supervised speech recognition,” in _Proc. Interspeech_ , Brno, 2021\.