Dima Rekesh1, Samuel Kriman1, Somshubra Majumdar1, Vahid Noroozi1, He Huang1,
---
Oleksii Hrinchuk1, Ankur Kumar2, Boris Ginsburg1

# Fast Conformer with Linearly Scalable Attention for Efficient Speech
Recognition

###### Abstract

Conformer-based models have become the most dominant end-to-end architecture
for speech processing tasks. In this work, we propose a carefully redesigned
Conformer with a new downsampling schema. The proposed model, named Fast
Conformer, is 2.8x faster than original Conformer, while preserving state-of-
the-art accuracy on Automatic Speech Recognition benchmarks. Also we replace
the original Conformer global attention with limited context attention post-
training to enable transcription of an hour-long audio. We further improve
long-form speech transcription by adding a global token. Fast Conformer
combined with a Transformer decoder also outperforms the original Conformer in
accuracy and in speed for Speech Translation and Spoken Language
Understanding. The models and training recipes have been released in the NeMo
toolkit [1].111https://github.com/NVIDIA/NeMo

Index Terms: speech recognition, speech translation, spoken language
understanding

##  1 Introduction

Conformer is a Transducer (RNNT) model for automatic speech recognition (ASR)
proposed by Gulati et al [2]. Conformer models obtain state-of-the-art results
on multiple speech benchmarks[3] thanks to their encoder architecture which
combines depth-wise convolutional layer for local features and self-attention
layer for global context. Conformers have also been rapidly adopted in
industry, especially for streaming ASR on-device and in the cloud. 222 F.
Beaufays, “Google Cloud launches new models for more accurate Speech AI”,
2022. C. Barnes, “Run Google Cloud Speech AI lo- cally, no internet connection
required,” , 2022 At the same time, Conformer models use more compute and
memory than convolution-only ASR models, e.g. Quartznet [4], because self-
attention layers have quadratic time and memory complexity vs. input sequence
length. The quadratic complexity imposes a severe limitation on the maximum
audio length which can be processed by Conformer. We redesign Conformer to
address these challenges. Namely:

  1. 1.

We redesign the downsampling schema and sub-sampling block to increase
downsampling to 8x (see Fig.1)

  2. 2.

We optionally replace original self-attention layers with a combination of
local attention and global context token post-training, similarly to
LongFormer [5].

The proposed encoder has 2.9x fewer multiply-add operations, while scaling
linearly with the sequence length. It is $2.8\times$ faster during inference.
We hereby refer to this model as Fast Conformer. At the same time, Fast
Conformer maintains highly competitive word error rates (WER) on ASR
benchmarks.

We experiment with a Longformer-based attention mechanism, and find that by
using limited context attention in combination with a single global attention
token we are able to achieve good results on long-form ASR, while being more
than $3\times$ faster during inference. We then test Fast Conformer model on
two additional speech processing tasks: speech translation (ST) and speech
language understanding (SLU). On ST, with transformer and transduder decoders,
we obtain strong scores for En-De translation with significant speedup
compared to Conformer. On SLU we obtain state-of-the-art results on the Speech
Intent Classification and Slot Filling task, and discuss the relatively modest
speedup compared to the Conformer baseline.

##  2 Fast Conformer architecture

###  2.1 Downsampling schema

Figure 1: The downsampling schemas for Conformer, SqueezeFormer, Efficient
Conformer and Fast Conformer. Fast Conformer increases sampling rate from 10ms
to 80m msec using 3 depth-wise convolutional sub-sampling layers. The
additional 2x reduction in the encoder output length versus Conformer yields
further compute-memory savings in RNNT decoder.

Conformer encoder is comprised of a stack of alternating multi-head attention
[6], depth-wise separable convolutional [7] and fully connected layers with
residual connections. The encoder starts with a sub-sampling module, which
increases the frame rate from 10 ms to 40 ms. The decrease in the sequence
length by 4x helps to reduce the computational and memory cost of the
attention layers in all following blocks. This subsampling module is
relatively expensive, accounting for roughly 20 % of the computation time of
each forward pass of the model for the "Large" Conformer (120 M parameters)
[8].

A straight-forward way to accelerate Conformer is to increase down-sampling
rate from 4x to 8x. For example, EfficientConformer [9] reduces the sequence
length of the speech features by 8x using progressive down-sampling: first, 2x
down-sampling in the first layer, then another 2x in the middle of encoder,
and final 2x in the last encoder layer. One of the drawbacks of progressive
subsampling is a computation imbalance between different attention layers. The
initial attention layers work on much longer sequences, so they have 16x more
computational cost as compared to final attention layers that operate on much
shorter sequences (see Fig. 1). Squeezeformer [8] combines progressive
downsampling with Temporal U-Net structure. Squeezeformer add extra
downsampling at the middle of encoder, and an upsampling layer at the end of
encoder to recover 4x time resolution (see Fig. 1). A similar strategy was
used in Uconv-Conformer [10].

One of the reasons why previous works limit final encoder downsampling by 4x
is related to the usage of the Conformer encoder with a CTC loss [11]. CTC
requires that the input to the loss function must be longer than the target
sequence length.333Note that this constraint does not apply to the RNNT loss
[12], and we are free to use any tokenization scheme as necessary. However, as
the RNNT decoder is autoregressive, it is far more efficient at reducing the
required number of calls to the Transducer Decoder and Joint by using large
sub-word vocabularies. Encoder output may become too short after 8x
downsampling if the model use character tokenization. For example, we found
that most of training samples in Librispeech [13] will not satisfy CTC
condition after 8X subsampling if we use character tokenization. To enable 8x
downsampling for Conformer-CTC, we switch from character tokenization to
Sentencepiece Byte Pair Encoding (BPE) [14] with vocabulary sizes ranging from
128 to 1024 tokens.

To accelerate Conformer, we made the following changes in the original design:

  1. 1.

8x downsampling at the start of the encoder, thereby reducing the compute cost
of subsequent attention layers by 4x

  2. 2.

Replacement of the original convolution sub-sampling layers with depthwise
separable convolutions [15]

  3. 3.

Reduction of the number of convolutional filters in the downsampling block to
256

  4. 4.

Reduction of the convolutional kernel size to 9.

A detailed comparison of Fast Conformer downsampling with previous works is
described in Table 1.

Table 1: Down-sampling schemas and subsampling layer type for Conformer,
EfficientConformer, Squeezeformer, and Fast Conformer. K is kernel size in
convolutional filters.

Model | Subsampling schema | Type | K
---|---|---|---
Conformer[3] | 2/4 | 2D Conv | 31
Squeezeformer[8] | progressive 2/4/8/4 | Depth-wise sep | 31
Eff. Conformer[9] | progressive 2/4/8 | Depth-wise sep | 15
Fast Conformer | 2/4/8 | Depth-wise sep | 9

To determine the contribution of each change to the model accuracy, we took
Conformer-RNNT as a baseline and gradually applied each design change. First,
we added another 2x convolutional subsampling layer. Next, we used depthwise-
separable convolutions in the second and third subsampling layers. Then we
reduced the number of channels in the subsampling layers from 512 to 256.
Finally, we reduced the convolutional kernel size in the conformer blocks from
31 to 9. Encoder inference speed was measured with a batch size of 128 on
A100/80G GPU using 20s speech samples. Results are shown in Table 2. The
encoder speed increased 2.8x, while maintaining model accuracy.

Table 2: Accuracy vs speed for each component of Fast Conformer downsampling
schema. Models were tested on LibriSpeech test-other incrementally for each
modification starting from the original Conformer. Encoder inference speed
(samples/sec) was measured with batch size 128 on 20 sec audio samples. The
number of parameters (M) is shown for the encoder only.

Encoder | WER, $\%$ | Inference, | Params, | GMACS
---|---|---|---|---
test-other | samples/s | M |
Baseline Conformer | 5.19 | 169 | 115 | 143.2
+8X Stride | 5.11 | 303 | 115 | 92.5
+Depthwise conv | 5.12 | 344 | 111 | 53.2
+256 channels | 5.09 | 397 | 109 | 48.8
+Kernel 9 | 4.99 | 467 | 109 | 48.7

###  2.2 Long-form audio transcription

While standard multi-head attention layers have been successful in processing
short utterances, their scalability to long sequences has been limited due to
the quadratic scaling of the self-attention operation with sequence length.
For example, Conformer can process at once maximum 10 minutes audio on a
single A100 GPU. To address this challenge, several alternative approaches
have been explored. A common approach is buffered transcription, where the
audio sequence is divided into shorter chunks, which are then transcribed
separately before being merged to form a complete transcription. Efficient-
Conformer used grouped attention to reduce the cost of early attention layers
on long sequences by grouping neighboring time elements of the sequence along
the feature dimension before applying scaled dot-product attention.

We decided to use an alternative approach inspired by Longformer [5] \- local
attention augmented with global tokens. We use a single global attention
token, which attends to all other tokens and has all other tokens attend to
it, using a separate set of query, key and value linear projections, while
others attend in a fixed-size window surrounding each token (see Fig. 2). By
switching to limited context attention, we extend the maximum duration that
the model can process at once on a single A100 GPU by 7x: from 10 minutes for
Conformer to 70 minutes for Fast Conformer (see Table 3). In order to
efficiently compute this attention, we utilize the overlapping chunks approach
introduced in Longformer.

Figure 2: Fast Conformer combines local attention with a global context token to process long speech samples. The figure is adapted from Longformer [5] Table 3: Maximum audio duration (min), which can be transcribed by model with batch size 1, A100 GPU. Model | Max duration, min
---|---
Conformer | 10
Fast Conformer | 18
Fast Conf + Limited Context | 70

##  3 Experiments

###  3.1 Automatic Speech Recognition

The Fast Conformer model was evaluated on following English ASR benchmarks:
LibriSpeech (LS)[13], English part of Multilingual LibriSpeech (MLS) [16].,
Mozilla Common Voice [17], and Wall Street Journal (WSJ) [18]. We used Fast
Conformer-RNNT and Fast Conformer-CTC along with baseline Conformer models in
Large configuration.

####  3.1.1 LibriSpeech

First, we trained all models on the LibriSpeech dataset only. We used the
SentencePiece unigram tokenizer with 128 tokens for CTC, and 1024 tokens for
RNNT models. Training of baseline Conformer-RNNT and -CTC models, was done
with AdamW optimizer with the Noam learning rate scheduler and peak learning
rates of 0.0025 and 0.001 respectively. We set a linear warmup schedule for
15k steps in all experiments. Fast Conformer models were trained the cosine
scheduler with peak learning rates of 0.005 and 0.001 respectively. Both
Conformer and Fast Conformer-RNNT models were trained for 80k steps, CTC
models, for 380k steps. The models were trained on 32 GPUs with a global batch
size of 2048. Last five checkpoints were averaged [19]. We used the Deepspeed
profiler 444https://www.deepspeed.ai/tutorials/flops-profiler/ to estimate
Multiply Accumulate operations (MACS) on a single 30s audio. The results are
shown in Table 4. Fast Conformer has practically the same accuracy as
Conformer. The proposed encoder is 3x more compute efficient than original
Conformer encoder; and it is significantly faster than EffecientConformer and
SqueezeFormer.

Table 4: ASR: Fast Conformer-Large with CTC and RNNT decoders trained on Librispeech. Greedy WER (%) was measured on Librispeech test-other. The number of parameters (M) and compute (Multiply-Acc, GMAC) are shown for encoder only. Encoder | WER, % | Params, | Compute,
---|---|---|---
test-other | M | GMACS
RNNT decoder
Conformer | 5.19 | 115 | 143.2
Fast Conformer | 4.99 | 109 | 48.7
CTC decoder
Conformer | 5.74 | 121 | 149.2
Eff. Conformer[9] | 5.79 | 125 | 101.3
SqeezeFormer[8] | 6.05 | 125 | 91.0
Fast Conformer | 6.00 | 115 | 51.5
Table 5: ASR: Fast Conformer-Large with CTC and RNNT decoders trained on
English ASR set with 24K hours combined from public speech datasets. Greedy
WER (%) was measured on Librispeech test-other, MCV 8, MLS and WSJ-92 test
sets

Encoder | LS | MCV 8 | MLS | WSJ-92
---|---|---|---|---
test-other | test | En | test
RNNT decoder
Conformer | 3.70 | 7.80 | 5.80 | 1.50
Fast Conformer | 3.79 | 8.18 | 5.76 | 1.42
CTC decoder
Conformer | 4.50 | 9.40 | 6.60 | 1.70
Fast Conformer | 4.19 | 9.00 | 6.42 | 1.59

####  3.1.2 Large-25k hours ASR Set

To test Fast Conformer capacity with respect to larger dataset, we trained
Fast Conformer and Conformer models on 25k hours of speech, composed from
LibriSpeech, Mozilla Common Voice, National Singapore Corpus and other public
English speech datasets. All models were trained for 300k steps and 1M steps,
respectively. AdamW was used with cosine scheduler with a 15k linear warmup
and learning rates of 0.0025 and 0.001, for RNNT and CTC respectively. The
models have been tested on LibriSpeech test-other, MLS, MCV and WSJ-93 test
sets. The results are presented in Table 5. Fast Conformer outperformed the
original Conformer on most of benchmarks.

###  3.2 Speech Translation

Next, we analyze the efficacy of Fast Conformer on Speech Translation (ST)
from English to German. We trained two architectures with the same Conformer-
like encoder and different autoregressive decoders: either RNNT, or 6-layer
Transformer trained with cross entropy loss.

In all the experiments, we initialized encoder with the corresponding weights
from ASR RNNT models trained on 25k hours of speech. The parameters of decoder
and joint module were initialized randomly. Our vocabulary consists of $16384$
YouTokenToMe555https://github.com/VKCOM/YouTokenToMe byte-pair-encodings
trained on German text. Our models have been trained on all available datasets
from IWSLT22 [20] competition which corresponds to $4$k hours of speech. Some
of the datasets did not contain German translations, so we generated them
ourselves with text-to-text machine translation model trained on WMT21 [21]
and in-domain finetuned on Must-C v2[22].

Table 6: Speech Translation, MUST-C V2 tst-COMMON dataset. SacreBLEU, total inference time, and relative inference speed-up were measured with a batch size $32$ for two speech translation models with Conformer-based encoder and either RNNT, or Transformer decoder. Encoder | BLEU | Time, sec | Speed-up
---|---|---|---
Transformer decoder
Conformer | 31.0 | 267 | 1X
Fast Conformer | 31.4 | 161 | 1.66X
RNNT decoder
Conformer | 23.2 | 83 | 1X
Fast Conformer | 27.9 | 45 | 1.84X

The results for all models are shown in Table 6. We note that RNNT loss is
generally not suitable for speech translation due to its implicit monotonic
alignment assumption. Surprisingly, Fast Conformer-RNNT translation model gets
BLEU score of 27.89. In addition, the inference of this model is up to 1.84X
faster comparing to Conformer.

###  3.3 Spoken Language Understanding

Next, we apply the pre-trained Fast Conformer encoder to spoken language
understanding (SLU). We study the _Speech Intent Classification and Slot
Filling_ (SICSF) task, which should detect user intents and extract the
corresponding lexical fillers for detected entity slots [23]. An intent can be
a composition of a scenario type and an action type. Slots and fillers are
represented by key-value pairs. The ground-truth intents and slots of input
are organized as a Python dictionary, represented as a string. The SICSF task
is to predict this structured Python dictionary as a string, based on the
input audio. Experiments are conducted using the SLURP [23] dataset, where
intent accuracy and SLURP-F1 are used as the evaluation metric.

We use as the baseline Conformer encoder, initialized from pretrained ASR
model, with a Transformer decoder. We also compare Fast Conformer against two
state-of-the-art models from ESPNet-SLU [24] and SpeechBrain [25]. ESPNet-SLU
and SpeechBrain both use a HuBERT [26] encoder pre-trained via a self-
supervised objective on the entire LibriLight-60k [27] dataset. ESPNet-SLU
further finetunes the encoder on LibriSpeech before training on SLURP.

The results for SLURP experiments are shown in Table 7. The model with pre-
trained Fast Conformer encoder significantly surpass the ESPNet-SLU and
SpeechBrain models, which have been pre-trained on nearly 60K hours of speech.
Fast Conformer attains very high accuracy quite close to Conformer. Its
decoding is also 10(%) faster than Conformer. The speed-up is not as high as
for ASR for following reason: the ratio of acoustic signal length (after 8x
downsampling) to target token length is roughly 1:2.22 for the SICSF task. .
The execution cost for encoder is dwarfed by slow autoregressive Transformer
decoder, and therefore we used batch 32 to balance the cost of the encoder
against the decoder to show speed-up.

Table 7: Speech intent classification and slot filling on SLURP dataset.
ESPNet-SLU and SpeechBrain-SLU models use a HuBERT [26] encoder pre-trained
via a self-supervised objective on LibriLight-60k [27]. Inference time and
relative speed-up against Conformer are measured with batch size 32.

Model |  | Intent
---
Acc.
| SLURP
---
F1
| Inference,
---
sec
| Rel.
---
Speed-up
SpeechBrain-SLU | 87.70 | 76.19 | - | -
ESPnet-SLU | 86.52 | 76.91 | - | -
Conformer/Fast Conformer+Transformer Decoder
Conformer | 90.14 | 82.27 | 210 | 1X
Fast Conformer | 90.68 | 82.04 | 191 | 1.1X

###  3.4 Long-form audio transcription

Fast Conformer with limited context and global token for long audio was
trained in the following way. This model has additional query, key and value
projection layers that are used for global attention. These layers are
initialized by copying weights from the trained attention layers. The rest of
the encoder is initialized with the checkpoint pre-trained on our internal 25k
hour set with full context attention. We then do additional fine-tuning on the
same dataset with limited context attention for 10k steps. We used learning
rate warmup of 1k steps, with maximum learning rate 1e-6 and cosine rate
annealing to 0. The size of the limited context was set to 128 steps on each
side of a token, which corresponds to around 10 seconds. We also randomly
concatenate training utterances during this fine-tuning, to train the model to
handle longer sequences.

The performance of Fast Conformer was evaluated using two long-form audio
datasets: TED-LIUM 3 [28], and Earnings-21 [29]. We compare our limited
context model with full context Fast Conformer as well as with base Conformer
trained on the same dataset. Since some utterances in the evaluation datasets
are too long even for the limited context model, we use buffered inference.
For Conformer and FastConformer with full context attention we used 20 second
buffers. For FastConformer with limited context we used buffer size of one
hour. Fast Conformer with new attention mechanism significantly outperforms
Conformer and Fast Conformer with global attention on both long-form ASR
benchmarks (see Table 8).

Table 8: FastConformer versus Conformer on long audio. We evaluated four versions of FastConformer: (1) FC with global attention (2) FC with limited context (3) FC with limited context and global token (4) FC with limited context and global token, trained on long concatenated utterances. Models have been evaluated on two long-audio bencmarks: TED-LIUM v3 and Earning 21. Greedy WER(%). Model | TED-LIUM v3 | Earnings21
---|---|---
Conformer | 9.71 | 24.34
Fast Conformer | 9.85 | 23.84
\+ Limited Context | 9.92 | 28.42
\+ Global Token | 8.00 | 20.68
\+ Concat utterances | 7.85 | 19.52

##  4 Conclusions

In this paper, we propose Fast Conformer, a redesigned Conformer with a new
downsampling schema. Fast Conformer uses about 2.9x less compute while
delivering roughly the same WER as the original Conformer. Evaluations on
tasks such as speech translation and spoken language understanding describe
the strong recognition accuracy of the model while obtaining significant
speed-ups in the encoder computation. When the attention module is replaced
with local attention, we show that greater efficiency enables long-form
transcription of 1-hour audio segment in a single forward pass. The results on
long-form audio are improved further by adding a single global attention
token.

## References

  * [1] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook _et al._ , ``NeMo: a toolkit for building AI applications using neural modules,'' in _Systems for ML Workshop, NeurIPS_ , 2019.
  * [2] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented transformer for speech recognition,'' in _Interspeech_ , 2020.
  * [3] P. Guo, F. Boyer, X. Chang, T. Hayashi, Y. Higuchi, H. Inaguma, N. Kamo, C. Li, D. Garcia-Romero, J. Shi _et al._ , ``Recent developments on Espnet toolkit boosted by Conformer,'' in _ICASSP_ , 2021.
  * [4] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, ``QuartzNet: Deep automatic speech recognition with 1D time-channel separable convolutions,'' in _ICASSP_ , 2020.
  * [5] I. Beltagy, M. E. Peters, and A. Cohan, ``Longformer: The long-document Transformer,'' _arXiv:2004.05150_ , 2020.
  * [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _NeurIPS_ , 2017\.
  * [7] F. Chollet, ``Xception: Deep learning with depthwise separable convolutions,'' in _CVPR_ , 2017.
  * [8] S. Kim, A. Gholami, A. Shaw, N. Lee, K. Mangalam, J. Malik, M. W. Mahoney, and K. Keutzer, ``Squeezeformer: An efficient Transformer for automatic speech recognition,'' in _NeurIPS_ , 2022.
  * [9] M. Burchi and V. Vielzeuf, ``Efficient Conformer: Progressive downsampling and grouped attention for automatic speech recognition,'' in _Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2021.
  * [10] A. Andrusenko, R. Nasretdinov, and A. Romanenko, ``Uconv-conformer: High reduction of input sequence length for end-to-end speech recognition,'' in _ICASSP_ , 2022.
  * [11] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _ICML_ , 2006.
  * [12] A. Graves, ``Sequence transduction with recurrent neural networks,'' in _ICML_ , 2012.
  * [13] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an ASR corpus based on public domain audio books,'' in _ICASSP_ , 2015.
  * [14] T. Kudo and J. Richardson, ``SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,'' _arXiv:1808.06226_ , 2018.
  * [15] F. Chollet, ``Xception: Deep learning with depthwise separable convolutions,'' in _CVPR_ , 2017.
  * [16] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, ``MLS: A large-scale multilingual dataset for speech research,'' _Interspeech_ , 2020\.
  * [17] ``Mozilla: A journey to less than 10% word error rate,'' https://hacks.mozilla.org/2017/11/a-journey-to-10-word-error-rate/.
  * [18] D. B. Paul and J. M. Baker, ``The design for the Wall Street Journal based CSR corpus,'' in _Proc. of the workshop on Speech and Natural Language_. ACL, 1992, pp. 357––362.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _NeurIPS_ , 2017.
  * [20] A. Anastasopoulos, L. Barrault, L. Bentivogli, M. Z. Boito, O. Bojar, R. Cattoni, A. Currey, G. Dinu, K. Duh, M. Elbayad _et al._ , ``Findings of the iwslt 2022 evaluation campaign,'' in _Proc. of the 19th International Conference on Spoken Language Translation (IWSLT_ , 2022.
  * [21] S. Subramanian, O. Hrinchuk, V. Adams, and O. Kuchaiev, ``NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21,'' _arXiv:2111.08634_ , 2021.
  * [22] R. Cattoni, M. A. Di Gangi, L. Bentivogli, M. Negri, and M. Turchi, ``MuST-C: A multilingual corpus for end-to-end speech translation,'' _Computer Speech & Language_, vol. 66, p. 101155, 2021.
  * [23] E. Bastianelli, A. Vanzo, P. Swietojanski, and V. Rieser, ``SLURP: A Spoken Language Understanding Resource Package,'' in _EMNLP_ , 2020.
  * [24] S. Arora, S. Dalmia, P. Denisov, X. Chang, Y. Ueda, Y. Peng, Y. Zhang, S. Kumar, K. Ganesan, B. Yan _et al._ , ``ESPnet-SLU: Advancing spoken language understanding through ESPnet,'' in _ICASSP_ , 2022.
  * [25] Y. Wang, A. Boumadane, and A. Heba, ``A fine-tuned Wav2vec 2.0/HuBERT benchmark for speech emotion recognition, speaker verification and spoken language understanding,'' _arXiv:2111.02735_ , 2021.
  * [26] W.-N. Hsu, B. Bolte, Y.-H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [27] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen _et al._ , ``Libri-Light: A benchmark for ASR with limited or no supervision,'' in _ICASSP_ , 2020.
  * [28] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Estève, ``TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation,'' in _Speech and Computer_ , 2018.
  * [29] M. D. Rio, N. Delworth, R. Westerman, M. Huang, N. Bhandari, J. Palakapilly, Q. McNamara, J. Dong, P. Żelasko, and M. Jetté, ``Earnings-21: A practical benchmark for ASR in the wild,'' in _Interspeech_ , 2021.