# Back From The Future: Bidirectional CTC Decoding Using Future Information In
Speech Recognition

###### Abstract

In this paper, we propose a simple but effective method to decode the output
of Connectionist Temporal Classifier (CTC) model using a bidirectional neural
language model. The bidirectional language model uses the future as well as
the past information in order to predict the next output in the sequence. The
proposed method based on bidirectional beam search takes advantage of the CTC
greedy decoding output to represent the noisy future information. Experiments
on the Librispeech dataset demonstrate the superiority of our proposed method
compared to baselines using unidirectional decoding. In particular, the boost
in accuracy is most apparent at the start of a sequence which is the most
erroneous part for existing systems based on unidirectional decoding.

Index Terms—  Bi-directional language model, Exposure bias, Connectionist
temporal classification, Beam Search, Shallow Fusion

##  1 Introduction

In recent years, the performance of automatic speech recognition (ASR) systems
have seen dramatic improvements due to the application of deep learning and
the use of large-scale datasets [1, 2, 3, 4, 5, 6, 7, 8, 9]. In particular,
there have been two strands of research pushing the state-of-the-art in this
field, namely sequence-to-sequence models [10, 11] and Connectionist Temporal
Classifier (CTC) based methods [12, 13]. This work focuses on the latter due
to its simplicity and good performance.

A key challenge in implementing such method lies in the process of decoding.
Since CTC-based models do not have built-in methods for decoding, the beam
search strategy is often used to achieve satisfactory accuracy. This requires
an external language model (LM) to provide prior information on which sequence
of words are the most probable.

While both CTC and sequence-to-sequence models are able to process the whole
sequence of speech data in a bidirectional manner, the traditional beam search
method does not take advantage of the future information since the current
prediction relies on the previous output. Although there have been attempts to
take advantage of the whole sequence in the decoding process of CTC-based
methods, its effectiveness has not been demonstrated in automatic speech
recognition.

In this paper, we propose the bi-directional decoding method with noisy future
context and training method robust to noisy context. As an ASR model, we
employ the Connectionist Temporal Classification (CTC) model instead of
autoregressive model because the latter is based on the label-synchronous [14]
decoding, resulting in difficulty to know which frame of speech is being
decoded at each time step. However, frame-synchronous [15, 16] decoding makes
easier.

The majority of bi-directional LMs [17, 18, 19] are designed for N-best LM
rescoring with architectural advancement from LSTM-RNN to neural random field
and transformer encoder. Although N-best LM rescoring can replace/be combined
with our decoding method, we insist that rescoring generally underperforms
shallow fusion. The former only finds the best solution among first-pass
search results, while the latter can use LM score at every decoding time step
of the beam search.

[20] use the bidirectional shallow fusion decoding algorithm by using
forward/backward uni-directional decoders on top of bi-directional encoder
architecture. Similar to our work, they use the result of greedy decoding from
the backward decoder as the future context. However, it is known that the
greedy decoding of the autoregressive model suffers from exposure bias
problem, which degrades the quality of generation and requires expensive
supervision (e.g., scheduled sampling, REINFORCE) to fix it.

##  2 CTC greedy and prefix beam search

Assume that sequence of speech representation
$\mathbf{x}=\\{x_{1},\cdots,x_{T_{x}}\\}$, corresponding sequence of labels
$\mathbf{y}=\\{y_{1},\cdots,y_{T_{y}}\\}$, and its alignment
$\boldsymbol{\pi}=\\{\pi_{1},\cdots,\pi_{T_{x}}\\}$ is given. The alignment
sequence ($\boldsymbol{\pi}$) consists of a special CTC character called blank
label as well as output labels in the set of label dictionary and is mapped to
an unique result label sequence by removing all blanks and repeated labels.

For each time step $t$, the greedy decoding of the CTC model choose the most
probable CTC output label using the probability $P(\pi_{t}|\mathbf{x})$,
namely

$\boldsymbol{\pi}^{*}=\operatorname*{arg\,max}_{\boldsymbol{\pi}}{\prod_{t}}P(\pi_{t}|\mathbf{x})$
(1)

Greedy decoding ends with having actual label sequence by being mapped to
$\mathcal{B}$, $\mathbf{y}^{*}=B(\boldsymbol{\pi}^{*})$. Greedy decoding is
not able to get the most probable output sequence since it does assume that
the probability of the sequence is independent for every frame. However it can
be done very simply and quickly because the only thing have to do is find the
most probable output label
$\operatorname*{arg\,max}_{\pi_{t}}P(\pi_{t}|\mathbf{x})$ for each time step
$t$.

Other than greedy decoding, we can efficiently calculate the probabilities of
successive extensions of every each labelling prefix with prefix search
decoding [12]. However, in full prefix search the maximum, number of prefixes
expands grows exponentially as input sequence length gets longer. For that
reason, prefix search decoding should be done with beam search decoding, which
limits the size of extensions for each search step.

The CTC model is based on conditional independence assumption, therefore
decoding often require an external language model to utilize dependency
between outputs during decoding. Assume that we have an external language
model which models the probability $P(y_{t}|y_{t-1},\cdots y_{1})$. CTC beam
search can be done for a certain beam width $B$ by finding $B$ most probable
candidates in each time step $t$. We should find the optimal CTC label
sequence $\hat{\textbf{y}}=\mathcal{B}(\boldsymbol{\pi})$ such that

$\operatorname*{arg\,max}_{\boldsymbol{\pi}}\prod_{t}P_{\text{AM}}(\pi_{t}|\mathbf{x})P_{\text{LM}}(y_{\pi_{t}}|\mathcal{B}(\boldsymbol{\pi}_{1:t-1}))^{\alpha}$
(2)

where $P_{\text{AM}}$ is CTC probability, $P_{\text{LM}}$ is language model
probability, $y_{\pi_{t}}$ is an output label of CTC label and $\alpha$ is
hyperparameter controlling the effect of language model. For each time step
$t$, get $B$ most probable beams based on the probability (2) and proceed to
next time step with $B$ beams.

Fig. 1: Bidirectional CTC Beam Search

##  3 CTC Bidirectional Beam Search

Fig. 2: Bi-directional LM with future shift, random insertion/deletion noise

In this section, we propose bi-directional decoding method with noisy future
context for CTC. In addition, training method making bi-directional language
model robust to noisy future context is presented.

###  3.1 Bidirectional Language Model

Assume that bi-directional context including future information is given. Then
we can predict the next output token with both forward language model from the
past information and backward language model from the future information with
the following:

$\begin{split}P(y_{t}|y_{1},\cdots,y_{t-1},&y_{t+1},\cdots,y_{T_{y}})\\\
&=\overleftrightarrow{f}_{y}(y_{t}|\overleftrightarrow{h}_{t})\\\
&=\overleftrightarrow{f}_{y}(y_{t}|\overrightarrow{h}_{t}+\overleftarrow{h}_{t})\\\
\end{split}$ (3)

where
$\overrightarrow{h}_{t}=\overrightarrow{f}_{h}(y_{t-1},\overrightarrow{h}_{t-1})$
represents the forward language model and
$\overleftarrow{h}_{t}=\overleftarrow{f}_{h}(y_{t+1},\overleftarrow{h}_{t+1})$
represents the backward language model respectively Fig. 2-(a).

Bidirectional language model can be realized by Transformer as well as RNN
with considering $h_{t}$ as memory in Transformer. We have used both LSTM and
Transformer-XL to train bidirectional language model. It needs two input
sequences, forward sequence for the past text and backward sequence for the
future text. Putting into two input sequences, we get two hidden states each,
$\overrightarrow{h}_{t}$ and $\overleftarrow{h}_{t}$ from bidirectional
language model. $\overleftrightarrow{h}_{t}$ can be obtained by adding these
two hidden states and it will be used to calculate final probability vector.

###  3.2 CTC Decoding with Bidirectional Language Model

As described in Fig.1, while decoding through the time, greedy decoding result
provides noisy future context for backward language model. In (1), for greedy
decoding result $\pi^{*}$, we may take
$\mathcal{B}(\boldsymbol{\pi^{*}}_{t+1:})$ as the future sequence for a
certain time step $t$. Therefore in order to decode CTC with bidirectional
beam search, we find the optimal output sequence
$\hat{\textbf{y}}=\mathcal{B}(\boldsymbol{\hat{\pi}})$ such that

$\begin{split}\boldsymbol{\hat{\pi}}=\operatorname*{arg\,max}_{\boldsymbol{\pi}}\prod_{t}&P_{\text{AM}}(\pi_{t}|\mathbf{x})\\\
\times&P_{\text{BiLM}}(y_{\pi_{t}}|\mathcal{B}(\boldsymbol{\pi}_{1:t-1}),\mathcal{B}(\boldsymbol{\pi^{*}}_{t+1:}))^{\alpha}\end{split}$
(4)

where $\alpha$ is the hyper-parameter controlling the effect of the language
model to the final decoding and the second term of product can be evaluated as
stated in $\eqref{eq:bilm}$.

With such bidirectional beam search, we may expect the decoding to get more
valuable semantic information from the future sequence
$\mathcal{B}(\boldsymbol{\pi^{*}}_{t+1:})$ helpful to predict next label
output. Beam search with unidirectional language model can be relatively
inaccurate at the start of the sentence because there are not much context for
language model at the front part. Bidirectional language model having context
from both front and back would be more grounded on support for the backward
context from the future information.

###  3.3 Future Exposure Bias

The well known exposure bias problem implies that the model is exposed only on
ground-truth context, while not exposed to predicted context. Both forward and
backward language model (FWLM, BWLM) affect from this exposure bias problem
due to its autoregressive nature. Especially the BWLM employ noisy context
given from greedy decoding, where noisy context cannot be trivially obtainable
from text corpus. To simulate noisy context given from CTC greedy decoding, we
propose two data augmentation methods.

####  3.3.1 Future Shift

For backward language model, exposure bias in the immediate future input can
cause critical problem in predicting the next output. In order to mitigate
this problem, we have to train the backward part in the bidirectional language
model with a future shift $\tau\in\mathbb{Z}$ so that there should be only
effect of future sequence $\mathbf{y}_{t+1+\tau:}$ where
$\mathbf{y}_{t+1:}=\mathcal{B}(\pi^{*}_{t+1:})$ on predicting $y_{t}$. If
$\tau=0$, there is no shift in future and it means this backward may suffer
from exposure bias problem. To achieve the expected result, backward language
model should be trained and decoded with a future shift $\tau$. Fig. 2-(b)
shows how to make shift on future context.

####  3.3.2 Random Noise In Future

To further minimize the difference between training data and real data, we
simulate noisy greedy decoding to be used in training backward language model.
Greedy decoding could make three kinds of errors: insertion, deletion and
substitution. Substitution error can be simulated by simply changing the input
of a certain step, $y_{t}$ into some other random input $y^{\prime}_{t}$.
Simulating insertion and deletion is more complicated as both insertion and
deletion break the alignment between the future sequence and input sequence.
Fig. 2-(c),(d) shows how to align future sequence with random noises when the
bidirectional LM is trained.

##  4 Experiments

###  4.1 Dataset

The CTC is trained over the LibriSpeech dataset [21] which contains 960 hours
of speech data of reading books. The models are tested in over two different
datasets; dev-clean and test-clean which have 5.4 hours respectively. We used
a character-level dictionary consisting of 48 tokens including special tokens
such as unknown, start of sentence, end of sentence and other special
characters. Every model is trained on a single NVIDIA V100 GPU.

The external bidirectional language model is trained on every train corpus
consisting of about 280,000 sentences. Sentences shorter than 10 letters and
consisting of almost non-alphabet characters have been removed.

###  4.2 Acoustic Model

The 161-dimensional log spectrogram is used as the input of the CTC model. Two
convolutional layers are used to compress the information of the log
spectrogram on the temporal dimension. Four bidirectional LSTM layers with
2048 hidden nodes are used for acoustic information analysis. Then, one fully
connected layer is used to compute the probability of each token from the
output of the LSTM layers. The dimension of the final output vector of the CTC
model is 49, which is one greater than the number of tokens because the CTC
model should compute the probability of the blank label.

###  4.3 Language Model

We used both LSTM and Transformer-XL in order to train bidirectional language
model described in Section 3.1. We used 6 LSTM layers with implementation
details following AWD-LSTM [22]. For Transformer-XL there are 6 Transformer
decoder layers with memory-augmented hidden states for previous hidden states
in auto-regressive decoding. Every hidden state has 1024-dimensional vector
for each layer.

There are two training parameters for bidirectional language model: future
shift $\tau$ and random noise proportion $\epsilon$. We have experimented for
$\tau=1,2,3$. Another training parameter is $\epsilon$ controlling the random
noise density in future sequence. We assume that the proportion of noise has
to follow the actual error rate occurred in greedy decoding. Since the
character error rate (CER) of CTC greedy decoding turned out around 5%, we
tried to set $\epsilon$ to be 0.05; 5 percents out of all characters in a
sequence is set to be noise. Among these noises, we set 45% to be insertion
noise, and another 20% to be deletion noise and the rest of them to be
substitution noises, following the statistics of the greedy result Table 1. We
are going to compare between the one with these modeling structure and the
other without it.

type | dev-clean | test-clean
---|---|---
insertion | 43.2% | 45.6%
deletion | 22.7% | 21.9%
substitution | 34.1% | 32.5%
Table 1: Error types proportion in greedy decoding

###  4.4 Beam Search

CTC prefix beam search finds the best $B$ beams for each time step $t$ with
the score

$\begin{split}score(\mathbf{y})&=\sum_{t}\Big{\\{}\log
P_{\text{CTC}}(y_{t}|X)\\\ &+\alpha\log
P_{\text{LM}}(y_{t}|y_{1:t-1})\Big{\\}}+\beta|\mathbf{y}|\end{split}$ (5)

where $\alpha$ is the language model weight and $\beta$ is for the length
reward for normalization with respect to the length due to the bias for
shorter utterances.

For all experiments, we set $B=20$, $\alpha=1$, $\beta=2$ since we are looking
for the effect of bidirectional decoding, not finding the optimal parameters.

Detailed algorithm is described in Algorithm 1. First, we get the greedy
decoding result $\mathbf{y}_{1:T_{y}}^{*}$. Using it, we are able to obtain
hidden states of backward language model for each $t$. These hidden states are
cached and used for the bidirectional language model through time. For time
$t$, we should get $t^{*}$ in line 10, the earliest time step after time $t$
having non-blank greedy output. With $t^{*}$ we can determine which part we
are going to use among the backward hidden states. Depending on the future
shift parameter $\tau$, $\overleftarrow{h}_{t^{*}+\tau}$ would be used as a
backward hidden state. With forward hidden state $\overrightarrow{h}^{b}_{t}$,
language model score $\boldsymbol{p}^{lm}$, $V$-dimensional vector having
probability for each output label, can be calculated. By regular CTC prefix
search algorithm (CTC_PREFIX_SEARCH) [12], $\boldsymbol{p}^{total}$ is
obtained, where the second argument of CTC_PREFIX_SEARCH is to be added to the
non-blank probability in this function. With pruning beams over $\delta$,
picking $B$ best beams based on the score $\boldsymbol{p}^{total}$ for the
next time step. The inference of backward language model is done just once
with greedy decoding result, there is not much difference in amount of total
calculation.

Algorithm 1 Bidirectional CTC beam search

0: $B$: Beam Width, $V$: Vocab Size, $\tau$: future shift, $\delta$: Beam
Threshold, $\alpha$: language model weight, $\beta$: length reward

1: $\mathbf{y^{*}}_{1:T_{y}}\leftarrow\mathcal{B}(\pi^{*}_{1:T_{x}})$ (Greedy
Decoding)

2: $\overleftarrow{h}_{T_{y}}\leftarrow\boldsymbol{0}$

3: for $t\leftarrow T_{y}-1$ to 0 do

4:
$\overleftarrow{h}_{t}\leftarrow\overleftarrow{f}_{h}(y^{*}_{t+1},\overleftarrow{h}_{t+1})$

5: end for

6: $\overrightarrow{h}_{0}\leftarrow\boldsymbol{0}$

7: $\hat{B}\leftarrow[\textless
s\textgreater],\overrightarrow{h}^{0}_{0}\leftarrow\overrightarrow{h}_{0}$

8: for $t\leftarrow 1$ to $T_{x}$ do

9: $\boldsymbol{p}^{\text{am}}\leftarrow\log P_{\text{CTC}}(y_{t}|\mathbf{X})$

10: $t^{*}\leftarrow\min(\\{s\in(t,T_{y}):\pi^{*}_{s}\neq\text{\textless
blank\textgreater})\\})$

11: for $b\leftarrow 1$ to $|\hat{B}|$ do

12: $\mathbf{y}^{b}\leftarrow\hat{B}[b]$

13:
$\overrightarrow{h}^{b}_{t}\leftarrow\overrightarrow{f}_{h}(y_{t-1},\overrightarrow{h}^{b}_{t-1})$

14: $\boldsymbol{p}^{\text{lm}}\leftarrow\log
P_{\text{BiLM}}(y_{t}|\overrightarrow{h}^{b}_{t}+\overleftarrow{h}_{t^{*}+\tau})$

15:
$\boldsymbol{p}^{total}\leftarrow\textrm{CTC\\_PREFIX\\_SEARCH}(\boldsymbol{p}^{am},\alpha\boldsymbol{p}^{lm}+\beta|\mathbf{y}^{b}|)$

16: for $c\leftarrow 1$ to $V$ do

17: if $p^{total}_{c}>\delta$ then

18: add $(c+\mathbf{y}^{b})$ to $\hat{B}$

19: end if

20: end for

21: end for

22: $\hat{B}$ $\leftarrow$ top $B$ beams in $\hat{B}$

23: Reorder $\overrightarrow{h}^{b}_{t}$ according to $\hat{B}$

24: end for

##  5 Results

###  5.1 Perplexity

In this section, we show the improvement of perplexity of the bidirectional
language model compared to the unidirectional language model. We trained both
LSTM and Transformer-XL varying the hyper-parameter $\tau$ from 1 to 3.

As we can see in Table 2, compared to the result of unidirectional language
model, bidirectional LM has improved the model performance significantly for
both LSTM and Transformer-XL based on the perplexity measure. This improvement
is quite obvious since predicting with future information is surely more
helpful than without it as well as future information is guaranteed to be
perfect. Similarly, closer the information is, the more helpful it is because
closer future is more related. Therefore smaller $\tau$ makes the results more
accurate. However, like we explained in 3.3, smaller value of $\tau$ also
increases the risk of inaccurate result from incorrect future information.

model | LSTM | T-XL
---|---|---
dataset | dev | test | dev | test
uni | 5.811 | 5.745 | 5.539 | 5.505
$\tau$=1 | 2.711 | 2.676 | 2.451 | 2.245
$\tau$=2 | 4.327 | 4.281 | 3.409 | 3.214
$\tau$=3 | 5.351 | 5.171 | 4.564 | 4.452
Table 2: Perplexity of Language Models

###  5.2 Decoding Results

Table 3 shows that for both datasets, bidirectional decoding is quite
effective. However, we speculate that the direct future information gives side
effects as we anticipated in advance, for observing the CER result of the case
$\tau=1$. For LSTM $\tau=2$ seems to be optimal whereas $\tau=3$ for
Transformer-XL. Furthermore, the best performance is obtained both for LSTM
and Transformer-XL with adding random noise to greedy result during training
with ratio of $\epsilon=0.05$. We note that the purpose of our experiment is
to prove the effectiveness of the proposed bidirectional decoding. Therefore,
the mentioned $\tau$ and $\epsilon$ value may not be optimal. More experiments
should be conducted in future to find out the generally optimal hyper-
parameters.

LM | LSTM | T-XL
---|---|---
dataset | dev | test | dev | test
greedy | 5.30 | 5.22 | 5.31 | 5.22
uni | 4.56 | 4.48 | 4.37 | 4.12
$\tau$=1 | 5.15 | 5.02 | 4.90 | 4.77
$\tau$=2 | 4.40 | 4.23 | 4.29 | 4.12
$\tau$=3 | 4.51 | 4.31 | 4.22 | 4.05
$\tau$=2, $\epsilon$=0.05 | 4.38 | 4.21 | 4.24 | 4.07
$\tau$=3, $\epsilon$=0.05 | 4.46 | 4.28 | 4.17 | 4.02
Table 3: Character Error Rate(CER) on LibriSpeech dev-clean and test-clean
dataset for greedy search decoding, unidirectional beam search, and
bidirectional beam search with $\tau=1,2,3$ and $\epsilon=0,0.05$.

###  5.3 Relative Error Position

In order to further analyse the effectiveness of the proposed method, we
visualized the statistics of the relative error position as a histogram in
Fig. 3. As shown in the figure, the unidirectional decoding shows obviously
poorer performance at the beginning of the sentence, which coincides with the
analysis in 3.2. On the contrary, bidirectional decoding shows relatively
uniform distribution through all relative positions, implying that the
proposed bidirectional decoding outperforms unidirectional one in overall
error rates. nhancement gets the maximum at the front 10% of the sequence,
implying that the bidirectional decoding is especially good at correcting
errors located in the front part.

Fig. 3: Histogram of relative positions where each error has occurred in test-
clean with Transformer-XL model and $\tau=3$: $x$-axis represents the relative
position of every error part of the recognition results. 0 means the beginning
of the speech and 1 means the end of the speech. $y$-axis shows the total
count of errors for each bin. Yellow line implies the improvement from
unidirectional to bidirectional.

In Table 4, there are a few examples that seems to be corrected by future
information properly. For instance, the second sentence the greedy decoding
result is tusday august eighteenth and unidirectional decoding has no choice
but to decode as it is. However for bidirectional decoding, it is able to see
the future part august eighteenth and successfully found out the right answer
“tuesday". Although the results are not flawless, but we can expect positive
effect from the future like these examples.

unidirectional |  bidirectional
---|---
sweak squeak |  squeak squeak
tusday august eighteenth |  tuesday august eighteenth
i name nine others and said |  i named nine others and said
Table 4: These examples show that bidirectional decoding uses future
information when necessary. In these cases, it is actually very hard to get
the right text without seeing the future information as in unidirectional
decoding.

##  6 Conclusions

In this paper, we proposed new decoding method for CTC speech recognition
model with language model trained bidrectionally. Bidirectional language model
can be obtained by traditional unidirectional language model and backward
language model for taking advantage of future information, for which greedy
decoding result is used. With several experiments, we have demonstrated that
the proposed method helps for decoding the front part of the speech compared
to unidirectional decoding. Furthermore, we have alleviated future exposure
bias problem by future shift and random noise. As a future work, we could
apply bidirectional decoding to transducer-based model [23, 24, 25] or label-
synchronous attention-based decoder [10, 26].

## References

  * [1] George E Dahl, Dong Yu, Li Deng, and Alex Acero,  “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,”  IEEE Transactions on audio, speech, and language processing, vol. 20, no. 1, pp. 30–42, 2011.
  * [2] Li Deng, Geoffrey Hinton, and Brian Kingsbury,  “New types of deep neural network learning for speech recognition and related applications: An overview,”  in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 8599–8603.
  * [3] Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu,  “Convolutional neural networks for speech recognition,”  IEEE/ACM Transactions on audio, speech, and language processing, vol. 22, no. 10, pp. 1533–1545, 2014.
  * [4] Haşim Sak, Andrew Senior, and Françoise Beaufays,  “Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,”  arXiv preprint arXiv:1402.1128, 2014.
  * [5] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio,  “End-to-end attention-based large vocabulary speech recognition,”  in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016, pp. 4945–4949.
  * [6] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.,  “Deep speech 2: End-to-end speech recognition in english and mandarin,”  in International conference on machine learning, 2016, pp. 173–182.
  * [7] Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas Usunier, Gabriel Synnaeve, and Ronan Collobert,  “Fully convolutional speech recognition,”  arXiv preprint arXiv:1812.06864, 2018.
  * [8] Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina Gonina, et al.,  “State-of-the-art speech recognition with sequence-to-sequence models,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4774–4778.
  * [9] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [10] William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals,  “Listen, attend and spell,”  CoRR, vol. abs/1508.01211, 2015.
  * [11] Linhao Dong, Shuang Xu, and Bo Xu,  “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5884–5888.
  * [12] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376.
  * [13] Alex Graves and Navdeep Jaitly,  “Towards end-to-end speech recognition with recurrent neural networks,”  in International conference on machine learning, 2014, pp. 1764–1772.
  * [14] Rohit Prabhavalkar, Kanishka Rao, Tara N Sainath, Bo Li, Leif Johnson, and Navdeep Jaitly,  “A comparison of sequence-to-sequence models for speech recognition.,”  in Interspeech, 2017, pp. 939–943.
  * [15] Awni Y Hannun, Andrew L Maas, Daniel Jurafsky, and Andrew Y Ng,  “First-pass large vocabulary continuous speech recognition using bi-directional recurrent dnns,”  arXiv preprint arXiv:1408.2873, 2014.
  * [16] Niko Moritz, Takaaki Hori, and Jonathan Le Roux,  “Streaming end-to-end speech recognition with joint ctc-attention based models,”  in Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2019.
  * [17] Chen Xie, Liu Xunying, Ragni Anton, Wang Yu, and Gales Mark,  “Future word contexts in neural network language models,”  IEEE Automatic Speech Recognition and Understanding, 2017.
  * [18] Wang Bin and Ou Zhijian,  “Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation,”  IEEE Spoken Language Technology, 2018.
  * [19] Shin Joongbo, Lee Yoonhyung, and Jung Kyomin,  “Effective sentence scoring method using bert for speech recognition,”  ACML, 2019.
  * [20] Zhang Xiangwen, Su Jinsong, Qin Yue, Liu Yang, Ji Rongrong, and Wang Hongji,  “Asynchronous bidirectional decoding for neural machine translation,”  AAAI, 2018.
  * [21] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  * [22] Stephen Merity, Nitish Shirish Keskar, and Richard Socher,  “Regularizing and optimizing lstm language models,”  arXiv preprint arXiv:1708.02182, 2017.
  * [23] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar,  “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,”  in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 193–199.
  * [24] Albert Zeyer, André Merboldt, Wilfried Michel, Ralf Schlüter, and Hermann Ney,  “Librispeech transducer model with internal language model prior correction,”  arXiv preprint arXiv:2104.03006, 2021.
  * [25] Chunxi Liu, Frank Zhang, Duc Le, Suyoun Kim, Yatharth Saraf, and Geoffrey Zweig,  “Improving rnn transducer based asr with auxiliary tasks,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 172–179.
  * [26] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.