# Conversation-oriented ASR with multi-look-ahead CBS architecture

###### Abstract

During conversations, humans are capable of inferring the intention of the
speaker at any point of the speech to prepare the following action promptly.
Such ability is also the key for conversational systems to achieve rhythmic
and natural conversation. To perform this, the automatic speech recognition
(ASR) used for transcribing the speech in real-time must achieve high accuracy
without delay. In streaming ASR, high accuracy is assured by attending to
look-ahead frames, which leads to delay increments. To tackle this trade-off
issue, we propose a multiple latency streaming ASR to achieve high accuracy
with zero look-ahead. The proposed system contains two encoders that operate
in parallel, where a primary encoder generates accurate outputs utilizing
look-ahead frames, and the auxiliary encoder recognizes the look-ahead portion
of the primary encoder without look-ahead. The proposed system is constructed
based on contextual block streaming (CBS) architecture, which leverages block
processing and has a high affinity for the multiple latency architecture.
Various methods are also studied for architecting the system, including
shifting the network to perform as different encoders; as well as generating
both encoders’ outputs in one encoding pass.

Index Terms—  streaming ASR, zero latency, conversational system

##  1 Introduction

We humans can infer the intention of a speaker in conversation, even in the
middle of an utterance, and prepare the following action to be taken. This
function allows us to respond at the appropriate timing, sometimes without
waiting for the end of an utterance, and to achieve a rhythmic and natural
conversation. To achieve this function in conversational systems, the speech
recognizer of the system is required to transcribe the input speech accurately
without delay as any instant in time. High accuracy can be achieved in speech
recognition by applying look-ahead, which provides the speech recognizer with
more forward clues to make reliable decisions, but leads to look-ahead
latency, which largely increases the delay of speech recognition. This
research aims to develop a highly accurate speech recognizer that can operate
with zero look-ahead.

One recent trend in speech recognition develops around the end-to-end models
[1, 2, 3, 4]. Among them, the recent Transformer-based methods achieve high
performance by taking advantage of the self-attention function, but also
require look-ahead in their structure [5, 6, 7, 8, 9, 10]. Conversational
speech recognition requires streaming ASR, but the look-ahead requirement also
exists to guarantee the performance [11, 12, 13]. Although it can be
implemented in causal by applying attention-mask to the look-ahead part [14,
15, 16], the degradation from full context implementations that allow look-
ahead is significantly large [17].

In this vein, there are attempts in the multi-latency approach, which combines
a short look-ahead ASR and a long look-ahead one [18, 19, 20, 21]. In [18], a
high-latency encoder (long look-ahead) operates on the outputs of a low-
latency (short look-ahead) encoder to correct the beam search results. In
[20], a second-pass non-streaming recognition is conducted to refine the
first-pass streaming outputs. A common feature of them is the cascaded
configuration, in which the high-latency, high-precision recognizers operate
on the results of the low-latency recognizers to compensate for them.

Similarly, the proposed system in this study is a multi-latency ASR that
combines a high-latency/high-accuracy encoder and a low-latency one. However,
it is unique in that the system operates both encoders in parallel and adopts
contextual block streaming ASR [22, 23] (referred to as CBS), which has a high
affinity with multi-latency architecture, as the base system. In the proposed
method, the decoder operates primarily with the output of the primary encoder,
which works with the look-ahead. However, the look-ahead portion of the
primary encoder, where there is no output from the primary encoder, is taken
by the auxiliary encoder, which operates with zero look-ahead. Thus, the whole
system constitutes a recognizer that operates accurately with zero look-ahead.

The CBS proposed by Tsunoo et al., the base system of our encoder, leverages
block processing to achieve streaming properties in the attention-based
encoder-decoder model architecture. CBS contains a contextual block streaming
encoder [22], which gracefully utilizes contextual information from the
previous block and achieves high recognition accuracy. Compared to frame-wise
computation, block-wise processing in CBS possesses higher efficiency for
using both primary and auxiliary encoders simultaneously. With an easily
adjustable look-ahead range in the block setting, CBS is also suitable for
realizing parameter sharing of multiple encoders.

Since the original CBS (referred to as CBS-E/D) is based on the encoder-
decoder architecture, the high computational cost in the decoder limits the
real-time performance of CBS. Hence, in this paper, we also propose
Transducer-based CBS (referred to as CBS-T), which combines the encoder of CBS
and Transducer [24, 25, 26] to speed up the whole process, and also try to
construct multi-latency ASR based on it.

The rest of the paper is organized as follows. Section 2 introduces our base
models: CBS-E/D and CBS-Transducer. In Section 3, we describe the proposed
multiple latency streaming ASR system and provide various methods for
constructing the system. In Section 4, we examine the effectiveness of the
proposed method through speech recognition experiments and analyze the
results. Finally, Section 5 concludes this paper.

##  2 Background

In this study, we adopt both contextual block streaming encoder-decoder
(CBS-E/D) and contextual block streaming Transducer (CBS-T) as our base
models.

###  2.1 Contextual block streaming encoder-decoder

As an attention-based encoder-decoder model, CBS-E/D conducts streaming
processing in both encoding and decoding. As shown in Fig. 1, for streaming
encoding, CBS-E/D utilizes block processing with a context inheritance
mechanism [22]. The speech input is segmented into blocks containing history,
target, and look-ahead frames with the numbers of $N_{l}$, $N_{c}$, and
$N_{r}$. When a block is passed on to the encoder, the target frames are
processed for the output with future contexts provided by the look-ahead
frames, as well as history contexts provided by history frames and a
contextual embedding vector inherited from the previous block. Streaming
decoding is achieved by a block boundary detection (BBD) algorithm [23], which
examines the outputs’ index boundaries and enables the beam search synchronous
to the encoded blocks. The streaming processing in CBS-E/D is calculated as
follows:

$\displaystyle
H_{b},\bm{\mathrm{c}}_{b}=\mathrm{BlockEncoder}(Z_{b},\bm{\mathrm{c}}_{b-1}),$
(1)
$\displaystyle\alpha(y_{0:i},H_{1:B})\approx\sum_{b=1}^{B}\sum_{j=I_{b-1}+1}^{I_{b}}\log
p(y_{i}|y_{0:j-1},H_{1:b}).$ (2)

Eq. (1) represents the streaming encoding where the $b$-th input block $Z_{b}$
with $|Z_{b}|=N_{l}+N_{c}+N_{r}$ and a contextual vector from the previous
block $\bm{\mathrm{c}}_{b-1}$ are processed to output the acoustic feature
$H_{b}$ and current contextual vector $\bm{\mathrm{c}}_{b}$. Eq. (2)
represents the score of the partial hypothesis $y_{0:i}$ during streaming beam
search decoding, where $y_{0}$ is the start-of-sequence token. $I_{b}$ denotes
the index boundary of the $b$-th input block derived from the BBD algorithm.

Fig. 1: Block processing in CBS encoder

###  2.2 Contextual block streaming Transducer

A CBS-T model combines the CBS encoder and the Transducer framework. A
Transducer framework contains three components: acoustic encoder, label
encoder, and joint network. Given a streaming input to a current time index
$t$, the output probability of each token is calculated as follows:

$\displaystyle\bm{\mathrm{h}}_{t}^{\mathsf{AE}}=\mathrm{AcousticEncoder}(\bm{\mathrm{x}}_{1:t}),$
(3)
$\displaystyle\bm{\mathrm{h}}_{u-1}^{\mathsf{LE}}=\mathrm{LabelEncoder}(y_{1:u-1}),$
(4)
$\displaystyle\bm{\mathrm{h}}=\mathrm{Tanh}(\mathrm{Linear}(\bm{\mathrm{h}}_{t}^{\mathsf{AE}})+\mathrm{Linear}(\bm{\mathrm{h}}_{u-1}^{\mathsf{LE}})),$
(5) $\displaystyle
P(y_{u}|y_{1:u-1},\bm{\mathrm{x}}_{1:t})=\mathrm{SoftMax}(\bm{\mathrm{h}}),$
(6)

where the acoustic feature $\bm{\mathrm{h}}_{t}^{\mathsf{AE}}$ extracted from
$\bm{\mathrm{x}}_{1:t}$ (Eq. (3)) and the feature
$\bm{\mathrm{h}}_{u-1}^{\mathsf{LE}}$ from the previous output token sequence
$y_{1:u-1}$ (Eq. (4)) are sent to the joint network, projected to the same
dimension, and added up (Eq. (5)) to calculate the output probabilities
against tokens in $\mathcal{V}$ based on the previous result (Eq. (6)). Since
the current symbol for each input frame is predicted based only on the past
output tokens, streaming decoding is naturally introduced into the Transducer
framework without additional effort.

In CBS-T, we utilize the CBS encoder as the acoustic encoder of a Transducer
model to conduct streaming feature extraction along with Transducer streaming
decoding, which achieves significant computational complexity reduction
compared to CBS-E/D.

##  3 Proposal

To achieve high accuracy with zero look-ahead, we propose a multiple latency
streaming ASR system, which leverages both a primary encoder with high latency
to generate accurate outputs and an auxiliary encoder to recognize the look-
ahead frames attended by the primary encoder with no additional look-ahead. In
this section, we first describe the proposed system and then provide different
methods for architecting the multiple latency streaming ASR.

###  3.1 Multiple latency streaming ASR with CBS models

Our proposed system can be constructed with both CBS-E/D and CBS-T streaming
ASR models. For block settings of the CBS encoder, we fix the size of history
frames $N_{l}$ and target frames $N_{c}$ as $8$ and $4$, while the look-ahead
range is controlled by choice of the look-ahead frame number $N_{r}$. For
instance, a primary encoder with the block setting of 8-4-4 ($N_{l}=8$,
$N_{c}=4$, and $N_{r}=8$) attends to four look-ahead frames, which induces a
$128$ ms delay with a frame rate of $32$ ms. Similarly, an auxiliary encoder
with the block setting of 8-4-0 attends to a zero look-ahead frame.

Algorithm 1 demonstrate how the proposed system works, where the look-ahead
frame number in the primary encoder equals the target frame number in the
auxiliary encoder (i.e., $N_{r}=N^{\prime}_{c}$). During recognition, the
primary encoder recognizes the target frames of the input block and outputs
sequence $\textbf{y}^{p}_{c}$. Simultaneously, the auxiliary encoder
recognizes the look-ahead frames of the input block and yields
$\textbf{y}^{a}_{r}$. $\textbf{y}^{p}_{c}$ is then appended to the previous
output sequence $\textbf{y}^{P}$. At the appropriate timing, the auxiliary
encoder outputs a special token $\langle/s\rangle$, and the speech recognition
process is terminated. We concatenate the previous output sequence
$\textbf{y}^{P}$ with the auxiliary encoder output $\textbf{y}^{a}_{r}$ as the
final result. Otherwise, the streaming ASR moves on to the next input block.

Since the target frame of the auxiliary encoder $N^{\prime}_{c}$ is fixed as
four, the assumption of $N_{r}=N^{\prime}_{c}$ constraints the look-ahead
range of the primary encoder and limits the accuracy of the streaming ASR.
Therefore, we extend the two-encoder system in Algorithm 1 with multiple
auxiliary encoders, as shown in the upper part of Fig. 2. Here the primary
encoder attends to eight look-ahead frames, which are recognized by two
auxiliary encoders with four and zero look-ahead frames, respectively. With a
more extensive look-ahead range ($256$ ms), the streaming ASR achieves higher
recognition accuracy while maintaining the operation with zero look-ahead.

Algorithm 1 Multi-latency streaming ASR

1:$T^{p}_{B}=N_{l}+N_{c}+N_{r}$ $\triangleright$ primary encoder block setting

2:$T^{a}_{B}=N^{\prime}_{l}+N^{\prime}_{c}$ $\triangleright$ auxiliary encoder
block setting

3:$\textbf{y}^{P}\leftarrow\emptyset$

4:for $t$ = $T^{p}_{B}$ to $T$ by $T^{p}_{B}$ do

5: $\textbf{y}^{p}_{c}$ = PrimaryEncoder($X[t-T^{p}_{B},t]$)

6: $\textbf{y}^{a}_{r}$ = AuxiliaryEncoder($X[t-T^{a}_{B},t]$)

7: $\textbf{y}^{P}\leftarrow\textbf{y}^{p}_{c}$ $\triangleright$ extend target
frame results

8: if $\langle/s\rangle$ in $\textbf{y}^{a}_{r}$ then $\triangleright$
appropriate timing for ending

9: break

10:$\textbf{y}=\textbf{y}^{P}+\textbf{y}^{a}_{r}$ $\triangleright$ final
result

Fig. 2: Structure of proposed system (case of $N_{r}=2\times N_{c}$). Data
inputs of History, Target, and Look-ahead frames are indicated by arrows
pointing at symbols H, T, and A, respectively. In method A, primary encoder
and auxiliary encoders share same structure.

###  3.2 Implementation methods

The proposed multiple latency streaming ASR contains a primary encoder with
look-ahead frames and an auxiliary encoder recognizing the look-ahead frames
that operate in parallel. The primary encoder is implemented following the
same structure as the encoder in existing CBS models. The auxiliary encoder
shares parameters with the primary encoder but operates without attending to
any look-ahead frames. In this study, we propose two different methods to
architect the proposed system, which are illustrated in Fig. 2.

Method A (Parallel model). In Method A, the system contains $N_{r}/N_{c}$
auxiliary encoders in parallel, each of which shares exactly the same
structure and parameters as the primary encoder. The $i$-th auxiliary-encoder
uses the input of the primary encoder shifted forward by $i\times N_{c}$
frames. This means the look-ahead frames for the $i$-th encoder are shortened
by $i\times N_{c}$ frames. In the training phase, the primary encoder and all
the auxiliary encoders are simultaneously trained by masking the last $N_{c}$,
$2\times N_{c}$, $\cdots$, $(N_{r}/N_{c})\times N_{c}$ look-ahead frames with
a certain probability. This is expected to correctly encode the target part
while enabling recognition of the look-ahead part without delay.

Method B (Unified model). In Method B, we utilize a single network to
recognize both target frames and look-ahead frames in one encoding pass. With
the block setting of $N_{l}$-$N_{c}$-$N_{r}$ , the model is trained to
recognize $N_{c}$ only as well as recognize $N_{c}$ and $N_{r}$ together.
During inference, the model recognizes both $N_{c}$ and $N_{r}$ in one
encoding pass, where the results for $N_{c}$ are regarded as the primary
encoder outputs, and the results for $N_{r}$ are used as the auxiliary encoder
outputs. Method B significantly reduces the computational cost during
inference. On the other hand, it solves more challenging problems than method
A due to a large number of outputs. The difficulty increases as the length of
$N_{r}$ is extended.

##  4 Experiments

Speech recognition experiments were conducted on the proposed multiple latency
streaming ASR system using ESPnet2 [27, 28].

###  4.1 Experimental setup

The models were trained and evaluated using the Wall Street Journal (WSJ) [29]
dataset. We applied SpecAugment [30] to the input data for robust model
training. For the output tokens, we used SentencePiece [31] to construct an 80
subword vocabulary from the training set with one additional token
$\langle/s\rangle$.

We conducted experiments with both CBS-E/D and CBS-T models. The CBS-E/D model
consisted of a CBS encoder with six Conformer [6] layers and decoder with six
Transformer [5] layers. For CBS-T, we used a CBS encoder with six Conformer
layers for the acoustic encoder and one layer of long short-term memory (LSTM)
network [32] for the label encoder.

All the models were trained by 150 epochs, and the final models were obtained
by averaging the snapshots of the ten epochs with the best accuracy for
CBS-E/D and minimal losses for CBS-T. For decoding, a beam search was
conducted with a beam size of ten for all. We used the averaged word error
rates (WER) on standard validation and test sets (dev93 and eval92) to measure
the recognition accuracy.

Table 1: Word error rates on WSJ dataset.

|  |  | Delay | WER
---|---|---|---|---
Model | Mode | Block Setting | [ms] |  [%] ($\downarrow$)
CBS-E/D | Single | 8-4-0 | 0 | 15.8
Single | 8-4-4 | 128 | 14.3
Multiple (A) | 8-4-4, 8-4-0 | 0 | 14.8
Multiple (B) | 8-4-4, 12-4-0 | 0 | 14.8
Single | 8-4-8 | 256 | 13.9
Multiple (A) | 8-4-8, 8-4-4, 8-4-0 | 0 | 14.3
Multiple (B) | 8-4-8, 12-8-0 | 0 | 14.2
CBS-T | Single | 8-4-0 | 0 | 16.4
Single | 8-4-4 | 128 | 14.4
Multiple (A) | 8-4-4, 8-4-0 | 0 | 14.5
Multiple (B) | 8-4-4, 12-4-0 | 0 | 14.6
Single | 8-4-8 | 256 | 13.7
Multiple (A) | 8-4-8, 8-4-4, 8-4-0 | 0 | 13.7
Multiple (B) | 8-4-8, 12-8-0 | 0 | 14.1

###  4.2 Experimental results

The experimental results are summarized in Table 1, where block settings are
shown in the format of $N_{l}$-$N_{c}$-$N_{r}$, and the processing delay
induced by look-ahead frames is recorded in the column of Delay. Results of
the proposals are listed under mode names Multiple (A) (Parallel model) and
Multiple (B) (Unified model). Baseline models are represented by mode name
Single, serving as upper-bounds (w/ look-ahead) and lower-bounds (w/ look-
ahead).

Comparing the results of CBS-E/D and CBS-T, we can see that CBS-T showed
inferior lower-bound results but outperformed CBS-E/D when look-ahead frames
were applied. Considering reducing decoding time, CBS-T showed higher
suitability for our proposal. Extending the primary encoder block setting from
$8$-$4$-$4$ to $8$-$4$-$8$ vastly improved the performance of the streaming
ASR. With the setting of $8$-$4$-$8$, CBS-T achieved a WER of $13.7\%$, which
is very close to the non-streaming result with the same model structure
($12.8\%$, not shown in the table). Regarding the different architectures of
the proposal, methods A and B presented the same level of performance in most
cases, while for CBS-T, method A showed better accuracy with an extensive
look-ahead range applied. Hence, for implementing the proposed system, method
A is more suitable when accuracy is more critical, while method B should be
adopted when the computational cost comes first. Overall, with the proposed
multiple latency system, we managed to maintain high recognition accuracy
while operating with zero look-ahead. Compared to the upper-bound result, zero
performance degradation was achieved when applying method A to CBS-T with
eight look-ahead frames.

##  5 Conclusion

In this study, we proposed a multiple latency streaming ASR system based on
the CBS models to operate encoders with various latency in parallel. Various
implementation methods were studied for constructing the system. Experimental
results have shown our proposal’s effectiveness in maintaining high
recognition accuracy with zero look-ahead. Our future work will focus on the
trade-off between computational cost and recognition accuracy. We are also
planning to incorporate the proposed model into the EoU (end-of-
utterance)-detection-free turn-taking model [33] to realize a rhythmic
conversation system.

## References

  * [1] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _Procs. ICML_ , 2014, pp. 1746–1772.
  * [2] J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho _et al._ , “Attention-based models for speech recognition,” in _Procs. NeurIPS_ , 2015.
  * [3] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _Proc. ICASSP_ , 2016, pp. 4960–4964.
  * [4] S. Watanabe, T. Hori, S. Kim, J. R. Hershey _et al._ , “Hybrid CTC/attention architecture for end-to-end speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 8, pp. 1240–1253, 2017.
  * [5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit _et al._ , “Attention is all you need,” in _Procs. NeurIPS_ , 2017, pp. 5998–6008.
  * [6] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar _et al._ , “Conformer: Convolution-augmented Transformer for speech recognition,” in _Procs. INTERSPEECH_ , 2020, pp. 5036–5040.
  * [7] L. Dong, S. Xu, and B. Xu, “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,” _Procs. ICASSP_ , pp. 5884–5888, 2018.
  * [8] S. Karita, N. Chen, T. Hayashi, T. Hori _et al._ , “A comparative study on transformer vs rnn in speech applications,” _Procs. ASRU_ , pp. 449–456, 2019.
  * [9] C. Lüscher, E. Beck, K. Irie, M. Kitza _et al._ , “Rwth asr systems for librispeech: Hybrid vs attention - w/o data augmentation,” in _Procs. INTERSPEECH_ , 2019.
  * [10] Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa _et al._ , “Improved Mask-CTC for non-autoregressive end-to-end ASR,” in _Proc. ICASSP_ , 2021, pp. 8363–8367.
  * [11] M. Shannon, G. Simko, S. yiin Chang, and C. Parada, “Improved end-of-query detection for streaming speech recognition,” in _Procs. INTERSPEECH_ , 2017\.
  * [12] S. yiin Chang, R. Prabhavalkar, Y. He, T. N. Sainath _et al._ , “Joint endpointing and decoding with end-to-end models,” _Proc. ICASSP_ , pp. 5626–5630, 2019.
  * [13] S. yiin Chang, B. Li, T. N. Sainath, C. Zhang _et al._ , “Turn-taking prediction for natural conversational speech,” in _Procs. INTERSPEECH_ , 2022\.
  * [14] D. Povey, H. Hadian, P. Ghahremani, K. Li _et al._ , “A time-restricted self-attention layer for asr,” _Procs. ICASSP_ , pp. 5874–5878, 2018.
  * [15] S. Sukhbaatar, E. Grave, P. Bojanowski, and A. Joulin, “Adaptive attention span in Transformers,” in _Procs. ACL_ , 2019, pp. 331–335.
  * [16] X. Chang, A. S. Subramanian, P. Guo, S. Watanabe _et al._ , “End-to-end ASR with adaptive span self-attention,” in _Procs. INTERSPEECH_ , 2020\.
  * [17] N. Moritz, T. Hori, and J. Le, “Streaming automatic speech recognition with the Transformer model,” in _Proc. ICASSP_ , 2020, pp. 6074–6078.
  * [18] J. Mahadeokar, Y. Shi, K. Li, D. Le _et al._ , “Streaming parallel transducer beam search with fast-slow cascaded encoders,” _ArXiv_ , vol. abs/2203.15773, 2022.
  * [19] A. Narayanan, T. N. Sainath, R. Pang, J. Yu _et al._ , “Cascaded encoders for unifying streaming and non-streaming asr,” _Proc. ICASSP_ , pp. 5629–5633, 2021.
  * [20] T. N. Sainath, Y. He, A. Narayanan, R. Botros _et al._ , “An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,” in _INTERSPEECH_ , 2021.
  * [21] Y. Shi, V. K. Nagaraja, C. Wu, J. Mahadeokar _et al._ , “Dynamic encoder transducer: A flexible solution for trading off accuracy for latency,” _ArXiv_ , vol. abs/2104.02176, 2021.
  * [22] E. Tsunoo, Y. Kashiwagi, T. Kumakura, and S. Watanabe, “Transformer ASR with contextual block processing,” in _Proc. ASRU_ , 2019, pp. 427–433.
  * [23] E. Tsunoo, Y. Kashiwagi, and S. Watanabe, “Streaming Transformer ASR with blockwise synchronous beam search,” in _Proc. SLT_ , 2021, pp. 22–29.
  * [24] A. Graves, “Sequence transduction with recurrent neural networks,” _ArXiv_ , vol. abs/1211.3711, 2012.
  * [25] Q. Zhang, H. Lu, H. Sak, A. Tripathi _et al._ , “Transformer Transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,” in _Proc. ICASSP_ , 2020, pp. 7829–7833.
  * [26] X. Chen, Y. Wu, Z. Wang, S. Liu _et al._ , “Developing real-time streaming Transformer Transducer for speech recognition on large-scale dataset,” in _Proc. ICASSP_ , 2021, pp. 5904–5908.
  * [27] S. Watanabe, T. Hori, S. Karita, T. Hayashi _et al._ , “ESPnet: End-to-end speech processing toolkit,” in _Procs INTERSPEECH_ , 2018, pp. 2207–2211.
  * [28] F. Boyer, Y. Shinohara, T. Ishii, H. Inaguma _et al._ , “A study of Transducer based end-to-end ASR with ESPnet: Architecture, auxiliary loss and decoding strategies,” _Procs. ASRU_ , pp. 16–23, 2021.
  * [29] D. B. Paul and J. Baker, “The design for the wall street journal-based CSR corpus,” in _Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York_ , 1992.
  * [30] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu _et al._ , “Specaugment: A simple data augmentation method for automatic speech recognition,” _ArXiv_ , vol. abs/1904.08779, 2019.
  * [31] T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” _arXiv preprint arXiv:1808.06226_ , 2018.
  * [32] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, pp. 1735–1780, 1997.
  * [33] J. Sakuma, S. Fujie, and T. Kobayashi, “Response timing estimation for spoken dialog systems based on syntactic completeness prediction,” _Procs. SLT_ , 2022 (to appear).