# CarneliNet: Neural Mixture Model for automatic speech recognition

###### Abstract

End-to-end automatic speech recognition systems have achieved great accuracy
by using deeper and deeper models. However, the increased depth comes with a
larger receptive field that can negatively impact model performance in
streaming scenarios. We propose an alternative approach that we call Neural
Mixture Model. The basic idea is to introduce a parallel mixture of shallow
networks instead of a very deep network. To validate this idea we design
CarneliNet – a CTC-based neural network composed of three mega-blocks. Each
mega-block consists of multiple parallel shallow sub-networks based on 1D
depthwise-separable convolutions. We evaluate the model on LibriSpeech, MLS
and AISHELL-2 datasets and achieved close to state-of-the-art results for CTC-
based models. Finally, we demonstrate that one can dynamically reconfigure the
number of parallel sub-network to accommodate the computational requirements
without retraining.

Index Terms—  Automatic speech recognition, Convolutional networks,
Connectionist Temporal Classification

##  1 Introduction

End-to-end automatic speech recognition (ASR) systems based on neural networks
(NN) have become a new state-of-the-art. While the first NN-based ASR models
(2014, [1] and [2]) had only 5-6 hidden layers, the latest ASR models are much
deeper. Collobert et al [3] proposed Wav2Letter, a fully-convolutional model
with 12 layers to allow the network to "see" a larger context. Jasper [4]
added to the model residual connections, which allowed increasing the depth to
54 convolutional layers. QuartzNet [5] and TDS [6] models replaced regular
convolution with depth-wise separable convolutions to make the model deeper
while keeping model size small. For example QuartzNet-15x5 has 74
convolutional layers and a very large receptive field.

Dong et al [7] adapted a Transformer model to ASR. The Transformer approach
was further expanded in [8, 9, 10, 11]. One of the latest model in the
attention-based line, Conformer [12], is an RNN-Transducer with a very deep
encoder composed of 16 stacked residual blocks with each block consisting of
two feed-forward blocks, a convolution module and multi-headed self-attention
module stacked together. Han et al proposed ContextNet [13] \- an RNN-
Transducer with a global context mechanism based on squeeze-and-excitation.
The ContextNet encoder has 23 blocks with 5 depth-wise separable convolutional
layers per block.

The increase in the model depth has a few negative side-effects on model
performance, e.g. slower training, higher memory consumption and increased
inference latency. In this work, we explore an alternative way to scale
networks with width, without increasing parameter count. We propose CarneliNet
- a non-autoregressive, hierarchical, Neural Mixture Model trained using
Connectionist Temporal Classification (CTC) loss. The model consists of a
stack of 3 mega-blocks, where each mega-block is comprised of a large number
of relatively shallow parallel convolutional modules. These sub-blocks, which
we call towers, are independent, so they can be computed in parallel to reduce
inference latency. We further leverage a dropout technique on these parallel
blocks, which we refer to as tower dropout, to strengthen the independence of
each block inside a mega-block from other "parallel" blocks. We demonstrate
that this augmentation technique allows us to scale the network’s
computational requirements during inference without retraining the model. The
model also has a relatively small receptive field which makes it a good
candidate for streaming ASR.

Fig. 1: Architecture of CarneliNet-$C$. The model starts with a prologue
block, followed by 3 mega-blocks with, respectively, 5, 6, and 7 parallel
towers, and an epilogue block. Each mega-block starts with a 2x-downsampling
operation and then branches out into several independent towers with $R$
repeated residual blocks. Each tower has exactly $C$ channels. Towers finish
with a tower dropout operation which allows us to remove towers during
inference without significant harm to the accuracy. Tower outputs are summed
together.

##  2 Related work

Recent work on multi-stream CNNs [14] introduces a model that uses several
independent multi-resolution time-delay NNs (TDNNs). The TDNN-F model consists
of parallel streams of self-attention encoders, where each stream has 3 layers
of 1D convolutions with dilated kernels whose dilation rates are unique for a
given stream, followed by a self-attention layer and factorized feed-forward
layer. The final embedding layer concatenates the encoder output from each
stream and linearly projects the concatenated vector to the final embedding.
In the follow-up paper [15], the authors explored a deeper version of multi-
stream TDNN models with a large numbers of streams. A similar approach was
suggested in [16] which replaces the original QuartzNet 1D time-channel
separable convolution with multi-stream convolutions. Each stream has a unique
dilated stride on convolutional operations. Streams are combined through
channel-wise attention module, which calculates the attention weight of each
convolutional stream by spatial channel-wise pooling.

While our idea resembles this prior work, we emphasise a few notable
differences: information sharing between parallel feature extractors via
aggregation at the end of mega-blocks instead of fully independent streams,
uniform time-channel separable convolutions with a single kernel size instead
of dilated convolutions. This work also presents an analysis of the impact of
individual towers on the final result. Additionally, the absence of large
kernels and dilated convolutions results in a smaller receptive field than
other multi-stream approaches, making the model more suitable for streaming
ASR.

##  3 Model architecture

CarneliNet is a convolutional CTC model with several parallel towers
processing the same input and combining outputs at each stage. Each tower has
a Citrinet-like structure [17] consisting of repeated 1D time-channel
separable convolutions with 1D Squeeze-and-Excitation (SE) [18] context
modules. We do not experiment with the internals of these computational
blocks, instead focusing on analysis of their composition with respect to each
other. In this light, CarneliNet derives from Citrinet [17], QuartzNet [5] and
ContextNet [13] and reuses their building blocks in a unique way.

Compositionally, the model has 5 parts: a prologue block, 3 mega-blocks with,
respectively, 5, 6 and 7 parallel towers, and an epilogue block. All mega-
blocks start with a block that consists of several 1D time-channel separable
convolutions, the last of which has stride 2. The output of the down-sampling
operation is passed as an input to several towers that compute outputs
independently from each other, in parallel. Each tower consists of $R$
repeated QuartzNet blocks [5] with the same kernel size and finishes with an
SE module, similar to ContextNet [13]. The number of channels in each tower,
denoted as $C$, is kept constant for simplicity. As this number directly
controls computational power of the model as demonstrated in section 5, we
refer to different variants of the models as CarneliNet-C. The schematic of
the CarneliNet-C architecture is depicted in Figure 1.

In order to further simplify the design of the network, we choose to keep
kernel size the same across all towers. While it is possible to tune the
kernel size of each individual tower and gain an small improvement in word
error rate (WER), finding an exact configuration of kernel sizes requires a
significant number of experiments and computation. Additionally, we aim to
keep the receptive field small while maintaining comparable speech recognition
quality by favouring smaller kernel sizes. Based on the results of ablations
in section 5.1, we choose $11$ as a primary kernel size.

Tower outputs are added together to obtain a single output of the mega-block.
While different aggregation strategies are possible, we use a summation,
shifting computation to the towers themselves. We found that summation worked
no worse than a family of weighted per-output or per-channel summations with
or without dependence on the input.

Independent parallel computation of towers gives us the flexibility to choose
which outputs are going to contribute to the final result. To save
computational resources we can reduce the number of towers in any mega-block
by discarding a few of them from the computational graph after the full model
is trained. This allows us to dynamically change the computational complexity
of the model without additional training as we show in section 5.3.

###  3.1 Tower Dropout

To overcome the inevitable reduction in quality caused by removal of towers
during inference, we introduce tower dropout. The idea of tower dropout is
similar to Stochastic Depth [19] but dropout is applied to outputs of towers
before final summation in the mega-block. Dropping an output of a tower before
summation ignores the computations of all blocks in the tower, effectively
reducing the model width.

More formally, if we denote the computed tensor at the output of tower $i$ as
$O_{i}\in\mathbb{R}^{T\times C}$, where $T$ is number of output timestamps and
$C$ is the number of channels, then during training time the final output
$M\in\mathbb{R}^{T\times C}$ of a mega-block with $N$ towers is computed as
$M=\sum_{i=1}^{N}w_{i}O_{i}/p$, where $w_{i}\sim\operatorname{Bernoulli}(p)$
is a Bernoulli random variable with probability $p$ of being $1$. We use this
technique of scaling outputs by inverse survival probability in order to avoid
the rescaling during the inference step.

During inference we select the active towers by setting a respective mask
variable $\delta_{i}$ to $1$. Setting the variable $\delta_{i}$ to $0$ ignores
the output of $i$-th tower. Using these masks we obtain the final weights
$w_{i}=\delta_{i}/K$, where $K=\sum_{i=1}^{N}\delta_{i}$ is the number of kept
towers. Final weights are used directly to compute an inference output of
mega-block $M=\sum_{i=1}^{N}w_{i}O_{i}$. The re-scaling of mask variables is
necessary to keep the expected value of the output unchanged
$\mathbb{E}\left[M\right]=\sum_{i=1}^{N}w_{i}\mathbb{E}\left[O_{i}\right]=\sum_{i=1}^{N}\frac{\delta_{i}}{K}\mathbb{E}\left[O_{i}\right]=\mathbb{E}\left[O\right]\frac{1}{K}\sum_{i=1}^{N}\delta_{i}=\mathbb{E}\left[O\right]$,
assuming the expected value of each tower output
$\mathbb{E}\left[O_{i}\right]$ is the same and denoted as
$\mathbb{E}\left[O\right]$.

##  4 Experiments

###  4.1 LibriSpeech

We train a number of CarneliNets with various numbers of channels per tower on
the LibriSpeech (LS) dataset [20]. The LS training set was used to train a
word-piece tokenizer with 1024 subwords with the help of the Huggingface
library [21].

We used the NovoGrad optimizer [22] to train all the models. Experimentally we
found that a high initial learning rate (LR) of $0.1$ works best for this
relatively shallow model. The other parameters of the optimizer were set to
$\beta_{1}=0.8,\beta_{2}=0.25$, similar to Citrinet [17]. We utilize a cosine
annealing scheduler [23] with a final LR of $10^{-5}$ preceded by 1000 steps
of warmup during training. We also use dropout [24] with value $0.1$ for all
the models. SpecAugment [25] was used as the main data augmentation method
with $2$ frequency masks for all models, $2$ time masks for CarneliNet-256 and
CarneliNet-384, and $10$ time masks for larger models. The training of models
lasted $1000$ epochs and was performed on 32 V100 GPUs with a batch size of
32.

We also use two external language models (6-gram and transformer) to perform
beam rescoring. These models were trained on the LibriSpeech language model
training text combined with train transcripts.

The results are presented in Table 1. We compare them with two CTC-based
models Citrinet [17], a deep model from which CarneliNet’s computational sub-
networks were derived, and Multistream CNN [15], a recent model that employs
parallel streams of self-attention encoders with different dilation rates. We
also provide results for RNN-transducer [26] models, namely ContextNet [13]
and Conformer [12] for reference. We demonstrate that CarneliNet achieves
scores close to the state-of-the-art non-autoregressive CTC-based models while
being highly scalable.

Table 1: CarneliNet vs state-of-the-art CTC-based and RNNT-transducer-based
models, LibriSpeech, WER($\%$)

Model | LM | Test | Params,
---|---|---|---
clean | other | M
Citrinet-1024 [17] | - | 2.52 | 6.22 | 142
6-gram | 2.10 | 5.06
Transf | 2.00 | 4.69
Multistream CNN [15] | - | 3.52 | 8.32 | 93.9
4-gram | 2.80 | 7.06
SRU | 1.75 | 4.46
ContextNet [13] | - | 2.1 | 4.6 | 112.7
LSTM | 1.9 | 4.1
Conformer [12] | - | 2.1 | 4.3 | 118.8
LSTM | 1.9 | 3.9
CarneliNet-256 | - | 4.52 | 11.59 | 10
6-gram | 3.30 | 8.41
Transf | 2.77 | 7.35
CarneliNet-384 | - | 3.60 | 9.37 | 21.1
6-gram | 2.73 | 7.11
Transf | 2.48 | 6.34
CarneliNet-512 | - | 3.62 | 8.90 | 36.5
6-gram | 2.81 | 6.82
Transf | 2.51 | 6.06
CarneliNet-1024 | - | 2.94 | 7.16 | 142
6-gram | 2.45 | 5.87
Transf | 2.32 | 5.31

###  4.2 MLS

We train the largest model, CarneliNet-1024, on the English part of the MLS
dataset [27] which includes 44K hours of English read-speech speech. Following
[17], we filter out training examples with typos and misalignments in
transcripts, utterances in Spanish and Russian from English portion of MLS dev
and test sets and transcripts with periods from test set. We also replace
hyphens with space characters to keep consistency with the LibriSpeech
dataset. The model hyper-parameters remained the same as for LibriSpeech with
the exception of reducing the learning rate to $0.05$ and using a 1024
sentence piece tokenizer [28] built on the MLS English training set corpus.
The model was trained for 100 epochs on 256 GPUs with batch size 32 per
device. We show the results in Table 2. The shallower CarneliNet performs on
par with its much deeper counterpart Citrinet.

Table 2: CarneliNet-1024 trained on normalized MLS English, evaluated on
normalized MLS English and LibriSpeech-other, WER($\%$)

Model | LM | LS-other | MLS | Params,
---|---|---|---|---
dev | test | dev | test | M
Citrinet-1024[17] | - | 5.79 | 5.69 | 6.99 | 8.46 | 142
6-gram | 4.72 | 4.83 | 5.76 | 6.79
Transf | 4.41 | 4.62 | 5.44 | 6.39
CarneliNet-1024 | - | 5.86 | 5.73 | 6.98 | 8.32 | 142
6-gram | 4.97 | 5.04 | 5.89 | 7.02
Transf | 4.74 | 4.86 | 5.60 | 6.68

###  4.3 AISHELL-2

We trained a CarneliNet-1024 model on AISHELL-2 [29], a Mandarin corpus with
1000 hours of training speech. Unlike the English language models, we used
character level tokenization instead of sentence piece tokenization for
Mandarin ASR. The model was trained from scratch using 64 V100 GPUs for 400
epochs with all the other parameters same as the ones used for LibriSpeech in
section 4.1, except for the initial LR which was set at $0.05$. For beam
rescoring, we used a 4-gram LM trained on the training corpus of AISHELL-2
with the beam width 128.

In Table 3 we compare the results with other strong models. CarneliNet
produces competitive scores with just greedy decoding compared to the
autoregressive Continuous Integrate-and-Fire (CIF) [30] model that relies on
frame-by-frame computation of integration weights and with a two pass
rescoring U2++ Conformer [31]. Compared to non-autoregressive counterparts,
such as NAR-Transformer [32], LASO [33] and QuartzNet, CarneliNet achieves the
best CER score on this dataset.

Table 3: AISHELL-2: Carnelinet-1024 trained from scratch, CER($\%$)

Model | LM | iOS | Android | mic
---|---|---|---|---
U2++ Conformer[31] | 0-LM | 6.70 | - | -
Attention | 5.50 | - | -
CIF [30, 34] | SAN | 5.68 | 6.09 | 6.20
QuartzNet [35] | - | 6.96 | - | -
NAR-Transformer [32] | - | 7.1 | 8.0 | 8.1
LASO-large w/ BERT [33] | - | 6.5 | 7.2 | 7.1
Carnelinet-1024 | - | 6.2 | 6.45 | 6.48
4-LM | 5.8 | 6.07 | 6.05

##  5 Ablation study

We evaluate the contribution of each component of the model to the final
accuracy by performing extensive ablation studies. Specifically, we choose
CarneliNet-$384$ with kernel size $11$ as a baseline model. All experiments
are performed on the LibriSpeech dataset. Models were trained for $400$ epochs
with exactly the same parameters as in subsection 4.1. The hardware setup is
also identical with 32 V100 GPUs, each processing a batch of size 32.

###  5.1 Kernel configurations

We first evaluate the effect of kernel size on the final quality of the model.
We train models with odd kernel sizes from $3$ to $23$ and evaluate the
performance on dev and test portions of LibriSpeech datasets. The best
results, as shown in Table 4, are observed with larger kernel sizes as the
number of parameters and receptive field naturally grow. However, the
improvement in recognition accuracy is negligible beyond kernel size of $11$.
Smaller kernels size provides us with the best trade-off between WER and
receptive field size for potential streaming applications.

Table 4: Accuracy vs kernel size. CarneliNet-384, LibriSpeech, greedy
WER($\%$).

Kernel | dev | test | Params,
---|---|---|---
size | clean | other | clean | other | M
3 | 3.87 | 10.52 | 4.10 | 10.46 | 20.7
5 | 3.72 | 9.95 | 3.84 | 9.96 | 20.7
7 | 3.66 | 9.72 | 3.81 | 9.83 | 20.8
9 | 3.60 | 9.95 | 3.88 | 9.79 | 20.9
11 | 3.49 | 9.56 | 3.70 | 9.32 | 21.0
13 | 3.38 | 9.65 | 3.71 | 9.47 | 21.0
15 | 3.47 | 9.46 | 3.70 | 9.35 | 21.1
17 | 3.40 | 9.49 | 3.74 | 9.51 | 21.2
19 | 3.45 | 9.44 | 3.66 | 9.47 | 21.2
21 | 3.45 | 9.36 | 3.64 | 9.38 | 21.3
23 | 3.46 | 9.45 | 3.70 | 9.60 | 21.4

###  5.2 Wide vs Deep

To measure the effect of network widening we compare Carnelinet against its
deep counterpart Citrinet. We also train a deep baseline with unified kernel
size of 11. This deep model is essentially a Citrinet with all kernel sizes
switched to $11$, to which we refer as Citrinet, ks=$11$. For all models we
fix the number of channels $C$ to $384$. From Table 5 it is apparent that the
deeper model with fine-tuned kernels shows a better performance, as supported
by section 4.1. However, the convergence and accuracy of the deep version is
quite sensitive to exact kernel size parameters and parameter unification
results in poorer performance. In addition to architecture parameter
stability, we also observed much better stability to change of training hyper-
parameters for Carnelinet.

Table 5: Wide vs Deep architectures (CarneliNet vs Citrinet). LibriSpeech,
greedy WER($\%$).

Configuration | dev | test
---|---|---
clean | other | clean | other
CarneliNet-384 | 3.49 | 9.56 | 3.70 | 9.32
Citrinet-384 | 3.49 | 9.31 | 3.62 | 9.28
Citrinet-384, ks=11 | 3.61 | 9.58 | 3.78 | 9.69

###  5.3 Tower dropout

In this section we investigate how tower dropout regularization is necessary
for ensuring that ASR accuracy does not get significantly worse when model is
dynamically reconfigured during inference. For this, we compare the results of
a model trained with no tower dropout (train TD = $0.0$) with the results of
models that were trained with tower dropout values (train TD =
$\\{0.1,0.2,0.5\\}$). We vary the number of towers removed during inference
and whether that number of towers is removed from the first mega-block only,
the last mega-block only, or all three blocks at once. Additionally, to
quantify the trade-off between obtaining a reduced model from a larger one
trained with tower dropout versus training a new model with a smaller number
of towers, we provide the results for reduced models trained from scratch.

The results are shown in Table 6. If at least $1$ tower is removed, the
regularized models perform much better than the baseline model trained with no
tower dropout. If no towers are removed, this regularization introduces a
small degradation to WER for smaller values of tower dropout. Larger values of
tower dropout are only beneficial in cases when most towers are removed and
introduce a considerable increase in WER when evaluated in other settings. For
such extreme cases it is better to train a reduced model from scratch.

Table 6: Accuracy of models trained with different values of Tower Dropout
(TD) as function of the number of towers removed at inference time. We removed
towers from different mega-blocks. Models with TD are compared to the model
with smaller number of towers trained from scratch. We also provide the number
of parameters (Params, M) for the models with removed towers. CarneliNet-384,
LibriSpeech, test-other, greedy WER($\%$).

Mega |  | Num. of towers removed during inference
---|---|---
block | 0 | 1 | 2 | 3 | 4
first | Params, M | 21.0 | 19.7 | 18.8 | 17.9 | 17.0
TD=0.0 | 9.32 | 11.09 | 15.43 | 24.17 | 54.43
TD=0.1 | 9.70 | 10.10 | 10.24 | 11.61 | 14.84
TD=0.2 | 10.03 | 10.44 | 10.50 | 11.11 | 13.47
TD=0.5 | 12.35 | 12.37 | 12.56 | 12.62 | 13.10
from scratch | 9.32 | 9.75 | 9.71 | 9.87 | 10.29
last | Params, M | 21.0 | 19.7 | 18.8 | 17.9 | 17.0
TD=0.0 | 9.32 | 14.14 | 17.21 | 20.75 | 25.51
TD=0.1 | 9.70 | 9.95 | 10.34 | 11.16 | 12.83
TD=0.2 | 10.03 | 10.11 | 10.36 | 11.12 | 11.73
TD=0.5 | 12.35 | 12.43 | 12.53 | 12.81 | 13.03
from scratch | 9.32 | 9.70 | 10.06 | 10.10 | 10.12
all | Params, M | 21.1 | 18.2 | 15.3 | 12.4 | 9.6
TD=0.0 | 9.32 | 16.73 | 29.24 | 47.10 | 79.09
TD=0.1 | 9.70 | 10.70 | 11.86 | 15.14 | 23.80
TD=0.2 | 10.03 | 10.75 | 11.53 | 13.49 | 19.94
TD=0.5 | 12.35 | 12.53 | 12.88 | 13.34 | 14.25
from scratch | 9.32 | 9.85 | 10.06 | 10.74 | 11.60

###  5.4 Weight scaling for model reconfiguration

The tower dropout during training allows us to obtain models that are robust
to dynamic disabling of towers during inference. As described in section 3 if
any of the sub-networks are turned off, then we adjust the summation weight
factor to the rate of enabled ones to keep the mathematical expectation of the
output unchanged, much like in traditional dropout.

In Table 7 we illustrate the importance of scaling the summation weight by
comparing the results to the same model that does not re-scale weights. The
quality of speech recognition is universally worse for the non-scaling cases.
When changes are major, e.g. when removing towers from all layers, the quality
degrades catastrophically without rescaling.

Table 7: Importance of summation weight scaling after model reconfiguration.
CarneliNet-384, LibriSpeech, test-other set, greedy WER($\%$).

Mega | Scaling? | Number of removed towers
---|---|---
block | 1 | 2 | 3 | 4
first | ✗ | 10.45 | 11.16 | 13.53 | 25.63
✓ | 10.44 | 10.50 | 11.11 | 13.47
last | ✗ | 10.21 | 10.64 | 12.02 | 15.35
✓ | 10.11 | 10.36 | 11.12 | 11.73
all | ✗ | 11.09 | 15.11 | 43.57 | 99.55
✓ | 10.75 | 11.53 | 13.49 | 19.94

###  5.5 Width scaling

The number of channels ($C$) has the single largest impact on the number of
trainable parameters and on computational requirements. We also expect that
larger models will perform better. To test this hypothesis we train models
with different numbers of channels $C\in\\{256,384,512,768,1024\\}$. The
obtained results are shown in Table 8. We observe a noticeable impact of model
width on the quality of speech recognition. While extremely wide models with
$C=1024$ are able to add additional improvements to the resulting scores,
there exists an upper bound to such scaling, beyond which further improvements
plateau.

Table 8: Accuracy vs tower width ($C$ \- number of channels in each tower).
LibriSpeech, greedy WER($\%$).

C | dev | test | Params,
---|---|---|---
clean | other | clean | other | M
256 | 4.51 | 12.04 | 4.59 | 11.94 | 9.9
384 | 3.49 | 9.56 | 3.70 | 9.32 | 21.0
512 | 3.49 | 9.26 | 3.64 | 9.10 | 36.3
768 | 2.96 | 8.00 | 3.24 | 7.79 | 80.8
1024 | 2.70 | 7.42 | 3.11 | 7.16 | 141

###  5.6 Depth scaling

We analyze the dependence of the network on the number of repeated blocks $R$
in each tower. The number of blocks is kept the same across all towers, so we
can vary only one parameter. Results presented in Table 9 confirm our
expectations that deeper models perform better at the expense of more
parameters and slower training and inference speed. We also observe an upper
bound when scaling depth, beyond which improvements stagnate, which we find to
be when depth $R$ reaches $7$, suggesting that additional architecture changes
such as Residual connections between blocks are needed in this setting.

Table 9: Accuracy vs Tower depth ($R$ \- the number of blocks per tower).
CarneliNet-384. LibriSpeech, greedy WER($\%$).

R | dev | test | Params,
---|---|---|---
clean | other | clean | other | M
2 | 5.77 | 15.09 | 5.83 | 14.95 | 11.4
3 | 4.30 | 11.88 | 4.65 | 11.83 | 14.6
4 | 4.00 | 10.40 | 3.99 | 10.55 | 17.8
5 | 3.49 | 9.56 | 3.70 | 9.32 | 21.0
6 | 3.38 | 9.26 | 3.60 | 9.25 | 24.2
7 | 3.28 | 8.78 | 3.46 | 8.83 | 27.4
8 | 3.32 | 9.17 | 3.57 | 9.22 | 30.6
9 | 3.46 | 9.46 | 3.68 | 9.37 | 33.8
10 | 3.46 | 9.32 | 3.72 | 9.17 | 36.9

###  5.7 Parallel QuartzNet

In order to demonstrate that a mixture model can be successfully applied not
only to the Citrinet model but to various models in general, we perform a
similar transformation to QuartzNet-15x5 [5]. Since QuartzNet contains blocks
with different kernel sizes and number of filters, the grouping of
computational sub-networks can be done in multiple ways. For example, one can
choose to put kernels with the same number of filters in the same mega-block.
However, we found that grouping by the same kernel size gives the best
results. We trained this model (Parallel QuartzNet) on LibriSpeech, with most
hyper-parameters borrowed from the original QuartzNet in the paper. Due to the
increased width, we added dropout of $0.1$, performed strong augmentation with
SpecAugment [25], and raised initial LR to $0.1$.

Table 10 shows the results of the trained models evaluated on LibriSpeech.
While there is a gap of $1.6\%$ in WER score, we note that the parallel
version still successfully trains and comes with benefits of dynamic
reconfiguration during inference without retraining. This demonstrates that
the process of constructing a mixture model can be performed for different
sub-networks and indicates an opportunity for experiments on different
architectures.

Table 10: Deep QuartzNet-15x5 vs Parallel QuartzNet with blocks grouped by
kernel size. LibriSpeech, 400 epochs, greedy WER($\%$).

Configuration | dev | test
---|---|---
clean | other | clean | other
QuartzNet | 3.83 | 11.08 | 3.90 | 11.28
Parallel QuartzNet | 5.17 | 12.61 | 5.13 | 12.84

##  6 Conclusions

In this work we discussed the implementation and advantages of a parallel
mixture of shallow networks for ASR. We illustrated the effectiveness of such
composition with a novel CTC-based architecture CarneliNet composed of three
mega-blocks with parallel sub-networks based on 1D depthwise-separable
convolutions. We evaluated the model on LibriSpeech, MLS and AISHELL-2
datasets and achieved close to state-of-the-art results for non-autoregressive
CTC-based models on LibriSpeech and MLS and state-of-the-art results for non-
autoregressive models on AISHELL-2. We demonstrated how such a model with
parallel sub-networks can be dynamically reconfigured without retraining to
adjust to computation requirements. We believe that this approach can be
transferred to additional architectures, as supported by the Parallel
QuartzNet experiments.

The models and training recipes have been released in the NeMo toolkit
[36].111https://github.com/NVIDIA/NeMo

##  7 Acknowledgments

The authors thank Elena Rastorgueva, Jocelyn Huang for extensive review of the
manuscript and the whole NVIDIA AI Applications team for their helpful
feedback and review.

## References

  * [1] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coate, and A. Ng,  “Deep Speech: Scaling up end-to-end speech recognition,”  arXiv:1412.5567, 2014.
  * [2] A. Graves and N. Jaitly,  “Towards end-to-end speech recognition with recurrent neural networks,”  ICML, 2014.
  * [3] R. Collobert, C. Puhrsch, and G. Synnaeve,  “Wav2letter: an end-to-end convnet-based speech recognition system,”  arXiv:1609.03193, 2016.
  * [4] J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J.M. Cohen, H. Nguyen, and R.T. Gadde,  “Jasper: An end-to-end convolutional neural acoustic model,”  arXiv:1904.03288, 2019.
  * [5] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang,  “QuartzNet: Deep automatic speech recognition with 1d time-channel separable convolutions,”  in ICASSP, 2020.
  * [6] A. Hannun, A. Lee, Q. Xu, and R. Collobert,  “Sequence-to-sequence speech recognition with time-depth separable convolutions,”  in Interspeech, 2019.
  * [7] L. Dong, S. Xu, and B. Xu,  “Speech-Transformer: A no-recurrence sequence-to-sequence model for speech recognition,”  in ICASSP, 2018.
  * [8] E. Tsunoo, Y. Kashiwagi, T. Kumakura, and S. Watanabe,  “Transformer ASR with contextual block processing,”  in ASRU, 2019.
  * [9] A. Mohamed, D. Okhonko, and L. Zettlemoyer,  “Transformers with convolutional context for ASR,”  arXiv:1904.11660, 2019.
  * [10] E. Tsunoo, Y. Kashiwagi, T. Kumakura, and S. Watanabe,  “Towards online end-to-end transformer automatic speech recognition,”  arXiv:1910.11871, 2019.
  * [11] C.-F. Yeh, J. Mahadeokar, K. Kalgaonkar, Y. Wang, D. Le, M. Jain, K. Schubert, C. Fuegen, and M. L. Seltzer,  “Transformer-Transducer: End-to-end speech recognition with self-attention,”  arXiv:1910.12977, 2019.
  * [12] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv:2005.08100, 2020.
  * [13] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu,  “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,”  arXiv:2005.03191, 2020.
  * [14] K. J. Han, R. Prieto, K. Wu, and T. Ma,  “State-of-the-art speech recognition using multi-stream self-attention with dilated 1d convolutions,”  ASRU, 2019.
  * [15] K. J. Han, J. Pan, V. Tadala, T. Ma, and D. Povey,  “Multistream CNN for robust acoustic modeling,”  in ICASSP, 2021.
  * [16] J. Luo, J. Wang, N. Cheng, G. Jiang, and J. Xiao,  “Multi-QuartzNet: Multi-resolution convolution for speech recognition with multi-layer feature fusion,”  in SLT, 2021.
  * [17] S. Majumdar, J. Balam, O. Hrinchuk, V. Lavrukhin, V. Noroozi, and B. Ginsburg,  “Citrinet: Closing the gap between non-autoregressive and autoregressive end-to-end models for automatic speech recognition,”  arXiv:2104.01721, 2021.
  * [18] J. Hu, L. Shen, and G. Sun,  “Squeeze-and-excitation networks,”  in ICVPR, 2018.
  * [19] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger,  “Deep networks with stochastic depth,”  in European conference on computer vision, 2016.
  * [20] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “LibriSpeech: an ASR corpus based on public domain audio books,”  in ICASSP, 2015.
  * [21] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush,  “HuggingFace’s Transformers: State-of-the-art natural language processing,”  arXiv:1910.03771, 2020.
  * [22] B. Ginsburg, P. Castonguay, O. Hrinchuk, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, H. Nguyen, and Cohen J. M.,  “Stochastic gradient methods with layer-wise adaptive moments for training of deep networks,”  arXiv:1905.11286, 2019.
  * [23] I. Loshchilov and F. Hutter,  “Sgdr: Stochastic gradient descent with warm restarts,”  arXiv:1608.03983, 2016.
  * [24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,  “Dropout: A simple way to prevent neural networks from overfitting,”  Journal of Machine Learning Research, 2014.
  * [25] D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E.D. Cubuk, and Q.V. Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  arXiv:1904.08779, 2019.
  * [26] A. Graves,  “Sequence transduction with recurrent neural networks,”  arXiv:1211.3711, 2012.
  * [27] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert,  “MLS: A large-scale multilingual dataset for speech research,”  Interspeech, 2020.
  * [28] T. Kudo and J. Richardson,  “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  arXiv:1808.06226, 2018.
  * [29] J. Du, X. Na, X. Liu, and H. Bu,  “Aishell-2: Transforming mandarin ASR research into industrial scale,”  arXiv:1808.10583, 2018.
  * [30] L. Dong and B. Xu,  “CIF: Continuous integrate-and-fire for end-to-end speech recognition,”  in ICASSP, 2020.
  * [31] D. Wu, B. Zhang, C. Yang, Z. Peng, W. Xia, X. Chen, and X. Lei,  “U2++: Unified two-pass bidirectional end-to-end model for speech recognition,”  arXiv:2106.05642, 2021.
  * [32] X. Song, Z. Wu, Y. Huang, C. Weng, D. Su, and H. Meng,  “Non-autoregressive transformer asr with ctc-enhanced decoder input,”  in ICASSP, 2021.
  * [33] Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen, and S. Zhang,  “Fast end-to-end speech recognition via non-autoregressive models and cross-modal knowledge transferring from BERT,”  arxiv:2102.07594, 2021.
  * [34] L. Dong, C. Yi, J. Wang, S. Zhou, S. Xu, X. Jia, and B. Xu,  “A comparison of label-synchronous and frame-synchronous end-to-end models for speech recognition,”  arxiv:2005.10113, 2020.
  * [35] J. Balam, J. Huang, V. Lavrukhin, S. Deng, S. Majumdar, and B. Ginsburg,  “Improving noise robustness of an end-to-end neural model for automatic speech recognition,”  arXiv:2010.12715, 2020.
  * [36] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook, et al.,  “NeMo: a toolkit for building AI applications using neural modules,”  arXiv:1909.09577, 2019.