# CLIP: Train Faster with Less Data

Muhammad Asif Khan
Qatar Mobility Innovations Center (QMIC)
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
&Ridha Hamila
Department of Electrical Engineering
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
&Hamid Menouar
Qatar Mobility Innovations Center (QMIC)
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Deep learning models require an enormous amount of data for training. However,
recently there is a shift in machine learning from model-centric to data-
centric approaches. In data-centric approaches, the focus is to refine and
improve the quality of the data to improve the learning performance of the
models rather than redesigning model architectures. In this paper, we propose
CLIP i.e., Curriculum Learning with Iterative data Pruning. CLIP combines two
data-centric approaches i.e., curriculum learning and dataset pruning to
improve the model learning accuracy and convergence speed. The proposed scheme
applies loss-aware dataset pruning to iteratively remove the least significant
samples and progressively reduces the size of the effective dataset in the
curriculum learning training. Extensive experiments performed on crowd density
estimation models validate the notion behind combining the two approaches by
reducing the convergence time and improving generalization. To our knowledge,
the idea of data pruning as an embedded process in curriculum learning is
novel.

##  1 Introduction

Deep learning models often require a huge amount of data to train and thus to
better generalize to unseen real-world examples. For instance, the state-of-
the-art (SOTA) classification models e.g., VGG Simonyan and Zisserman (2015),
ResNet He et al. (2016), GoogleNet Szegedy et al. (2015) are all trained on
the ImageNet dataset Russakovsky et al. (2015) which has around 14,197,122
images (at the time of writing this paper). Although the huge amount of
training data generally improves the learning accuracy, not all the training
samples contribute to the training performance. In fact, it is more the
quality of data that plays a major role in training. More particularly, the
distribution of training data and annotation errors play can significantly
affect training and generalization performance.

Recently, data-centric approaches are advocated by machine learning (ML)
experts as the new paradigm for training ML models. In essence, well-crafted
data techniques can help ML models to learn better and faster. We leverage two
such data-centric approaches for training ML models namely curriculum learning
(CL) and dataset pruning (or data pruning).

Curriculum learning is a training strategy in ML in which the training data is
organized (rather than random) and exposed to the model at a predefined pace
(rather than the whole data in a single training epoch). CL is inspired by
human learning behavior i.e., humans learn better when using a curriculum to
organize content in the order of increasing difficulty. The idea of CL was
first introduced long ago in Elman (1993) and formalized in Bengio et al.
(2009). CL has been applied in several ML tasks such as object detection and
localization Tang et al. (2018); Sangineto et al. (2019), machine translation
Platanios et al. (2019); Wang et al. (2019) and reinforcement learning
Narvekar et al. (2020).

CL aims that the model to converge faster and better generalize by exposing
the model to train data in a controlled manner using the difficulty scores of
training samples. However, in practice, not all training samples contribute to
the training. Intuitively, such samples can be eliminated from the training
data. This process of eliminating the least significant training samples from
a dataset is called dataset pruning. The effectiveness of data pruning has
been investigated in Li et al. (2018a); Yang et al. (2022); Paul et al.
(2021).

In this paper, we propose an effective training strategy termed "CLIP" based
on CL with dataset pruning. In CLIP, we start by exposing an ML model to a
subset of training data and increase the training data according to a pre-
defined pacing function. During the training iterations, the size of the
training subset is pruned by eliminating the least-contributing samples. This
effectively makes the size of the dataset smaller as the training continues.
By combining CL with data pruning, the model converges faster and generalizes
better.

We investigate the proposed scheme (CLIP) in the crowd counting problem which
is a hot topic in computer vision Khan et al. (2022). Crowd counting is a non-
trivial problem with applications in surveillance for smart policing in public
places, situational awareness during disasters Sambolek and Ivasic-Kos (2021),
traffic monitoring, and wildlife monitoring Chandana and Vasavi (2022).
Traditionally crowd counting in images employed handcrafted local features
such as body parts Topkaya et al. (2014), shapes Lin and Davis (2010),
textures Chen et al. (2012a), edges Wu and Nevatia (2006), foreground and
gradients Tian et al. (2010). These methods perform poorly on images of dense
crowds due to severe occlusions and scale variations. To overcome these
challenges, CNN-based crowd counting has been introduced in Zhang et al.
(2015); Boominathan et al. (2016). Several state-of-the-art CNN models are
developed over the course of time mainly to improve the accuracy in more
challenging scenes Zhang et al. (2016); Zeng et al. (2017); Li et al. (2018b);
Jiang et al. (2019); Cao et al. (2018); Song et al. (2021); Gu and Lian
(2022); Wang and Breckon (2022). These SOTA crowd counting models are mainly
focused on proposing model architectures with little focus on data. In CLIP,
we aim to improve the performance of existing models (without any
modifications to the architecture) using data-centric techniques.

The contribution of the paper is as follows: We propose an efficient training
strategy combining curriculum learning and dataset pruning termed as CLIP.
CLIP effectively speeds up the model convergence and also improves model
generalization to unseen data at test time. We extensively evaluated CLIP in
crowd density estimation tasks using well-known crowd counting models. The
results are compared against the standard training used in the original
models. To our knowledge, the use of dataset pruning in curriculum learning
has not been proposed earlier. Also, the use of CL and data pruning alone or
together in crowd counting tasks is a new direction to improve the performance
of existing crowd counting models.

##  2 Related Work

The SOTA models for crowd counting and density estimation are using a
convolution neural network (CNN). The first CNN-based crowd counting model was
proposed in Zhang et al. (2015). The CrowdCNN model Zhang et al. (2015) is a
single-column CNN network having six convolution layers. Following the
approach, there has been a long list of CNN-based model of different types
including multi-column networks Zhang et al. (2016); Boominathan et al.
(2016); Sam et al. (2017); Sindagi and Patel (2017), pyramid networks Zeng et
al. (2017); Cao et al. (2018); Wang and Breckon (2022), encoder-decoder models
Jiang et al. (2019); Gao et al. (2019), and transfer learning based models Li
et al. (2018b); Liu et al. (2019); Aich and Stavness (2018); Tang et al.
(2022). The best-performing models today are those using transfer learning
which uses a pretrained image classification model such as VGG Simonyan and
Zisserman (2015), ResNet He et al. (2016) or Inception Szegedy et al. (2015)
as a front-end to extract features and then a small CNN network uses these
features to estimate the crowd density. However, typically these models are
very large requiring a long time to train and converge.

Curriculum learning has not been applied explicitly in crowd counting tasks.
However, it has been evaluated in other computer vision tasks. A notable work
on CL in Hacohen and Weinshall (2019) investigated the potential benefits of
CL in the classification problem using CIFAR10 and CIFAR100 datasets. Their
results show that CL brings significant accuracy improvement. The work also
shows that different pacing functions converge to similar final performances.
A recent study on the CL Wu et al. (2021) with an extensive set of experiments
shows that CL significantly reduces the convergence time while improving the
learning performance. The results also show that it is actually the pacing
function rather than curricula (alone) that improves learning performance.

The impact of data pruning and its potential benefits are investigated in Yang
et al. (2022); Paul et al. (2021). In Yang et al. (2022), authors propose data
pruning under generalization constraint. In this method, the redundant
training samples are identified and removed based on the loss function. The
model accuracy using the pruned dataset (with fewer samples than the original
dataset) is comparable (slightly less) to the model trained on the whole
dataset. In Paul et al. (2021), authors used two types of scores (gradient
normed and L2-norm) to identify the least significant samples during the first
few epochs of the training phase and prune them to get a compact dataset for
the remaining training phase. The method shows that better accuracy was
achieved on the CIFAR10 dataset even after removing half of the training
samples.

##  3 Proposed Scheme

The proposed scheme i.e., is implemented by combining (i) curriculum learning
and (ii) data pruning.

Curriculum learning has two components i.e., a scoring function and a pacing
function. A pre-training model is used to calculate per-sample loss that
serves as the sample scores. We used a CSRNet Li et al. (2018b) to calculate
per-sample losses that serve as sample scores (difficulties). A pacing
function is then used to initially expose the model to a small subset of data
and the size of the subset is iteratively increased. We used two different
pacing functions (linear and quadratic) (Fig. 1) in our study however, due to
very close results, we are reporting results for the latter function only.

The proposed scheme (CLIP) additionally applies data pruning i.e., removing
the least significant samples from the training subsets during the training.
To implement data pruning in CLIP, we first empirically found the loss value
of each sample of the original training dataset which showed that different
samples contribute differently in the learning process with some samples
having very little contribution. Thus, it is intuitive that such samples can
be eliminated without significantly impacting the model’s average training
loss. The per-sample losses of different datasets can be observed in Fig. 2.

To integrate data pruning in the curriculum learning process, we define
additional parameters $\epsilon$. The value of $epsilon$ can be either fixed
or can be calculated using an increasing function with decreasing rate (or
vice versa depending on the size of the dataset). Algorithm 1 illustrates the
CLIP procedure.

(a) Linear pacing function (Solid lines represent full data and broken lines
represent pruned data). (b) Quadratic pacing function (Solid lines represent
full data and broken lines represent pruned data).

Figure 1: Illustration of two pacing functions in curriculum learning with
dataset pruning ($\epsilon=0.05$).

Require: scoring function $f$, pacing function $g$, data $X$, sample
elimination ratio $\epsilon$

Result: mini-batches [$B_{1},B_{2},...B_{M}$]

1

2$results$ = sort $X$ using $f$

3 for  _$i=1,\cdots M$_ do

4 $size\leftarrow g(i)$

5 $size=size-\epsilon(size)$

6 $X_{i}=X[1,...,size]$

7 uniformly sample $B_{i}$ to $results$

8 append $B_{i}$ to $result$

9

10 end for

return _$results$_

Algorithm 1 CLIP - Curriculum Learning with Iterative dataset Pruning.

(a) Mall dataset (b) ShanghaiTech Part-A (c) ShanghaiTech Part-B

Figure 2: Per-sample losses of ShanghaiTech part-A (top) and ShanghaiTech Part
B (bottom) dataset using various models.

###  3.1 Model Training

We choose three models i.e., MCNN Zhang et al. (2016), CSRNet Li et al.
(2018b) and CSRNet_lite Khan et al. (2023) and three different datasets i.e.,
ShanghaiTech Part-A and Part-B, Mall Chen et al. (2012b), and CARPK Hsieh et
al. (2017). The original labels are in the form of dot annotations which are
used to create density maps that serve as ground truth for the images. A
density map is generated by convolving a delta function $\delta(x-x_{i})$ with
a Gaussian kernel $G_{\sigma}$, where $x_{i}$ is a pixel containing the head
position.

$D=\sum_{i=1}^{N}{\delta(x-x_{i})*G_{\sigma}}$ (1)

where, $N$ denotes the total number of annotated points (i.e., the total count
of heads) in the image. We empirically determined a fixed value of $\sigma$
that provides a good estimation of the head sizes for each individual dataset.
To prevent overfitting, we use standard data augmentation techniques such as
horizontal flipping, and random brightness and contrast. We use Adam optimizer
Kingma and Ba (2015) with a learning rate of $0.0001$. The loss function used
is the standard Euclidean distance between the target and predicted density
maps which are defined in Eq. 2.

$L(\Theta)=\frac{1}{N}\sum_{1}^{N}{||D(X_{i};\Theta)-D_{i}^{gt}||_{2}^{2}}$
(2)

where $N$ is the number of samples in training data, $D(X_{i};\Theta)$ is the
predicted density map with parameters $\Theta$ for the input image $X_{i}$,
and $D_{i}^{gt}$ is the ground truth density map.

To implement data pruning, we used a fixed small value of $\epsilon=0.05$ such
that it eliminates a small number of least significant samples from the
training subset in each iteration until the training data size reaches the
maximum (original dataset size - total samples removed).

##  4 Evaluation and Results

We first investigated the effectiveness of the proposed scheme. The loss curve
in Fig. 3 shows a clear advantage of CLIP over standard training. The training
loss drops very quickly in CLIP with much fewer samples fed to the model as
compared to the standard training. This implies the correctness of our
intuition which is further investigated with an extensive set of experiments
on several datasets.

###  4.1 Evaluation Metrics

We evaluated the performance of CLIP using four widely used metrics i.e., Mean
Absolute Error (MAE) and Grid Average Mean Error (GAME), Structural Similarity
Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR). The first two metrics
evaluate the counting accuracy of the model whereas the latter two evaluate
the quality of the predicted density maps.

MAE and GAME can be calculated using the following Eq. 3 and Eq. 4:

$MAE=\frac{1}{N}\sum_{1}^{N}{(e_{n}-\hat{g_{n}})}$ (3)

where, $N$ is the size of the dataset, $g_{n}$ is the target or label (actual
count) and ${e_{n}}$ is the prediction (estimated count) in the $n^{th}$ crowd
image.

$GAME=\frac{1}{N}\sum_{n=1}^{N}{(\sum_{l=1}^{4^{L}}{|e_{n}^{l}-g_{n}^{l}|)}}$
(4)

The value of $L=4$ denoted that each density map is divided into a $4\times 4$
grid size with a total of $16$ patches.

The SSIM and PSNR metrics can be calculated using the below Eq. 5 and Eq. 6:

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{x}\sigma_{y}C_{2})}{(\mu_{z}^{2}\mu_{y}^{2}+C_{1})(\mu_{z}^{2}\mu_{y}^{2}+C_{2})}$
(5)

where $\mu_{x},\mu_{y},\sigma_{x},\sigma_{y}$ denotes the means and standard
deviations of the labels (density maps) and predictions (density maps),
respectively.

$PSNR=10log_{10}\left(\frac{Max(I^{2})}{MSE}\right)$ (6)

where $Max(I^{2})$ is the maximal pixel value in the image, e.g., 255 if pixel
values are stored as 8-bit unsigned integer type.

Figure 3: Loss convergence comparison. The bottom X-axis shows the number of
samples used in standard training. The top X-axis shows the number of samples
used in CLIP. The number of samples increases from left to right. The model
loss decreases from left to right.

The results for all the four metrics on ShanghaiTech Part-B and ShanghaiTech
Part-A datasets are presented in Fig. 4 and 5 showing improvement using CLIP
as follows: On ShanghaiTech Part-B, CLIP reduces the MAE (as compared to
standard training) by $26.4\Rightarrow 20.2\;(23\%\;\text{reduction})$ using
MCNN Zhang et al. (2016), $10.6\Rightarrow 8.2\;(22\%\;\text{reduction})$
using CSRNet Li et al. (2018b), and $9.6\Rightarrow
8.2\;(14\%\;\text{reduction})$ using CSRNet_lite Khan et al. (2023). On
ShanghaiTech Part-A, CLIP reduces the MAE by $110.2\Rightarrow
102.3\;(7\%\;\text{reduction})$ using MCNN Zhang et al. (2016),
$68.2\Rightarrow 65.2\;(4\%\;\text{reduction})$ using CSRNet Li et al.
(2018b), and $66.4\Rightarrow 63.3\;(4\%\;\text{reduction})$ using
CSRNet_lite. A similar performance was achieved using the GAME metric as well.
Although there is a clear gain in accuracy over both datasets using three
different models, the actual advantage of CLIP over standard training is
achieving higher accuracy with faster training and less number of training
samples as depicted in Fig. 3. The reader is encouraged to notice the minor
improvements over SSIM and PSNR values in Fig. 4 and 5 which represent the
quality of the predicted density maps.

Figure 4: Performance gain achieved using CLIP versus standard training on
ShanghaiTech Part-B Zhang et al. (2016) dataset. Figure 5: Performance gain
achieved using CLIP versus standard training on ShanghaiTech Part-B Zhang et
al. (2016) dataset. Figure 6: Sample predictions over three datasets. The left
column shows sample images each from Mall Chen et al. (2012b), ShanghaiTech
Part-B Zhang et al. (2016), and ShanghaiTech Part-A datasets Zhang et al.
(2016). The middle column shows predictions using CSRNet Li et al. (2018b)
model with standard training. The right column shows predictions using CSRNet
with CLIP training.

We also evaluated the performance of the proposed scheme (CLIP) over the Mall
dataset Chen et al. (2012b) and CARPK dataset Hsieh et al. (2017) and found
similar improvements of CLIP compared to standard training. Some interesting
predictions showing better predictions using CLIP as compared to standard
training are shown in Fig. 6.

##  5 Conclusion

In this paper, we propose CLIP - curriculum learning with iterative dataset
pruning. CLIP is an efficient training strategy improving model learning
performance as well as reducing convergence time. The evaluation results
validate the benefits of CLIP in crowd counting tasks. We believe CLIP can be
effectively applied in other deep learning tasks particularly when the dataset
size is too large. CLIP can be beneficial when the original dataset has
erroneous samples with noisy labels.

## References

  * Aich and Stavness [2018] Shubhra Aich and Ian Stavness.  Global sum pooling: A generalization trick for object counting with small datasets of large images.  _arXiv preprint arXiv:1805.11123_ , 2018.
  * Bengio et al. [2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, page 41–48, New York, NY, USA, 2009. Association for Computing Machinery.  ISBN 9781605585161.
  * Boominathan et al. [2016] Lokesh Boominathan, Srinivas S. S. Kruthiventi, and R. Venkatesh Babu.  Crowdnet: A deep convolutional network for dense crowd counting.  _Proceedings of the 24th ACM international conference on Multimedia_ , 2016.
  * Cao et al. [2018] Xinkun Cao, Zhipeng Wang, Yanyun Zhao, and Fei Su.  Scale aggregation network for accurate and efficient crowd counting.  In _ECCV_ , 2018.
  * Chandana and Vasavi [2022] V. Sri Chandana and S. Vasavi.  Autonomous drones based forest surveillance using faster r-cnn.  In _2022 International Conference on Electronics and Renewable Systems (ICEARS)_ , pages 1718–1723, 2022.  doi: 10.1109/ICEARS53579.2022.9752298.
  * Chen et al. [2012a] Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang.  Feature mining for localised crowd counting.  In _BMVC_ , 2012a.
  * Chen et al. [2012b] Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang.  Feature mining for localised crowd counting.  In _Proceedings of the British Machine Vision Conference_ , pages 21.1–21.11. BMVA Press, 2012b.
  * Elman [1993] Jeffrey L. Elman.  Learning and development in neural networks: the importance of starting small.  _Cognition_ , 48:71–99, 1993.
  * Gao et al. [2019] Chenyu Gao, Peng Wang, and Ye Gao.  Mobilecount: An efficient encoder-decoder framework for real-time crowd counting.  In _Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi’an, China, November 8–11, 2019, Proceedings, Part II_ , page 582–595. Springer-Verlag, 2019.
  * Gu and Lian [2022] Siqi Gu and Zhichao Lian.  A unified multi-task learning framework of real-time drone supervision for crowd counting.  _CoRR_ , abs/2202.03843, 2022.  URL https://arxiv.org/abs/2202.03843.
  * Hacohen and Weinshall [2019] Guy Hacohen and Daphna Weinshall.  On the power of curriculum learning in training deep networks.  _ArXiv_ , abs/1904.03626, 2019.
  * He et al. [2016] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * Hsieh et al. [2017] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H. Hsu.  Drone-based object counting by spatially regularized regional proposal network.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 4165–4173, 2017.
  * Jiang et al. [2019] Xiaolong Jiang, Zehao Xiao, Baochang Zhang, Xiantong Zhen, Xianbin Cao, David S. Doermann, and Ling Shao.  Crowd counting and density estimation by trellis encoder-decoder networks.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6126–6135, 2019.
  * Khan et al. [2022] Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  Revisiting crowd counting: State-of-the-art, trends, and future perspectives.  _Image Vis. Comput._ , 129:104597, 2022.
  * Khan et al. [2023] Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  LCDnet: a lightweight crowd density estimation model for real-time video surveillance.  _Journal of Real-Time Image Processing_ , 20, 2023.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2015.
  * Li et al. [2018a] Junyu Li, Ligang He, Shenyuan Ren, and Rui Mao.  Data fine-pruning: A simple way to accelerate neural network training.  In _NPC_ , 2018a.
  * Li et al. [2018b] Yuhong Li, Xiaofan Zhang, and Deming Chen.  Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1091–1100, 2018b.
  * Lin and Davis [2010] Zhe Lin and Larry S. Davis.  Shape-based human detection and segmentation via hierarchical part-template matching.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 32(4):604–618, 2010.  doi: 10.1109/TPAMI.2009.204.
  * Liu et al. [2019] Weizhe Liu, Mathieu Salzmann, and Pascal V. Fua.  Context-aware crowd counting.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5094–5103, 2019.
  * Narvekar et al. [2020] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone.  Curriculum learning for reinforcement learning domains: A framework and survey.  _J. Mach. Learn. Res._ , 21:181:1–181:50, 2020.
  * Paul et al. [2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  In _NeurIPS_ , 2021.
  * Platanios et al. [2019] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom Michael Mitchell.  Competence-based curriculum learning for neural machine translation.  _ArXiv_ , abs/1903.09848, 2019.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Sam et al. [2017] D. Sam, S. Surya, and R. Babu.  Switching convolutional neural network for crowd counting.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4031–4039, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  * Sambolek and Ivasic-Kos [2021] Sasa Sambolek and Marina Ivasic-Kos.  Automatic person detection in search and rescue operations using deep cnn detectors.  _IEEE Access_ , 9:37905–37922, 2021.  doi: 10.1109/ACCESS.2021.3063681.
  * Sangineto et al. [2019] E. Sangineto, Moin Nabi, Dubravko Culibrk, and N. Sebe.  Self paced deep learning for weakly supervised object detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41:712–725, 2019.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1409.1556.
  * Sindagi and Patel [2017] Vishwanath A. Sindagi and Vishal M. Patel.  Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting.  _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 1–6, 2017.
  * Song et al. [2021] Qingyu Song, Changan Wang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Jian Wu, and Jiayi Ma.  To choose or to fuse? scale selection for crowd counting.  In _AAAI_ , 2021.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–9, 2015.
  * Tang et al. [2022] Haihan Tang, Yi Wang, and Lap-Pui Chau.  Tafnet: A three-stream adaptive fusion network for rgb-t crowd counting.  _ArXiv_ , abs/2202.08517, 2022.
  * Tang et al. [2018] Yuxing Tang, Xiaosong Wang, Adam P. Harrison, Le Lu, Jing Xiao, and Ronald M. Summers.  Attention-guided curriculum learning for weakly supervised classification and localization of thoracic diseases on chest radiographs.  _ArXiv_ , abs/1807.07532, 2018.
  * Tian et al. [2010] Yan Tian, Leonid Sigal, Hernán Badino, Fernando De la Torre, and Yong Liu.  Latent gaussian mixture regression for human pose estimation.  In _ACCV_ , 2010.
  * Topkaya et al. [2014] Ibrahim Saygin Topkaya, Hakan Erdogan, and Fatih Porikli.  Counting people by clustering person detector outputs.  In _2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 313–318, 2014.  doi: 10.1109/AVSS.2014.6918687.
  * Wang and Breckon [2022] Qian Wang and T. Breckon.  Crowd counting via segmentation guided attention networks and curriculum loss.  _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * Wang et al. [2019] Wei Wang, Isaac Caswell, and Ciprian Chelba.  Dynamically composing domain-data selection with clean-data selection by “co-curricular learning” for neural machine translation.  In _ACL_ , 2019.
  * Wu and Nevatia [2006] Bo Wu and Ramakant Nevatia.  Detection and tracking of multiple, partially occluded humans by bayesian combination of edgelet based part detectors.  _International Journal of Computer Vision_ , 75:247–266, 2006.
  * Wu et al. [2021] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur.  When do curricula work?  _ArXiv_ , abs/2012.03107, 2021.
  * Yang et al. [2022] Shuo Yang, Zeke Xie, Hanyu Peng, Minjing Xu, Mingming Sun, and P. Li.  Dataset pruning: Reducing training data by examining generalization influence.  _ArXiv_ , abs/2205.09329, 2022.
  * Zeng et al. [2017] Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang.  Multi-scale convolutional neural networks for crowd counting.  _2017 IEEE International Conference on Image Processing (ICIP)_ , pages 465–469, 2017.
  * Zhang et al. [2015] Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang.  Cross-scene crowd counting via deep convolutional neural networks.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 833–841, 2015.
  * Zhang et al. [2016] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma.  Single-image crowd counting via multi-column convolutional neural network.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 589–597, 2016.  doi: 10.1109/CVPR.2016.70.


# Streaming end-to-end speech recognition with
jointly trained neural feature enhancement

###### Abstract

In this paper, we present a streaming end-to-end speech recognition model
based on Monotonic Chunkwise Attention (MoCha) jointly trained with
enhancement layers. Even though the MoCha attention enables streaming speech
recognition with recognition accuracy comparable to a full attention-based
approach, training this model is sensitive to various factors such as the
difficulty of training examples, hyper-parameters, and so on. Because of these
issues, speech recognition accuracy of a MoCha-based model for clean speech
drops significantly when a multi-style training approach is applied. Inspired
by Curriculum Learning [1], we introduce two training strategies: Gradual
Application of Enhanced Features (GAEF) and Gradual Reduction of Enhanced Loss
(GREL). With GAEF, the model is initially trained using clean features.
Subsequently, the portion of outputs from the enhancement layers gradually
increases. With GREL, the portion of the Mean Squared Error (MSE) loss for the
enhanced output gradually reduces as training proceeds. In experimental
results on the LibriSpeech corpus and noisy far-field test sets, the proposed
model with GAEF-GREL training strategies shows significantly better results
than the conventional multi-style training approach.

Index Terms: end-to-end speech recognition, data augmentation, monotonic
chunkwise attention, attention-based encoder-decoder, acoustic simulator

##  1 Introduction

Since the introduction of end-to-end all neural speech recognition models [2],
there have been growing interests in these new models. These models have
significant advantages in structural simplicity and simplified inference to
generate texts. Compared to the conventional speech recognition system
consisting of multiple discrete components such as an Acoustic Model (AM), a
Language Model (LM), a pronunciation dictionary, and a decoder based on a
Weighted Finite State Transducer (WFST), a complete end-to-end all neural
speech recognition system realizes all these functionalities using a single-
structured neural network. Further improvements in these end-to-end speech
recognition systems have been obtained thanks to a better choice of target
units such as Byte Pair Encoded (BPE) and unigram language model [3] units,
improved training methodologies, and so on. Another major advantage of these
end-to-end models is that they require much smaller memory footprint compared
to the conventional WFST-based models, which enables their widespread use for
on-device applications [4, 5].

Attention-based Encoder-Decoder (AED) is perhaps the most well-known type of
such end-to-end speech recognition systems [2]. Recently, it has been reported
that the performance of the AED system outperforms the conventional WFST-based
decoders for large vocabulary speech recognition tasks [6]. In spite of these
achievements in performance, the biggest shortcoming of the AED-based speech
recognition system is its lack of streaming capability. To overcome this
restriction, several variations of AED approaches including Monotonous
Chunkwise Attention (MoCha) have been proposed [7]. In our previous work [4],
we successfully employed MoCha -based model for on-device dictation
applications.

Recently, speech recognition has been widely adopted for AI speakers [8] and
home appliance devices [5]. Therefore, far-field speech recognition has become
increasingly more important. It has been observed that various kinds of data
augmentation [9] or approaches motivated by auditory processing [10] is
especially helpful for far-field noisy environments. On-the-fly data
augmentation using an acoustic simulator [8, 11] has been especially
successful for these far-field speech recognition scenarios. When data
augmentation using an acoustic simulator is employed for full-attention
models, we observe that it even enhances performance for clean utterances
perhaps because of better regularization as will be shown in Sec. 4. However,
the MoCha attention is generally unstable compared to the full attention [12].
Therefore, it has been very difficult to obtain good speech recognition
accuracy if the same data augmentation strategy is directly applied to MoCha
training, as will be also shown in Sec. 4.

In this work, we place an enhancement block consisting of two layers of Long
Short-Term Memories (LSTMs) in front of the streaming MoCha speech recognition
model. We refer this model to as Neural Enhancement-Automatic Speech
Recognition (NE-ASR). This combined model is jointly trained from scratch
using random initialization without any needs to train each sub-model
separately beforehand. Even though there have been many attempts to jointly
optimize an enhancement block along with a speech recognition model, our
approach is unique in the following two aspects:

  * •

Gradual Application of Enhanced Feature (GAEF)

  * •

Gradual Reduction of Enhancement Loss (GREL)

To the best of our knowledge, our work is the first attempt in gradually
reducing the enhancement loss in the joint training of the entire models which
has the net effect of gradually combining two separate enhancement and encoder
blocks into a single combined encoder block. As will be shown in Sec. 4, this
GREL strategy significantly enhances speech recognition performance. This NE-
ASR model shows improved performance compared to the baseline MoCha model for
LibriSpeech [13] test-clean and test-other, and significantly better
performance than the conventional data augmentation technique on the same test
set. For re-recorded far-field noisy test sets, our approach reduces Word
Error Rates (WERs) relatively by 55.93 % and 12.36 % over the baseline and the
multi-style trained MoCha models, respectively.

##  2 Related works

###  2.1 Monotonic Chunkwise Attention

AttentionDecoderSOFTMAX$\mathbf{c}_{l-1}$$\mathbf{y}^{(\text{embed})}_{l-1}$$\mathbf{h}^{(\text{enc})}[m]$$\mathbf{h}^{(\text{dec})}_{l-1}$EncoderFeatureExtractionAcousticAugmentationEnhancementLayersFeatureExtractionSignalNormalizationAttentionDecoderSOFTMAX$\mathbf{c}_{l-1}$$\mathbf{y}^{(\text{embed})}_{l-1}$$\mathbf{h}^{(\text{enc})}[m]$$\mathbf{h}^{(\text{dec})}_{l-1}$CombinedEncoderFeatureExtractionAcousticAugmentation$\mathbf{x}^{(\text{enh})}[m]$$\mathbf{x}^{(\text{aug})}[m]$$\hat{\mathbf{x}}[m]$$s[n]$$s^{(\text{aug})}[n]$$\hat{s}[n]$$s^{(\text{aug})}[n]$$s[n]$$(1-w)\mathbf{x}^{(\text{enh})}[m]+w\hat{\mathbf{x}}[m]$$\mathbf{x}^{(\text{aug})}[m]$$\mathbb{L}=\mathbb{L}_{CE}+\mathbb{L}_{CTC}+\lambda\mathbb{L}_{MSE}$$\mathbb{L}=\mathbb{L}_{CE}+\mathbb{L}_{CTC}$As
$\lambda\rightarrow 0$ and $w\rightarrow 0$ $\hat{y}_{l}$$\hat{y}_{l}$(a)(b)

Fig. 1:  The joint training strategy for a combination of neural enhancement
and an end-to-end speech recognition: The initial training pipeline is shown
in Fig. 1 (a). $\lambda$ is a weight for the Mean Squared Error (MSE) loss
$\mathbb{L}_{\text{MSE}}$ in (6). $w$ is a weight for the normalized clean
feature $\hat{\mathbf{x}}[m]$ in (3.2). As training proceeds, $\lambda$ and
$w$ reduce to zero, which transfigures the training pipeline into the one in
Fig. 1 (b).

In this section, we briefly describe the Monotonic Chunkwise Attention (MoCha)
algorithm [7]. In the MoCha model, there are two attention mechanisms: a hard
monotonic attention followed by a soft chunkwise attention. The hard monotonic
attention is employed to determine which element should be attended from a
sequence of hidden encoder outputs $\mathbf{h}^{\text{({\it enc})}}[m]$. The
hard monotonic attention is obtained from the hidden encoder output
$\mathbf{h}^{\text{({\it enc})}}[m]$ at the frame index $m$ and the hidden
decoder output $\mathbf{h}^{(\text{\it dec})}_{l-1}$ at the output label index
$l-1$ as follows:

$\displaystyle e^{\text{({\it mono})}}_{l}[m]$ $\displaystyle=\text{\it
MonotonicEnergy}(\mathbf{h}^{(\text{\it enc})}[m],\mathbf{h}^{(\text{\it
dec})}_{l-1})$ (1a) $\displaystyle a^{\text{({\it mono})}}_{l}[m]$
$\displaystyle=\sigma(e^{\text{(\it mono)}}_{l}[m])$ (1b) $\displaystyle
z_{l}[m]$ $\displaystyle\sim\text{\it Bernoullli}(a^{(\text{\it
mono})}_{l}[m])$ (1c)

where $\sigma(\cdot)$ is a logistic sigmoid function and MonotonicEnergy is
the energy function defined as follows [7]:

MonotonicEnergy $\displaystyle(\mathbf{h}^{\text{({\it
enc})}}[m],\;\mathbf{h}^{\text{({\it dec})}}_{l-1})$
$\displaystyle\;=g\frac{v^{\intercal}}{\|v\|}$
$\displaystyle\tanh(W^{\text{({\it dec})}}\mathbf{h}^{\text{({\it
dec})}}_{l-1}+W^{\text{({\it enc})}}\mathbf{h}^{\text{({\it enc})}}[m]+b)+r$
(2)

where $v,W^{\text{({\it dec})}},W^{\text{({\it enc})}},b,g$, and $r$ are
learnable variables. After finding out the position to attend using (1c), a
soft attention with a fixed chunk size is employed to generate the context
vector that will be given as an input to the decoder. For all model training
in this paper, an Adam optimizer [14] is employed. We observe that careful
learning rate scheduling is required to train high-performing MoCha models. In
this work, we employ a pre-training stage similar to that in our previous work
[4]. During the pre-training stage, when a new layer is added or the pool size
of a max-pooling layer is changed, the learning rate is lowered to 1.0e-4 and
linearly increased to 3.0e-4. In LibriSpeech training described in Sec. 4,
this pre-training stage continues up to 2.25 epochs. After the pre-training
stage, the learning rate is maintained at 3.0e-4 up to the eight full epochs.
After finishing eight epochs of training, the learning rate is reduced using
the Newbob method. Without careful learning rate control, MoCha model does not
converge well especially when far-field data augmentation is applied to the
training set.

###  2.2 Data augmentation using an acoustic simulator

The acoustic simulator we used in our joint-training strategy is based on the
structure described in our earlier works [8, 11, 15]. Our acoustic simulator
is designed to simulate the general case with $I$ sound sources and $J$
microphones in a cuboid-shaped room with acoustically reflective walls [15].
However, for brevity, we consider a single microphone case in this paper. The
relationship between the sound source $x_{0}[n]$ and the microphone output
$s^{\text{({\it aug})}}[n]$ is given by the following equation:

$\displaystyle s^{\text{({\it
aug})}}[n]=h_{0}[n]*x_{0}[n]+\sum_{i=1}^{I-1}\alpha_{i}h_{i}[n]*x_{i}[n],$ (3)

where $x_{i}[n]$ is the signal from the $i$-th source. $\alpha_{i}$ and
$h_{i}[n]$ are the scaling coefficient and the room impulse response
associated with each sound source. When the image method is employed, the room
impulse response $h_{i}[n]$ is given by the following equation [15]:

$\displaystyle
h_{i}[n]=\sum_{k=0}^{\infty}\frac{r^{g_{i,k}}}{d_{i,k}}\delta\left[n-\left\lceil{\frac{d_{i,k}f_{s}}{c_{0}}}\right\rceil\right],$
(4)

where $d_{i,k},r,g_{i,k},f_{s}$, and $c_{0}$ are the distance between the
microphone and the sound source image, the reflection coefficient of the wall,
the number of reflections in the acoustic path connecting the microphone and
the sound source image, the sampling rate of a signal, and the speed of sound
in the air, respectively. In this work, we employ a similar configuration to
our previous work [8] regarding the distribution of room sizes, reverberation
times ($T_{60}$) , Signal-to-Noise Ratios in Decibel (SNR dB), sound source
locations and so on.

##  3 Joint-training strategy

###  3.1 Neural Enhancement - Automatic Speech Recognition (NE-ASR) model

In this section, we explain our joint training strategy of enhancement and
speech recognition models in detail. The entire structure of the Neural
Enhancement and Automatic Speech Recognition (NE-ASR) model is shown in Fig.
1. This NE-ASR model structure does not change during the training phase.
However, since we gradually reduce the weight for the enhancement loss
$\lambda$ and the weight $w$ for the normalized clean feature to zero as
training proceeds, the structure is finally equivalent to the model in Fig. 1
(b).

$s[n]$ at the bottom of Fig. 1 is the clean speech signal with $n$ being the
sample index. Using the acoustic simulator described in Sec. 2.2 [8, 16], we
obtain simulated far-field noisy utterances $s^{\text{({\it aug})}}[n]$. For
on-the-fly data augmentation, we use the example server system described in
[11]. As will be discussed in Sec. 3.4, there is a mismatch in time-delay and
energy between the augmented signal $s^{\text{({\it aug})}}[n]$ and the
original signal $s[n]$. This mismatch is compensated using the Delay-Energy
Normalization (DEN) block. This DEN block is introduced because we believe
that the neural network cannot be trained effectively to learn the mapping
from noisy features to clean features if the time delay or the energy level
difference vary randomly.

The output from the DEN block is represented by $\hat{s}[n]$ in Fig. 1. We use
power-mel features of the order of 40 [16, 17], which is motivated by the
power-law nonlinearity with a power coefficient of $1/15$ [18]. We use a
window length of $25$ ms with the period between successive windows of $10$
ms. We use power-mel features instead of the more widely used log-mel features
since power-mel features have shown better performance in our previous studies
[16, 17]. We obtain power-mel features $\mathbf{x}^{\text{({\it aug})}}[m]$
and $\hat{\mathbf{x}}[m]$ from the augmented and the normalized clean speech
signals respectively with $m$ being the frame index. The enhancement layers in
Fig. 1 consist of two layers of LSTM. The unit sizes of these two layers are
1024 and 40 respectively. The enhanced feature $\mathbf{x}^{{\text{({\it
enh})}}}[m]$ is obtained as the output of the enhanced layers, and the linear
combination of the enhanced feature $\mathbf{x}^{{\text{({\it enh})}}}[m]$ and
the normalized clean feature $\hat{\mathbf{x}}[m]$ is used as the input to the
MoCha based streaming end-to-end speech recognition model. This feature
combination will be described in detail in the following Sec. 3.2. In the
MoCha model, the encoder consists of six layers of uni-directional LSTMs with
the unit size of 1536 that are interleaved with 2:1 max-pooling layers in the
lower three layers as in [4]. Thus, the overall temporal reduction factor is
8:1.

###  3.2 Gradual application of enhanced features (GAEF)

In machine learning, it has been frequently observed that models are trained
more effectively if they are trained with easier examples in the early stage
of training [1]. Motivated by this observation, we start training the NE-ASR
model with easier examples $\hat{\mathbf{x}}[m]$. In the initial stage of
training, the output from the enhancement block $\mathbf{x}^{\text{({\it
enh})}}[m]$ might be still noisy, since this enhancement layers are not
sufficiently trained. The input to the MoCha speech recognition model
$\mathbf{x}^{\text{({\it comb})}}[m]$ is a linear combination of the enhanced
feature $\mathbf{x}^{\text{({\it enh})}}[m]$ and the normalized clean feature
$\hat{\mathbf{x}}[m]$ that is given by the following equation:

$\displaystyle\mathbf{x}^{\text{({\it comb})}}[m]$
$\displaystyle=(1-w)\mathbf{x}^{\text{({\it enh})}}[m]+w\hat{\mathbf{x}}[m],$
$\displaystyle w:\;\;$ $\displaystyle 1.0\rightarrow 0.0\;\;\text{as training
proceeds}.$ (5)

where $w$ is the weighting coefficient. As the training proceeds, the $w$
decreases linearly from one to zero. In our experiments with LibriSpeech
training in Sec. 4, $w$ becomes zero after eight full epochs of training.

###  3.3 Gradual reduction of enhancement loss (GREL)

The loss function $\mathbb{L}$ that we used in this joint training is a
combination of the Cross-Entropy (CE) loss $\mathbb{L}_{\text{\it CE}}$,
Connectionist Temporal Classification (CTC) loss $\mathbb{L}_{\text{\it CTC}}$
[19] and the Mean Squared Error (MSE) loss, which is given by the following
equation:

$\displaystyle\mathbb{L}$ $\displaystyle=\mathbb{L}_{\text{\it
CE}}+\mathbb{L}_{\text{\it CTC}}+\lambda\mathbb{L}_{\text{\it MSE}},$ (6)
$\displaystyle\lambda:\;\;$ $\displaystyle 1.0\rightarrow 0.0\;\;\text{as
training proceeds}.$

The CTC loss $\mathbb{L}_{\text{\it CTC}}$ and the CE loss
$\mathbb{L}_{\text{\it CE}}$ are computed from the softmax outputs from the
encoder and the decoder respectively, as shown in Fig. 1 [4, 20]. The MSE loss
is computed from the enhancement layer output $\mathbf{x}^{\text{({\it
enh})}}[m]$ using the normalized clean feature $\hat{\mathbf{x}}[m]$ as the
target. Similar to the case of GAEF in Sec. 3.2, in our experiments with
LibriSpeech training in Sec. 4, $\lambda$ value is one at the beginning of
training and linearly reduces to zero after finishing eight epochs of
training.

###  3.4 Delay-Energy Normalization (DEN)

In this section, we describe delay-energy normalization employed to obtain the
normalized speech signal $\hat{s}[n]$ in Fig. 1. From (3), assuming that
$x_{0}[n]=s[n]$ is the target sound source and the remaining sound sources
$x_{i}[n],\;1\leq i\leq I-1$ are noise sound sources, this equation expressed
as follows:

$\displaystyle s^{\text{({\it aug})}}[n]=h_{0}[n]*s[n]+\nu[n],$ (7)

where $\nu[n]$ is the combined noise-terms in (3) defined by
$\sum_{i=1}^{I-1}\alpha_{i}h_{i}[n]*x_{i}[n]$. From (7), we observe that
$s^{\text{({\it aug})}}[n]$ has delays compared to the target signal
$x_{0}[n]$ because of the room impulse response $h_{0}[n]$. From (4), we
observe that the first arriving impulse response is delayed by
$\frac{f_{s}\times d}{c_{0}}$. Thus, in the DEN block, this amount of delay
component is added to the clean speech. The energy of $y[n]$ also may be very
different from that of $s[n]$. We measure the 95-percentile of frame energies
of $s^{\text{({\it aug})}}[n]$, and $s[n]$ is scaled to have the same
95-percentile of frame energies.

Table 1:  Word Error Rates (WERs) using three different neural network models on the LibriSpeech corpus and the far-field noisy test set. If a specific training strategy is employed, it is marked by the O symbol. Otherwise, it is marked by the X symbol.  Model Type | Training Strategy | Noise Types
---|---|---
| Far-field
---
Data Augmentation
GAEF |  GREL |  | LibriSpeech
---
test-clean
| LibriSpeech
---
test-other
| Far-Field
---
Noisy
BLSTM with Full Attention (1024 LSTM unit size) |  X |  - |  - |  4.19 % |  13.95 % |  70.22 %
O |  - |  - |  4.16 % |  11.44 % |  30.33 %
ULSTM with MoCha (1536 LSTM unit size) |  X |  - |  - |  6.06 % |  17.90 % |  78.83 %
O |  - |  - |  7.10 % |  16.42 % |  39.64 %
NE-ASR without DEN |  O |  X |  X |  8.01 % |  19.43 % |  43.79 %
O |  O |  X |  6.99 % |  19.35 % |  45.61 %
O |  X |  O |  6.86 % |  16.28 % |  36.73 %
O |  O |  O |  5.98 % |  15.70 % |  35.35 %
| NE-ASR
---
with DEN
O |  O |  O |  5.79 % |  15.63 % |  34.74  %

##  4 Experimental Results

In this section, we present speech recognition results obtained using the NE-
ASR model on the LibriSpeech database [13] and in-house far-field noisy test
set. For training, we used the entire 960 hours LibriSpeech training set
consisting of 281,241 utterances. For evaluation, we used the official 5.4
hours test-clean and 5.1 hours test-other sets. To evaluate the performance on
far-field noisy environments, re-recording was done by playing back 100
command utterances using a loud speaker at 5-meter distance in a real room.
The reverberation time in this room was measured to be $T_{60}=430$ ms. We
simulated far-field additive noise by playing back four different types of
noise using loud speakers: babble, music, two different types of sound from a
television. We created noisy utterances at five different SNR dB levels: 0-dB,
5-dB, 10-dB, 15-dB, and 20-dB. This far-field noisy test set was previously
used in our work described in [11].

Table 1 shows the speech recognition experimental results using three
different models. To implement and train this model, we use the Keras
framework [21] with the Tensorflow 2.3 toolkit. For all the cases in Table 1,
the models are trained twice up to 25 full epochs and these two WERs are
averaged. During the inference for evaluation, we use the beam size of 12. The
BFA stands for Bidirectional LSTMs (BLSTMs) in the encoder with a Full
Attention (BFA), which has a similar structure to those our previous work [16,
17]. For MoCha and NE-ASR models, we use Unidirectional LSTMs (ULSTMs) with
the MoCha attention for streaming speech recognition. The unit sizes of the
BLSTM and ULSTM in the encoder is 1024 and 1536, respectively. For all these
three models, a single layer of ULSTM of a unit size of 1000 is employed for
the decoder part. As shown in this Table, in the case of BFA, the performance
for all these three test sets improve by applying on-the-fly data augmentation
using the acoustic simulator [11]. We note that augmentation using noisy
utterances even enhances performance on the test-clean test set perhaps
because of better regularization. However in the case of the MoCha model,
after applying the same data augmentation, the performance on the test-clean
test set degrades by 14.65 % relatively even with a very careful learning rate
schedule mentioned in Sec. 2.1. Without careful learning rate scheduling, the
model even does not converge. In the case of the NE-ASR model, GREL strategy
significantly improves the performance compared to the MoCha model with the
far-field data augmentation. GAEF itself does not show performance improvement
when it is employed without GREL. However, when we employ both of the GAEF and
GREL algorithms, the best performance is achieved. As shown in Table 1, the
DEN approach brings further improvement. Overall, the NE-ASR model with the
GAEF-GREL training strategy shows significantly better results than the
conventional data augmentation technique with relative improvements of 18.45
%, 4.81 %, and 12.36 % on the LibriSpeech test-clean, test-other and the re-
recorded far-field noisy test sets respectively.

##  5 Conclusions

In this paper, we present a Neural Enhancement-Automatic Speech Recognition
(NE-ASR) model that consists of an enhancement model and an end-to-end speech
recognition model. These two models are jointly trained from scratch using
random initialization without pre-training each model separately. Inspired by
Curriculum Learning [1], we propose two training strategies in this work:
Gradual Application of Enhanced Feature (GAEF) and Gradual Reduction of
Enhancement Loss (GREL). With GAEF, training initially starts using clean
training features. Subsequently the portion of outputs from the enhancement
block gradually increases. With GREL, the Mean Squared Error (MSE) loss for
the enhanced output gradually reduces as training proceeds. In experimental
results on the LibriSpeech test-clean, test-other [22], and the re-recorded
far-field noisy test sets, the NE-ASR model with GAEF-GREL training strategies
shows significantly better results than the conventional data augmentation
technique with relative improvements of 18.45 %, 4.81 %, and 12.36 %
respectively. It has been very difficult to maintain the speech recognition
accuracy on clean speech with a MoCha-based model when data augmentation is
done using the acoustic simulator to enhance far-field noisy performance [8].
However, using the NE-ASR model with GAEF-GREL training strategies, the
performance on LibriSpeech test-clean and test-other sets is even
substantially better than the baseline MoCha model without noticeable training
stability issues.

## References

  * [1] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th Annual International Conference on Machine Learning_ , ser. ICML ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 41–48. [Online]. Available: https://doi.org/10.1145/1553374.1553380
  * [2] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in _Advances in Neural Information Processing Systems 28_ , C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates, Inc., 2015, pp. 577–585. [Online]. Available: http://papers.nips.cc/paper/5847-attention-based-models-for-speech-recognition.pdf
  * [3] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 66–75. [Online]. Available: https://www.aclweb.org/anthology/P18-1007
  * [4] K. Kim, K. Lee, D. Gowda, J. Park, S. Kim, S. Jin, Y.-Y. Lee, J. Yeo, D. Kim, S. Jung, J. Lee, M. Han, and C. Kim, “Attention based on-device streaming speech recognition with large speech corpus,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Dec. 2019, pp. 956–963.
  * [5] A. Garg, G. P. Vadisetti, D. Gowda, S. Jin, A. Jayasimha, Y. Han, J. Kim, J. Park, K. Kim, S. Kim, Y. yoon Lee, K. Min, and C. Kim, “Streaming On-Device End-to-End ASR System for Privacy-Sensitive Voice-Typing,” in _Proc. Interspeech 2020_ , 2020, pp. 3371–3375. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-3172
  * [6] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-of-the-art speech recognition with sequence-to-sequence models,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , April 2018, pp. 4774–4778.
  * [7] C.-C. Chiu and C. Raffel, “Monotonic chunkwise attention,” in _International Conference on Learning Representations_ , Apr. 2018. [Online]. Available: https://openreview.net/forum?id=Hko85plCW
  * [8] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. N. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in google home,” in _Proc. Interspeech 2017_ , 2017, pp. 379–383. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2017-1510
  * [9] C. Kim, K. Kumar and R. M. Stern, “Robust speech recognition using small power boosting algorithm,” in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Dec. 2009, pp. 243–248.
  * [10] C. Kim, K. Chin, M. Bacchiani, and R. M. Stern, “Robust speech recognition using temporal masking and thresholding algorithm,” in _INTERSPEECH-2014_ , Sept. 2014, pp. 2734–2738.
  * [11] C. Kim, S. Kim, K. Kim, M. Kumar, J. Kim, K. Lee, C. Han, A. Garg, E. Kim, M. Shin, S. Singh, L. Heck, and D. Gowda, “End-to-end training of a large vocabulary end-to-end speech recognition system,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Dec. 2019, pp. 562–569.
  * [12] H. Miao, G. Cheng, P. Zhang, T. Li, and Y. Yan, “Online Hybrid CTC/Attention Architecture for End-to-End Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 2623–2627. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2018
  * [13] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _IEEE Int. Conf. Acoust., Speech, and Signal Processing_ , April 2015, pp. 5206–5210.
  * [14] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [15] C. Kim, E. Variani, A. Narayanan, and M. Bacchiani, “Efficient implementation of the room simulator for training deep neural network acoustic models,” in _INTERSPEECH-2018_ , Sept 2018, pp. 3028–3032. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-2566
  * [16] C. Kim, M. Shin, A. Garg, and D. Gowda, “Improved vocal tract length perturbation for a state-of-the-art end-to-end speech recognition system,” in _INTERSPEECH-2019_ , Graz, Austria, Sept. 2019, pp. 739–743. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-3227
  * [17] C. Kim, M. Kumar, K. Kim, and D. Gowda, “Power-law nonlinearity with maximally uniform distribution criterion for improved neural network training in automatic speech recognition,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Dec. 2019, pp. 988–995.
  * [18] C. Kim and R. M. Stern, “Power-Normalized Cepstral Coefficients (PNCC) for Robust Speech Recognition,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , pp. 1315–1329, July 2016.
  * [19] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proceedings of the 23rd International Conference on Machine Learning_ , ser. ICML ’06. New York, NY, USA: ACM, 2006, pp. 369–376. [Online]. Available: http://doi.acm.org/10.1145/1143844.1143891
  * [20] S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 4835–4839.
  * [21] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.