CORRESPONDING AUTHOR: Amir Salimi (e-mail: asalimi@ualberta.ca). This work has
been funded by the Canadian Institutes of Health Research and the Heart and
Stroke Foundation of Canada.

# Exploring Best Practices for ECG Signal Processing in Machine Learning

Amir Salimi1  Sunil Vasu Kalmady2  Abram Hindle.1,2  Osmar Zaiane  Padma Kaul
University of Alberta, Computing Science, Edmonton, Alberta, Canada
University of Alberta, Canadian Vigour Centre, Edmonton, Alberta, Canada

###### Abstract

In this work we search for best practices in pre-processing of
Electrocardiogram (ECG) signals in order to train better classifiers for the
diagnosis of heart conditions. State of the art machine learning algorithms
have achieved remarkable results in classification of some heart conditions
using ECG data, yet there appears to be no consensus on pre-processing best
practices. Is this lack of consensus due to different conditions and
architectures requiring different processing steps for optimal performance? Is
it possible that state of the art deep-learning models have rendered pre-
processing unnecessary? In this work we apply down-sampling, normalization,
and filtering functions to 3 different multi-label ECG datasets and measure
their effects on 3 different high-performing time-series classifiers. We find
that sampling rates as low as 50Hz can yield comparable results to the
commonly used 500Hz. This is significant as smaller sampling rates will result
in smaller datasets and models, which require less time and resources to
train. Additionally, despite their common usage, we found min-max
normalization to be slightly detrimental overall, and band-passing to make no
measurable difference. We found the blind approach to pre-processing of ECGs
for multi-label classification to be ineffective, with the exception of sample
rate reduction which reliably reduces computational resources, but does not
increase accuracy.

###### keywords:

electrocardiogram, machine learning, signal processing

##  1 Introduction

We focus on the Electrocardiogram (ECG) pre-processing decisions faced by
researchers working towards automatic classification of cardiovascular
conditions. The ever-growing number of tools and approaches available for
machine learning often require a number of a priori guesses to be made by
those who process the training data and design the architectures. The task of
training models which predict heart disease is no exception.

Routinely utilized by clinicians for diagnoses of cardiovascular
abnormalities, an ECG is a recording of the electrical activity of the
cardiovascular system [1, 2]. In recent years, machine learning models have
achieved remarkable results in automatic diagnosis of some heart conditions
when trained with enough labeled ECG data [3, 4, 5, 6, 7]. However, training
such models requires a large amount of data [3, 4, 7], where decisions
regarding how the data is pre-processed (e.g., filtering, scaling, data
augmentation) can be critical for the model’s performance, hardware
requirements, and training time.

Pre-processing functions are transformations applied to signals—ECGs, in this
case—in order to reduce noise and simplify the learning task. Previous works
utilize a diverse range of pre-processing functions, datasets, and
architectures, often with great results [8]. As we will discuss in Section 2,
the variety of approaches in past works makes finding a trend between the pre-
processing functions used and performance results not feasible. We believe
that in order to make claims about the viability of ECG pre-processing methods
(that is, whether they result in better outcomes) it is best to consider
multiple datasets, architectures, and heart conditions.

Here, we aim to simplify this decision space for other researchers by testing
the effect of down-sampling, band-passing, and normalization on the outcome of
3 different cutting-edge multi-label classifiers for time-series when trained
on 3 different ECG datasets. We show that down-sampling from the standard
recording rate of 500Hz can significantly reduce training times and hardware
requirements, without being detrimental to model performance. We found min-max
normalization to be slightly detrimental overall, and band-passing to make no
measurable difference. In Section 5 we discuss the experiment setup and
performance measures, and discuss the results of down-sampling, band-passing,
and normalization in Sections 6, 7, and 8. Our methodology and datasets can be
replicated using our code, which is publically
available111https://github.com/imilas/ecg_augmentation.

Table 1: Functions used by Top 5 teams in Physionet2020 [8] Rank | Group | Scaling | Normalize | BandPass | Window
---|---|---|---|---|---
1 | Wide and Deep Transformer [9] |  | ✓ | ✓ | 15s, 0 Pad
2 | Adaptive ResNet [7] | ✓(257 Hz) | ✓ |  | 16s, 0 Pad
3 | SE-ResNet [10] |  |  |  | 30s, 0 Pad
4 | Scatter Transform and DNNs [11] |  | ✓ |  | 20.5s
5 | Adversarial Domain Generalization [12] | ✓(250 Hz) | ✓ | ✓ |

##  2 Previous Work

The state-of-the-art benchmarks for the automatic classification of cardio-
vascular diseases continually improve as more datasets become available and
computation costs decrease. However, no consensus seems to exist on the best
pre-processing steps for such datasets. In this work we test various
approaches to pre-processing taken by previous works in-order to measure their
effects on classification performance.

Here, we define pre-processing as “any function which aims to simplify or
transform the data in-order to enhance the performance outcomes and/or reduce
training over-head”. [8] highlight 4 common pre-processing methods for ECG
signals, based on their analysis of entries to the Physionet2020 Challenge
[13, 14]: resizing (which we do not experiment on), resampling, band-passing,
and normalizing. Examples of works using one or more of these methods are
given in Table 1.

###  2.1 Resizing

Signals are time-series, and time-series vary in length. Resizing refers to
assigning a fixed length (or window) to all signals. Here, we use a fixed
window length of 5000 (10 seconds) for all experiments, and pad signals with
zeros on the right, if the full size $x$ is lower than our desired fixed
length. If $x$ is longer than the fixed length, then samples 0-$x$ are used.

###  2.2 Resampling (Down-Sampling):

ECGs are typically sampled at the rate of 500Hz [15, 2], meaning that each
second of an ECG recording contains 500 samples. Down-sampling an ECG is often
no different than resolution reduction of an image. Since a lower sampling
rate leads to smaller data, it can often drastically lower training times and
hardware requirements, but important information may be lost. In this work we
use the simple and commonly used nearest-neighbour interpolation [16] method
to measure the effect of different sampling rates on the performance of ML
models. We discuss our findings with regards to performance and model sizes in
Section 6.

###  2.3 Band-Passing

Many sources can introduce noise to ECG signals. Powerlines are a common
example [2], which depending on the regional infrastructure can introduce
noise at various frequencies. It is common for ECG recording hardware to use
built-in band rejection (or notch filters) to deal with such issues, and many
ECG libraries apply notch filters to ECGs for this reason [17]. However, there
are other unaccounted sources of noise which need to be dealt with at the
software level. In signal processing, a band-pass filter is used for the
removal of frequencies outside of a given range (or cut-offs) [18]; all
frequencies below the high-pass cut-off and above the low-pass cut-off are
removed or reduced. This can improve performance as noise can interfere with
the learning process. In this work we compare the use of different cut-offs as
well as the case when no filtering was applied to the dataset. The results are
shown in Section 7.

###  2.4 Normalizing

Normalization of input values has been shown to produce better or equal value
results in image and time-series classification [19]. Particularly in the case
of ECGs, there can be differences in recorded amplitudes depending on the
equipment used [2], making normalization a logical step in pre-processing.
Here we compare the performance of min-max normalization to raw ECGs and show
the results in Section 8.

##  3 Models

There is a large variety in the architectures used to learn from ECG datasets,
or time-series data more generally. Here, we use three models with proven
results in state of the art time-series classification benchmarks, implemented
by the TSAI library [20]. The Inception-Time Network, which is the
1-dimensional application of the Inception-Network [21, 22] and MiniRocket, a
quick and mostly deterministic feature extractor for time-series [23]. A
recent survey of time-series classification methods by [24] highlights both of
these models as excellent performers in various multi-variable time-series
classification benchmarks. In addition, Inception-Time has excelled in ECG
classification benchmarks [25]. The third architecture tested here is
xresnet1d101, a resnet model for time-series which was found to be the overall
best performer in a recent benchmark study of best architectures for
classification of ECGs by [26].

##  4 Datasets

We conduct our experiments on three datasets, the CPSC dataset [27], Chapman-
Shaoxing dataset [28], and PTB-XL [29]. The datasets are all multi-label, and
recorded at the sampling rate of 500Hz. All have been released as part of the
8 datasets of labeled 12-lead ECGs provided by the Physionet2021 challenge [3,
4]. To simplify our datasets and experiments, we only use the labels which
appear in more than 5% of the ECGs in each dataset. Here we only use 10
seconds of data from each ECG. ECGs shorter than this length are right padded
with zeros, therefore we can represent each ECG as a matrix with dimensions of
12x5000 (before down-sampling). The breakdown of these datasets and their
label counts after our modifications are given in Tables 2, 3, and 4.

Table 2: Modified CPSC Dataset Label | Count
---|---
right bundle branch block | 1857
ventricular ectopics | 700
atrial fibrillation | 1221
1st degree av block | 722
premature atrial contraction | 616
sinus rhythm | 918
st depression | 869
Total | 6903
Table 3: Modified Chapman Dataset Label | Count
---|---
left ventricular high voltage | 1295
atrial fibrillation | 1780
t wave abnormal | 1876
sinus bradycardia | 3889
supraventricular tachycardia | 587
sinus rhythm | 1826
sinus tachycardia | 1568
nonspecific st t abnormality | 1158
Total | 9910
Table 4: Modified PTB-XL Label | Count
---|---
left axis deviation | 5146
myocardial ischemia | 2175
myocardial infarction | 5261
left ventricular hypertrophy | 2359
ventricular ectopics | 1154
atrial fibrillation | 1514
t wave abnormal | 2345
abnormal QRS | 3389
sinus rhythm | 18092
left anterior fascicular block | 1626
incomplete rbbb | 1118
Total | 21311

###  4.1 CPSC

Released in 2018 as part a multi-label ECG classification competition [27] and
used in recent benchmarks [26]. ECG signal duration in this dataset is between
6 and 60 seconds, with an average duration of 15.79 seconds. Originally, this
dataset has 9 labels, reduced to 7 for our work.

###  4.2 Chapman-Shaoxing

Has not been subject to many benchmarks. This dataset contains 10,646 12-lead
ECGs, originally with 11 labels, reduced to 8 in our work [28].

###  4.3 PTB-XL

Used in recent benchmarks [26], this dataset contains 21,837 12-lead ECGs
[29]. The original dataset has 54 labels, reduced to 11 in our work.

##  5 Experiment Methodology

In Section 2 we discussed common pre-processing functions, and how some high-
performing works have used one or more of these functions, while others have
omitted them entirely (as seen in Table 1). We discussed the three high-
performing time-series architectures which we will be using, and in Section 4,
we described the three datasets of multi-labeled ECGs that the models can
learn from. Best practices for ECG pre-processing could be established if we
observe consistent patterns of improvement in performance scores when applying
a pre-processing function to different datasets, and comparing the
performances of different models.

To compare pre-processing types, we apply the methods we want to analyze to
each dataset, and train models on the transformed datasets. We use the
performance of the model to score and rank the pre-processing function. Given
enough of these performance scores for the various models, datasets, and
diseases, we use statistical tests such as Wilcoxon signed rank test and
Kruskal-Wallis to determine whether the pre-processing method was a
significant factor in the performance of the models. For example, we would
have the CPSC dataset at various sampling rates: CPSC_500Hz, CPSC_250Hz, etc.
For each version of the dataset (i.e, the dataset pre-processed using a
different function), we train and test each model 20 times. For 20 rounds, the
data is randomly split in 80/10/10 training/validation/testing sets. The same
seeding is used for the random splits such that we can use pair-wise tests to
compare experiment results. Each model is trained on the training set, at the
end of each epoch the new weights are saved to disk if the model has improved
its highest F1 score on the validation set. When no improvements in the F1
measure (on the validation set) is seen after 30 epochs, the training stops,
and the highest scoring weights (based on validation set performance) are
loaded in the model and the F1 score on the test set is saved as the
performance score of the experiment. This means that for each dataset/pre-
processing/model combination, we have 20 performance scores.

In this case, we can use the Wilcoxon signed rank test to measure whether each
model out-performs every other model. To rank and compare the models, we use
the Wilcox-Holm post-hoc analysis used for creating critical-difference
diagrams by [30]. This analysis uses a pairwise Wilcoxon test with Holm-
Bonferroni p-value correction, and shows whether the differences between each
model pair is significant. While [30] use the average rank to show relative
performance, we use the “Mean Reciprocal Rank” (MRR) for each model. Here,
$rank_{i}$ refers to the model’s rank relative to the other models when
trained/validated/tested on the same subsets of the data, and $Q=20$ since for
each model/dataset combination we have 20 F1 scores:

$\text{MRR}=\frac{1}{Q}\sum_{i=1}^{Q}\frac{1}{rank_{i}}$

Another method for comparison of results is the Kruskal-Wallis test [31, 32].
This test measures whether the median of several sets are different. We use
this test to determine whether the pre-processing method had any effect on the
performance of models for each disease. Since we are comparing 3 different
pre-processing methods for 20 different heart conditions, we use the Holm-
Bonferroni correction [33, 34], and divide our initial alpha of 0.05 by 60
(since we run 3*20 tests), which gives us the new alpha
$\alpha^{*}=8.33e^{-4}$. We reject the null hypothesis (i.e, the pre-
processing method has an effect on the outcome) if $p<\alpha^{*}$. We’ve
provided the raw p-value results in Table 5, such that less conservative means
of determining pre-processing significance can be applied.

Table 5: Kruskal-Wallis P-Value for importance of factor in diagnosis of each disease  Label / Heart Condition | Sampling Rate | Bandpass | Normalizing
---|---|---|---
1st degree av block | 3.50e-02 | 2.128e-01 | 2.74e-01
abnormal QRS | 3.56e-08 | 7.514e-01 | 2.41e-08
atrial fibrillation | 6.53e-04 | 9.828e-02 | 4.14e-03
incomplete right bundle branch block | 2.13e-11 | 8.064e-01 | 3.30e-02
left anterior fascicular block | 2.06e-01 | 1.707e-01 | 2.00e-04
left axis deviation | 2.33e-04 | 3.918e-02 | 7.56e-06
left ventricular high voltage | 2.84e-01 | 9.227e-02 | 1.44e-14
left ventricular hypertrophy | 6.34e-05 | 6.495e-01 | 2.09e-12
myocardial infarction | 2.86e-10 | 3.122e-02 | 1.62e-02
myocardial ischemia | 2.82e-04 | 4.939e-01 | 2.80e-07
nonspecific st t abnormality | 2.51e-02 | 5.428e-01 | 1.49e-02
premature atrial contraction | 5.97e-08 | 2.842e-01 | 3.28e-01
right bundle branch block | 1.89e-02 | 7.902e-01 | 2.38e-02
sinus bradycardia | 1.43e-05 | 7.902e-01 | 9.28e-01
sinus rhythm | 3.23e-01 | 9.566e-01 | 3.23e-01
sinus tachycardia | 2.54e-03 | 2.685e-02 | 4.93e-01
st depression | 6.40e-03 | 9.764e-01 | 4.43e-05
supraventricular tachycardia | 9.73e-01 | 2.441e-01 | 5.42e-01
t wave abnormal | 1.29e-01 | 7.262e-01 | 1.53e-01
ventricular ectopics | 4.02e-21 | 2.357e-01 | 3.56e-01
Table 6: P-value for Kruskal-Wallis test across conditions. Do sampling rate, band-passing, normalizing, and model architecture affect the F1 result? Sampling Rate | Band-passing | Normalization | Model
---|---|---|---
3.65e-01 | 9.04e-01 | 2.00e-04 | 1.11e-4

##  6 Scaling/Down-sampling

The ECG signals in our dataset are recorded at 500Hz. We apply the scaling
rates of 0.1, 0.25, 0.5, 0.75, and 1, which down-samples the signals to
sampling rates of 50Hz, 125Hz, 250Hz, 375Hz, and the unmodified 500Hz. After
scaling, we applied “min-max” norm, which we assumed to be the safest choice
considering the popularity of normalization as a pre-processing method (see
Section 8. The down-sampling method used here is Pytorch’s “nearest-exact”
interpolation function [35], which is a GPU implementation of the same
algorithm by [36] and the commonly used PIL libarary [37]. Effectively, each
dataset has 5 different sampling rates, and each of the 3 models learns from
these datasets by randomly selecting 80% of the dataset for training, 10% for
validation, and 10% for testing. In total, for each dataset, we run 15
experiments (3 architectures, and 5 different sampling rates) and for each
experiment, we train and evaluate 20 times leading to 20 performance measures,
where each performance measure is macro-average F1 score on the test set. We
use the Wilcox-Holm post-hoc analysis described in the methodology (Section 5)
to get the rank and relative performance for each model.

###  6.1 Down-sampling Results

The MRR for each model when predicting each label is shown in Figure 1. Here,
no consistent pattern can be seen across diseases and scaling rates. However,
some trends are observable: for example, Atrial fibrillation—a common label in
the 3 datasets—deep learning models with higher sampling rates performed the
best. Another interesting label is Sinus Rythm, also common in all 3 datasets,
where the inception network with lowest possible sampling rate clearly out-
performs. Kruskal-Wallis p-values in Table 5 show that sampling rate is an
important factor in 10 out of 20 diseases (the 10 significant p-values are
highlighted in the Table under the “Sampling Rate” column). From these
observations we can conclude that sampling rate is an important factor for
better diagnosis of some heart conditions.

We also show the Spearman correlation between sampling rate and performance in
Table 8. There is a per-model breakdown of the correlation values, as well as
the values when all datasets and all models are considered. Here, correlation
measures the mapping between two continuous variables: sampling rates between
50-500Hz and F1 performance. Overall, when considering all models and all
datasets, we see a negligible correlation of $0.008$. This suggests that
sampling-rate is not an important factor in the overall results of our
training. This weak relationship is also shown in Table 6 where the Kruskal-
Wallis test results across all diseases measured a p-value of 0.365

Table 7: Higher sampling rates required more training VRAM (in Gigabytes). Relative ratios should be considered, since absolute values are affected by window size, precision, batch size, etc. Hz | inception | xresent1d | rocket
---|---|---|---
50 | 1.8 | 3.2 | 1.5
250 | 3.9 | 6.4 | 3.2
500 | 6.5 | 11.0 | 5.0

Main Takeaway: Based on these results, sampling rate does appear to be an
important factor in diagnosis of multiple labels, however, it does not
standout as a particularly important factor when training a multi-label model,
unless some labels in the models are weighted differently than others. This is
a significant find as sampling-rate has a direct effect on the speed and
hardware required for training deep-learning models. As shown in Table 7,
higher sampling rates have a higher VRAM cost, which affects training times
and energy consumption. These results suggest that it is possible to reduce
sampling rates from the standard 500Hz to values lower than 250Hz without
major loss in over-all performance.

Table 8: Weak Spearman correlations are observed between F1 and sampling-rate (50-500Hz), and F1 and norm. ($0\xrightarrow{}$raw, $1\xrightarrow{}$min-max). | Rate | Norm
---|---|---
Inception | -5.23e-02 | -1.17e-01
MiniRocket | 2.70e-02 | -1.21e-02
xresnet | 5.17e-02 | -9.50e-02
All Models | 8.00e-03 | -7.41e-02
Figure 1: Performance of the models for each disease at varying scaling rates.
The 15 models are ranked against each other, higher MRR means better relative
performance

Figure 2: Performance of the models for each disease at varying HighPass
frequencies. Higher MRR means better relative performance

Figure 3: Performance of the models for each disease using different
normalization types. Higher MRR means better relative performance

##  7 Band-Pass Filtering

A band-pass filter removes all frequencies outside its range, which can be an
effective noise removal technique. The band-pass filter range applied to ECG
data varies greatly in past works. Typically, the high-pass filter takes on a
value of 0.05-1Hz, and the low-pass filter is applied somewhere between
30-150Hz [2, 15]. Is there an “optimal” range for a band-pass filter in this
context? Is band-pass filtering even necessary when using large datasets and
large deep learning models?

In Section 6.1, we discussed how down-sampling does not appear to have a major
effect on the over-all model performance. Here, in order to speed up training,
we downsample each dataset to 250Hz. We then apply 3 different band-pass
filters to the data before training the 3 models. The 3 ranges are 1-30Hz,
1-50Hz, 1-100Hz. We compare these 3 pre-processing methods to see if the
change in the low-pass filter frequency makes a difference in the final
outcome, we then make a comparison to the case when no filtering was applied
to the dataset.

###  7.1 Band-Pass Filtering Results

The MRR for each model when predicting each label is shown in Figure 2. The
Kruskal-Wallis results in Table 5 do not indicate a difference between the
performance of the 4 model groups in question (models trained using the 3
different band-passing functions, and models trained on the raw data). This is
an interesting result which may explain the lack of consistency in the
application of signal filtering in past ECG classification works. In addition,
as shown in Table 6, a p-value of 0.904 was measured by the Kruskal-Wallis
test between F1 measures and different band-passing methods when considering
all diseases.
Main Takeaway: When training on multiple labels from large, carefully curated
datasets of ECGs, band-passing is not a necessary step. Band-pass filtering
does not hurt the final outcomes, and can be applied at the GPU level with low
overhead. Therefore in most situations, particularly when dealing with
potentially noisy data (for example, where DC offset may not have been removed
prior), it is advisable to use filtering. We cannot make any concrete
statements about best cut-off frequencies, as it likely varies based on the
recording hardware used and the cardio-vascular conditions being diagnosed.

##  8 Normalization

What we refer to as ”normalization” is the transformation of an input
signal—which in this case is an ECG—to a distribution of values in the range
of [0,1] [8]. Here we use the “Min-Max” normalization method, a commonly used
pre-processing technique for ECG signals [2, 38, 39, 40]. This technique uses
the following formula where $Y[n]$ is the new amplitude for time-step $n$,
$X[n]$ is the old amplitude, $min$ is the minimum value for $X[n]$, and $max$
is the maximum value for $X[n]$ [2].

$Y[n]=\frac{x[n]-min}{max-min}$

Here we ran the models an the datasets with no pre-processing, and compare the
results to the models with only normalization applied.

###  8.1 Normalization Results

The MRR for each model when predicting each label is shown in Figure 3.
P-values in Table 5 show normalization to be an important factor for 7 out of
20 diseases. For example, “left ventricular high voltage” and “left
ventricular hypertrophy” performed better without normalization in all 3
models. To calculate the correlation values between F1 and the normalization
type in Table 8, we mapped the label of raw datasets to 0, and min-maxed
datasets to 1. The correlation values have a slight negative bias, indicating
that when normalization was important, more often than not models training on
raw ECGs performed better. In addition, as shown in Table 6, a small p-value
of 0.0002 was measured by the Kruskal-Wallis test between F1 measures and the
two normalization groups; this indicates that normalization (or lack thereof)
was overall an important factor.

Main Takeaway Surprisingly, Min-Max normalization has a slight negative impact
on the performance relative to raw ECGs. We cannot discourage the use of all
normalization methods as it is only one of the common normalization methods
used [2]. However, these results indicate that staple pre-processing methods
in other fields of deep-learning cannot be presumed effective for ECG
classification.

##  9 Discussion

Prior to this work, we found no consistency in the pre-processing methods used
in classification of cardiovascular conditions using ECG. To address this gap,
we trained 3 different classifiers on 3 datasets of ECGs and examined the
effect of 3 pre-processing methods on the performance of the models in
classifying various heart conditions. Our findings suggest that the
performance of different pre-processing and architecture combinations varies
depending on the condition of interest. This would explain the lack of
consistency in previous works, where datasets, label weights, and performance
measures vary.

We found rates as low as 50Hz can yield comparable if not better results than
the commonly used 500Hz sampling rate. This is significant as smaller sampling
rates can drastically reduce time and hardware requirements for training
models. We also found min-max normalization to be slightly detrimental, and
band-passing to make no measurable difference in model performance. Beyond
these findings, the overall takeaway of this work is that there are no “one-
size-fits-all” solutions to the problem of ECG pre-processing, and the
“optimal” best pre-processing method is label and architecture dependent.
While this result may seem unsatisfactory, at the very least this work shows
that researchers aiming to utilize SOTA, high-performing models for different
labels and datasets cannot assume that the pre-processing method used by other
works will be optimal. This work also highlights the need for an effective,
automated pre-processing step in the ECG classification domain (or time-series
in general), which appears to be an open problem.

Here, we ignored the possible interdependence between pre-processing methods.
An approach which considers the sequence in which the methods are applied
would exponentially increase the—already costly—time and hardware requirements
for this work. Additionally, past works rarely discuss the sequencing in which
their pre-processing methods were applied, making their replication of all the
more difficult.

Many recent works have noted the higher accuracy of an ensemble model with
various architectures relative to any single model [2, 8, 30, 26, 6], although
the diminishing returns of accuracy vs training time and hardware costs should
also be considered. Our findings here suggest that the use of various
architectures as well as different pre-processing methods could yield slight
improvements in benchmarks. However, taking such an approach would be brute-
forcing the problem, rather than solving it.

## References

  * [1] Neda Dianati Maleki, Arash Ehteshami Afshar, and Paul W Armstrong,  “Use of electrocardiogram indices of myocardial ischemia for risk stratification and decision making of reperfusion strategies,”  Journal of Electrocardiology, vol. 47, no. 4, pp. 520–524, 2014\.
  * [2] Anthony Ngozichukwuka Uwaechia and Dzati Athiar Ramli,  “A comprehensive survey on ecg signals as new biometric modality for human authentication: Recent advances and future challenges,”  IEEE Access, 2021.
  * [3] Matthew A Reyna, Nadi Sadr, Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Sardar Ansari, et al.,  “Will two do? varying dimensions in electrocardiography: the physionet/computing in cardiology challenge 2021,”  in 2021 Computing in Cardiology (CinC). IEEE, 2021, vol. 48, pp. 1–4.
  * [4] Matthew A Reyna, Nadi Sadr, Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Sardar Ansari, et al.,  “Issues in the automated classification of multilead ecgs using heterogeneous labels and populations,”  personnel, vol. 4, pp. 5, 2021.
  * [5] Antônio H Ribeiro, Manoel Horta Ribeiro, Gabriela MM Paixão, Derick M Oliveira, Paulo R Gomes, Jéssica A Canazart, Milton PS Ferreira, Carl R Andersson, Peter W Macfarlane, Wagner Meira Jr, et al.,  “Automatic diagnosis of the 12-lead ecg using a deep neural network,”  Nature communications, vol. 11, no. 1, pp. 1–9, 2020.
  * [6] Tsai-Min Chen, Chih-Han Huang, Edward SC Shih, Yu-Feng Hu, and Ming-Jing Hwang,  “Detection and classification of cardiac arrhythmias by a challenge-best deep learning neural network model,”  Iscience, vol. 23, no. 3, pp. 100886, 2020.
  * [7] Zhibin Zhao, Hui Fang, Samuel D Relton, Ruqiang Yan, Yuhong Liu, Zhijing Li, Jing Qin, and David C Wong,  “Adaptive lead weighted resnet trained with different duration signals for classifying 12-lead ecgs,”  in 2020 Computing in Cardiology. IEEE, 2020, pp. 1–4.
  * [8] Shenda Hong, Wenrui Zhang, Chenxi Sun, Yuxi Zhou, and Hongyan Li,  “Practical lessons on 12-lead ecg classification: Meta-analysis of methods from physionet/computing in cardiology challenge 2020,”  Frontiers in Physiology, p. 2505, 2022.
  * [9] Annamalai Natarajan, Yale Chang, Sara Mariani, Asif Rahman, Gregory Boverman, Shruti Vij, and Jonathan Rubin,  “A wide and deep transformer neural network for 12-lead ecg classification,”  in 2020 Computing in Cardiology. IEEE, 2020, pp. 1–4.
  * [10] Zhaowei Zhu, Han Wang, Tingting Zhao, Yangming Guo, Zhuoyang Xu, Zhuo Liu, Siqi Liu, Xiang Lan, Xingzhi Sun, and Mengling Feng,  “Classification of cardiac abnormalities from ecg signals using se-resnet,”  in 2020 Computing in Cardiology. IEEE, 2020, pp. 1–4.
  * [11] Maximilian P Oppelt, Maximilian Riehl, Felix P Kemeth, and Jan Steffan,  “Combining scatter transform and deep neural networks for multilabel electrocardiogram signal classification,”  in 2020 Computing in Cardiology. IEEE, 2020, pp. 1–4.
  * [12] Hosein Hasani, Adeleh Bitarafan, and Mahdieh Soleymani Baghshah,  “Classification of 12-lead ecg signals with adversarial multi-source domain generalization,”  in 2020 Computing in Cardiology. IEEE, 2020, pp. 1–4.
  * [13] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley,  “Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,”  circulation, vol. 101, no. 23, pp. e215–e220, 2000.
  * [14] Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, et al.,  “Classification of 12-lead ecgs: the physionet/computing in cardiology challenge 2020,”  Physiological measurement, vol. 41, no. 12, pp. 124003, 2020.
  * [15] Shen Luo and Paul Johnston,  “A review of electrocardiogram filtering,”  Journal of electrocardiology, vol. 43, no. 6, pp. 486–496, 2010\.
  * [16] Olivier Rukundo and Hanqiang Cao,  “Nearest neighbor value interpolation,”  arXiv preprint arXiv:1211.1768, 2012.
  * [17] Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen,  “NeuroKit2: A python toolbox for neurophysiological signal processing,”  Behavior Research Methods, vol. 53, no. 4, pp. 1689–1696, feb 2021\.
  * [18] Richard G Lyons,  Understanding digital signal processing, 3/E,  Pearson Education India, 1997.
  * [19] Samit Bhanja and Abhishek Das,  “Impact of data normalization on deep neural network for time series forecasting,”  arXiv preprint arXiv:1812.05519, 2018.
  * [20] Ignacio Oguiza,  “tsai - a state-of-the-art deep learning library for time series and sequential data,” Github, 2022.
  * [21] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi,  “Inception-v4, inception-resnet and the impact of residual connections on learning,”  in Thirty-first AAAI conference on artificial intelligence, 2017\.
  * [22] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean,  “Inceptiontime: Finding alexnet for time series classification,”  Data Mining and Knowledge Discovery, vol. 34, no. 6, pp. 1936–1962, 2020.
  * [23] Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb,  “Minirocket: A very fast (almost) deterministic transform for time series classification,”  in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 248–257.
  * [24] Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall,  “The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances,”  Data Mining and Knowledge Discovery, vol. 35, no. 2, pp. 401–449, 2021.
  * [25] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek,  “Deep learning for ecg analysis: Benchmarks and insights from ptb-xl,”  IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1519–1528, 2021.
  * [26] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek,  “Deep learning for ecg analysis: Benchmarks and insights from ptb-xl,”  IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1519–1528, 2020.
  * [27] Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, et al.,  “An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection,”  Journal of Medical Imaging and Health Informatics, vol. 8, no. 7, pp. 1368–1373, 2018.
  * [28] Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski,  “A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients,”  Scientific Data, vol. 7, no. 1, pp. 1–8, 2020.
  * [29] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter,  “Ptb-xl, a large publicly available electrocardiography dataset,”  Scientific data, vol. 7, no. 1, pp. 1–15, 2020.
  * [30] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller,  “Deep learning for time series classification: a review,”  Data Mining and Knowledge Discovery, vol. 33, no. 4, pp. 917–963, 2019.
  * [31] William H Kruskal and W Allen Wallis,  “Use of ranks in one-criterion variance analysis,”  Journal of the American statistical Association, vol. 47, no. 260, pp. 583–621, 1952.
  * [32] Eva Ostertagova, Oskar Ostertag, and Jozef Kováč,  “Methodology and application of the kruskal-wallis test,”  in Applied Mechanics and Materials. Trans Tech Publ, 2014, vol. 611, pp. 115–120.
  * [33] Sture Holm,  “A simple sequentially rejective multiple test procedure,”  Scandinavian journal of statistics, pp. 65–70, 1979.
  * [34] Hervé Abdi,  “Holm’s sequential bonferroni procedure,”  Encyclopedia of research design, vol. 1, no. 8, pp. 1–8, 2010.
  * [35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala,  “Pytorch: An imperative style, high-performance deep learning library,”  in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., pp. 8024–8035. Curran Associates, Inc., 2019.
  * [36] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu,  “scikit-image: image processing in python,”  PeerJ, vol. 2, pp. e453, 2014.
  * [37] Alex Clark,  “Pillow (pil fork) documentation,” 2015.
  * [38] Ming Li and Shrikanth Narayanan,  “Robust ecg biometrics by fusing temporal and cepstral information,”  in 2010 20th International Conference on Pattern Recognition. IEEE, 2010, pp. 1326–1329.
  * [39] Yaoguang Li and Wei Cui,  “Identifying the mislabeled training samples of ecg signals using machine learning,”  Biomedical signal processing and control, vol. 47, pp. 168–176, 2019.
  * [40] Shih-Chin Fang and Hsiao-Lung Chan,  “Human identification by quantifying similarity and dissimilarity in electrocardiogram phase space,”  Pattern Recognition, vol. 42, no. 9, pp. 1824–1831, 2009.


Yassir Fathullah1,2 Work done during internship at Meta AI., Chunyang Wu2,
Yuan Shangguan2, Junteng Jia2, Wenhan Xiong2, Jay Mahadeokar2, Chunxi Liu2,
Yangyang Shi2, Ozlem Kalinli2, Mike Seltzer2, Mark J. F. Gales1

# Multi-Head State Space Model for Speech Recognition

###### Abstract

State space models (SSMs) have recently shown promising results on small-scale
sequence and language modelling tasks, rivalling and outperforming many
attention-based approaches. In this paper, we propose a multi-head state space
(MH-SSM) architecture equipped with special gating mechanisms, where parallel
heads are taught to learn local and global temporal dynamics on sequence data.
As a drop-in replacement for multi-head attention in transformer encoders,
this new model significantly outperforms the transformer transducer on the
LibriSpeech speech recognition corpus. Furthermore, we augment the transformer
block with MH-SSMs layers, referred to as the Stateformer, achieving state-of-
the-art performance on the LibriSpeech task, with word error rates of
1.76%/4.37% on the development and 1.91%/4.36% on the test sets without using
an external language model.

Index Terms: speech recognition, transducer, librispeech, state space model,
attention-free, transformer, stateformer

##  1 Introduction

Recurrent neural networks (RNNs) have historically been a core approach for a
wide range of sequence modelling tasks such as speech recognition [1, 2],
machine translation [3, 4] and language modelling [5, 6]. However, RNNs were
rapidly replaced with the introduction of the transformer [7] and large pre-
trained models [8]. The effectiveness of the transformer has also caused other
fields such as computer vision to consolidate towards attention-based models
[9, 10].

One of the key reasons behind the success of transformers and their widespread
use is the self-attention mechanism. Unlike previous approaches, self-
attention was shown to be highly effective at capturing global features of a
sequence by modelling all pairwise interactions. Furthermore, while
transformers are exceptionally good at capturing global long-range
dependencies they are less able in modelling local patterns. To this end,
there has been a range of work on combining convolutional networks with
attention for sequence modelling, and has been found to be highly effective
for speech recognition [11, 12, 13].

Meanwhile, the deep learning community has slowly been paying more attention
to alternative recurrent neural network approaches to effectively and
efficiently model sequences [14, 15, 16, 17]. Specifically, a well-established
signal processing and control theory technique, the state space model (SSM)
[18], which historically has been widely used in many continuous or discrete
time-series and control problems [19, 20] has been under renewed scrutiny.
However, the recurrent time-variant nature of general SSMs has traditionally
made them computationally expensive and inaccessible to many large-scale
sequence tasks. Nonetheless, recent work has shown that it is possible to
simplify and scale state space models and train them in parallelized manner,
by equivalently rephrasing them as a convolution with variable-length kernels
[14, 15]. Further work has also shown that simplified, structured versions of
the time-invariant state space model can be highly efficient, handle long-
range dependencies and be a formidable alternative to self-attention on some
sequence tasks, such as language modelling [21, 22, 23].

In this work, we evaluate and extend the work on SSMs for speech recognition,
proposing a multi-head state space model, equipped with a novel gating
mechanism. Since SSMs have shown promising performance on long sequence tasks
we hypothesize that a multi-headed approach could better handle both short and
long-term modelling simultaneously. We investigate the use of such multi-head
state space models both as a replacement and complement to self-attention in
the acoustic encoder of a neural network transducer model. Our technical
contributions include:

  1. (a)

Stacked and multi-head generalization. We extend the SSM approach by allowing
multi-head processing of linearly projected lower-dimensional signals and
stack such a layer for better performance.

  2. (b)

Head gating. We propose an inter-head gating approach in which different SSMs
within the multi-head layer communicate by gating each other.

  3. (c)

Combination with attention. We also augment the transformer encoder by
including a bidirectional SSM residual block prior to the attention block for
state-of-the-art performance. This model is referred to as the Stateformer.

With these contributions, we advance the state of attention-free models on the
LibriSpeech speech recognition task, outperforming strong attention-based
baselines. We also show that the Stateformer can achieve state-of-the-art
performance on this task with word error rates of 1.76%/4.37% on the
development and 1.91%/4.36% on the test sets, without using an external
language model.

##  2 State Space Model: The Linear RNN

The time-invariant state space model [24] is a fully linear recurrent network
taking the following form:

$\left.\begin{array}[]{c}\begin{aligned}
\bm{x}_{k}&={\bm{A}}\bm{x}_{k-1}+{\bm{B}}\bm{u}_{k}\\\
\bm{y}_{k}&={\bm{C}}\bm{x}_{k}+{D}\bm{u}_{k}\end{aligned}\end{array}\thickspace\right\\}\thickspace\thickspace\bm{y}={\tt
SSM}(\bm{u})$ (1)

It simply transforms an arbitrary input signal $\bm{u}$ into an output signal
$\bm{y}$ through some hidden process $\bm{x}$. Since this model is linear, it
can also be phrased as a convolution [14, 25] allowing it to be trained in a
parallelizable manner without recurrences. More importantly, this model can be
made highly efficient and effective by restricting the parameters
$\bm{A},\bm{B},\bm{C},\bm{D}$ to be block-diagonal and ensuring that the
transition matrix $A$ is stable i.e., the SSM generates bounded outputs for
bounded inputs [22, 26]. Furthermore, the effectiveness of this model is
highly dependent on its initialization. Work has found that this system can
encode the history of an input signal effectively with a proper initialization
scheme [21, 27]. The combination of these ideas culminates in a model called
S4 [22].

The S4 model is inherently unidirectional. For non-causal applications such as
the audio-encoder for offline speech recognition, a bidirectional S4 can be
used with non-linear activations and pointwise linear layers [28]:

$\displaystyle\bm{y}$ $\displaystyle\leftarrow{\tt Cat}([{\tt S4}(\bm{u}),{\tt
Rev}({\tt S4}({\tt Rev}(\bm{u})))])$ (2) $\displaystyle\bm{y}$
$\displaystyle\leftarrow{\tt Linear}({\tt Activation}(\bm{y}))$

The next section will build upon this model by using multiple parallel heads
and introducing an inter-head gating mechanism.

##  3 Multi-Head State Space Model

This section will describe a number of technical and architectural proposals
for the audio-encoder in the transducer. Section 3.1 introduces the stacked
MH-SSM. Section 3.2 describes a novel gating mechanism which allows different
SSM heads to communicate. Section 3.4 combines the MH-SSM with self-attention
for a new transformer architecture. Finally, Section 3.3 describes how the MH-
SSM can be used to replace the convolutional frontend.

###  3.1 Stacked & Multi-Head Extension

We extend the S4 layer with a significantly more flexible approach. Taking
inspiration from multi-head self-attention, we project the input
$D_{i}$-dimensional signal into $H\in\\{2,4,8,\dots\\}$ separate signals of
dimension $\bar{D_{i}}=D_{i}/H$, and process each using an independent SSM
that is randomly initialized.

Figure 1: The MH-SSM block first projects an input signal to several lower-
dimensional signals, each fed to a separate SSM. The output is gated,
concatenated and repeated a second time.

While this can be followed by a simple non-linear activation such as ReLU or
GELU, we opt for a novel gating mechanism described in the following section.
Finally, we repeat this procedure once again for a stacked module, see Figure
1. This module would then operate as a drop-in replacement for the $S4$ in
Equation 2 to form a bidirectional model. This multi-head design provides the
flexibility to learn both meaningful time-steps and different types of
temporal dynamics on sequence data.

###  3.2 Inter-Head Gating

By default, prior work has used the GELU activation function, see Equation 2.
In some experiments [22, 28], the GLU activation was also found beneficial.
However, a multi-head state space architecture with $H$ heads offers many more
gating possibilities. We propose an inter-head gating (IHG) approach in which
half the number of heads are used to gate the remaining heads, allowing
different heads to communicate and generally leading to improved results, as
illustrated in Figure 2.

Figure 2: Previous approaches would either apply GELU or GLU to the output of
a single SSM. IHG gates the output of an SSM using another independent SSM.

The IHG output is computed by mixing the heads according to (where $\sigma$
refers to the sigmoid):

$\displaystyle a^{(h)}$
$\displaystyle=y^{(h)}\cdot\sigma\left(y^{(h+H/2)}\right),\thickspace\thickspace
h=\\{1,...,H/2\\}$ (3)

allowing different heads to communicate and gate each other, generally leading
to improved results. It should be noted that the number of heads $H$ needs to
be even. While our approach is wildly different, the notion of allowing heads
to communicate has also been suggested in [29] regarding attention, where
linear layers mix information across heads and have been shown to improve
performance.

###  3.3 Multi-Scale Frontend

Audio-encoders typically subsample the input sequence using a convolutional
frontend to reduce sequence length and increase computational tractability [1,
30]. In this work, we utilize a multi-scale (MS) state space front end using
the MH-SSM to make use of its ability to model longer-range dependencies, see
Figure 3. The frontend intertwines MH-SSM blocks that capture temporal
dependencies and time reduction layers which reduce the frame rate resulting
in the same output size and striding as typical convolutional frontends.

Figure 3: Filterbanks are passed on to a linear layer with 128D outputs. These
are fed through 2 smaller MH-SSM residual blocks followed by a time reduction
(TR) layer which splices every two frames. This is again followed by another
set of MH-SSM blocks and a TR layer resulting in 512D features with an
effective 40ms frame rate. Figure 4: Stateformer: Bidirectional state space
augmented transformer block. It simply has an additional block prior to the
attention unit with a bidirectional MH-SSM. The pure MH-SSM architecture is
similar but without the self-attention block.

###  3.4 Stateformer: State Space Augmented Transformer

A pure multi-head state space model architecture is attractive due to its
ability to capture both short and long-range dependencies. However, since the
state space model is equivalent to a linear RNN, it is expressively more
limited in the temporal dimension. Therefore, we propose a model combining the
MH-SSM with attention by simply inserting a pre-norm bidirectional block prior
to the self-attention unit in the transformer architecture, referred to as the
Stateformer, see Figure 4.

##  4 Experimental Evaluation

###  4.1 Data

We evaluate the proposed models on the LibriSpeech dataset [31] consisting of
about 960 hours of speech data sampled at 16kHz. SpecAugment [32] and speed
perturbation [33] were used for data augmentation. We used a sliding window of
25ms with a 10ms frame shift to extract 80-dimensional filterbank.

###  4.2 Baselines

All speech recognition models were based on the transducer framework [30],
which has three components: encoder, predictor and joiner. We train various
transducers by keeping the predictor and joiner fixed, and compare various
Transformer [34], Conformer [11] and S4 [22] encoders against the proposed
approaches. The 80-dim input feature was first linearly projected to a
dimension of 128. Furthermore, we explore the style of subsampling frontends
by comparing:

  1. (a)

Time Reduction Layers (TR): Splicing 128-dim frames to 512 dimensions,
reducing the sequence length by 4x.

  2. (b)

Convolutional Layers (CNN): 2D convolutional network with a total stride of 4
and 512 output channels [35, 11].

  3. (c)

Multi-scale Layers (MS): Proposed multi-scale frontend which intertwines MH-
SSM blocks with TR layers, reducing the sequence length by 4x.

###  4.3 Implementation Details

Baseline and proposed models are implemented using an extension of the Fairseq
framework [36]. The encoder model dimension was set to 512 and kept fixed for
all experiments; the model size was controlled by the number of encoder
layers. The baselines used the convolutional frontend and consists of 20-36
S4, Transformer or Conformer blocks with 8 self-attention heads (not
applicable to S4), and a feed-forward net dimension of 2048. Different to
[11], our Conformer baseline did not use a macaron style block as it was not
found useful and had the convolutional module prior to the attention with a
kernel size of 31. The prediction network consisted of a three-layer
512-dimensional LSTM with layernorm and dropout. Both the encoder and
predictor outputs were projected to 1024 dimensions before being fed into an
additive joiner with a single linear layer of $|{\mathcal{Y}}|=4097$ sentence-
piece [37] output units. Similar to the S4 baseline, our proposed MH-SSM
simply replaced the self-attention layer of the transformer; the configuration
of this layer, such as the number of stacks, heads, use of inter-head gating
and frontend subsampler are investigated. The Stateformer uses the same setup
as the transformer combined with the best MH-SSM configuration. Large (greater
than 100M parameters) baselines and proposed models were also trained using
auxiliary classifiers similar to [38], in which intermediate encoder outputs
are trained to predict frame labels every 4 layers.

All models used the Adam optimizer [39] with a learning rate linearly warming
up to the peak value in 10k iterations, fixed until the 60th epoch and
thereafter, exponentially decayed by a factor of 0.96 each epoch. A dropout
value of 0.10 is used in all encoders and 0.30 in all predictors and the batch
size is set based on occupying maximal GPU memory. All models were trained up
to 200 epochs using 32 NVIDIA A100 GPUs. Hyperparameters and level of
checkpoint averaging were based on the development set.

##  5 Results & Discussion

###  5.1 Main Results

Table 1 shows the word error rate performance of our models of the large
configuration on LibriSpeech with our baselines and state-of-the-art models
including ContextNet [40], Transformer [34], Conformer [11, 41] and the
recently introduced Branchformer [12] and E-Branchformer [13]. No external
language model was used.

Table 1: WER% performance of baseline and proposed models on Librispeech
compared with best results found in the literature (no external language
model). At approximately 140.3M parameters, our attention-free MH-SSM model is
competitive with ContextNet and outperforms many other reported models. At
139.8M parameters, our Stateformer is competitive with the best-reported
Conformer and outperforms all other models.

Model | Params | dev | test
---|---|---|---
clean | other | clean | other
AED |  |  |  |  |
Branchformer [12] | 116.2M | 2.2 | 5.5 | 2.4 | 5.5
E-Branchformer [13] | 148.9M | – | – | 2.14 | 4.55
Conformer [13] | 147.8M | – | – | 2.16 | 4.74
Transducer |  |  |  |  |
Transformer [34] | 139M | – | – | 2.4 | 5.6
Transformer [42] | 160M | – | – | 2.2 | 4.7
ContextNet [40] | 112.7M | 2.0 | 4.6 | 2.1 | 4.6
Conformer [11] | 118.8M | 1.9 | 4.4 | 2.1 | 4.3
Conformer [41] |  $\simeq$120M | 2.0 | 4.7 | 2.2 | 4.8
_Baselines_ |  |  |  |  |
S4 36L | 129.6M | 2.21 | 5.63 | 2.41 | 5.68
Transformer 36L | 129.0M | 2.16 | 5.28 | 2.32 | 5.34
Conformer 24L | 133.7M | 1.95 | 4.84 | 2.21 | 5.04
_Proposed Models_ |  |  |  |  |
MH-SSM 32L | 140.3M | 1.80 | 4.96 | 2.01 | 4.61
Stateformer 25L | 139.8M | 1.76 | 4.37 | 1.91 | 4.36

The performance of our (attention-free) multi-head state space model (MH-SSM)
is able to achieve competitive results of $1.80/4.96/2.01/4.61$ outperforming
existing transformer transducers and competing with ContextNet. It also
significantly outperforms the original S4 which is unable to outperform
transformer baselines. Furthermore, this model is able to outperform one of
the Conformer transducers [41] on all but dev-other. The Stateformer further
pushes the performance by combining MH-SSM blocks with self-attention. The
achieved WERs of $1.76/4.37/1.91/4.36$ are highly competitive outperforming
essentially all models with one exception, the original Conformer. In this
case, the Stateformer is able to outperform the Conformer in all but dev and
test-other for which it is competitive. Overall, this demonstrates the power
of multi-head state space models both as a standalone model and when combined
with self-attention blocks.

###  5.2 Ablation Studies

####  5.2.1 Baselines

Table 2 reports the WER performance of smaller baseline models. Overall we can
observe that the attention-free S4 (SSM baseline) can outperform the
transformer with a time reduction frontend. With a convolutional frontend the
larger Conformer is best followed by the transformer. The transformer benefits
more from a convolutional frontend as it complements self-attention. Models
with convolutional aspects, Conformer and S4 (which can be seen as a variable
length convolution) benefit less.

Table 2: Baseline S4, Transformer and Conformer WER% performance with various
frontends (FE). With a CNN frontend, a larger Conformer is best followed by
the Transformer.

Model | Params | FE | dev | test
---|---|---|---|---
clean | other | clean | other
S4 20L | 78.1M | TR | 2.45 | 6.88 | 2.71 | 6.72
78.8M | CNN | 2.36 | 6.60 | 2.67 | 6.47
Transformer 20L | 76.7M | TR | 2.96 | 7.09 | 3.05 | 7.18
77.5M | CNN | 2.45 | 5.82 | 2.62 | 6.15
Conformer 20L | 91.9M | TR | 2.17 | 5.54 | 2.43 | 5.45
92.7M | CNN | 2.04 | 5.31 | 2.26 | 5.37

####  5.2.2 Stacking SSM Layers

There are a number of differences between a standard S4 and the MH-SSH block,
specifically, the stacking of SSM layers within a single residual block. Table
3 shows the contrast between stacking SSM layers within a residual block
versus opting for deeper models. Stacking was found marginally better, and
more importantly, allows the model to scale to larger sizes.

Table 3: Impact of stacking state space layers in a residual block. Stacking 2
layers was found to be marginally more effective than increasing the number of
layers.

Layers | Params | Stack | dev | test
---|---|---|---|---
clean | other | clean | other
16 | 56.8M | 1 | 2.57 | 7.13 | 2.79 | 6.86
66.3M | 2 | 2.36 | 6.88 | 2.52 | 6.59
20 | 67.6M | 1 | 2.42 | 6.92 | 2.67 | 6.56

####  5.2.3 Number of Heads and IHG

Next, we compare the number of SSM heads and the impact of using IHG instead
of standard GLU activations, see Table 4. All of the MH-SSM models use a
simpler TR frontend. At 74.7M parameters, the 4H with IHG model is able to
outperform the 20L transformer baseline and rival the one with CNN frontend.
Overall, the table shows the effectiveness of combining multi-head with head
gating.

Table 4: Ablation study investigating the number of heads and use of IHG in
the MH-SSM model. All models use the TR frontend and have 74.7M parameters.

$\\#$H | IHG | dev | test
---|---|---|---
clean | other | clean | other
2 | ✗ | 2.33 | 6.92 | 2.58 | 6.49
4 | ✗ | 2.23 | 6.74 | 2.52 | 6.46
2 | ✓ | 2.13 | 6.81 | 2.47 | 6.44
4 | ✓ | 2.19 | 6.38 | 2.42 | 6.25
8 | ✓ | 2.17 | 6.49 | 2.43 | 6.20

####  5.2.4 Multi-Scale Frontend

Using the best found MH-SSM from the previous section with 4 heads and IHG
enabled, we evaluate the impact of including a multi-scale frontend, see Table
5. With a minor increase of 4.5M parameters, the performance of the MS + MH-
SSM system is significantly better. The corresponding Stateformer further
improves performance by a notable margin, outperforming the Conformer baseline
in Table 2.

Table 5: Simple comparison of TR vs MS frontend (FE) for the MH-SSM model;
including a final Stateformer based on the best found MH-SSM model.

Model | Params | FE | dev | test
---|---|---|---|---
clean | other | clean | other
MH-SSM 16L | 74.7M | TR | 2.19 | 6.38 | 2.42 | 6.25
79.2M | MS | 2.14 | 6.12 | 2.39 | 5.99
Stateformer 16L | 96.1M | MS | 2.06 | 5.01 | 2.27 | 5.07

##  6 Conclusions

We proposed a multi-head state space model (MH-SSM) for the audio-encoder of
transducer-based speech recognition. Parallel heads of SSMs, together with a
novel inter-head gating mechanism was shown highly effective yielding a new
class of high-performing models. In addition, we combine MH-SSM blocks and
self-attention to form a new type of transformer architecture–the Stateformer.
On the LibriSpeech speech recognition task, the proposed MH-SSM outperforms
transformer baselines by a large margin. Furthermore, the Stateformer further
improves the performance, achieving state-of-the-art performance without
external language models.

## References

  * [1] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in _International Conference on Acoustics, Speech and Signal Processing_ , 2016\.
  * [2] K. Rao, H. Sak, and R. Prabhavalkar, ``Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,'' in _Automatic Speech Recognition and Understanding Workshop_ , 2017.
  * [3] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau _et al._ , ``Learning phrase representations using RNN encoder–decoder for statistical machine translation,'' in _Conference on Empirical Methods in Natural Language Processing_ , 2014.
  * [4] D. Bahdanau, K. Cho, and Y. Bengio, ``Neural machine translation by jointly learning to align and translate,'' in _International Conference on Learning Representations_ , 2015.
  * [5] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, ``Recurrent neural network based language model,'' in _Interspeech_ , 2010\.
  * [6] S. Merity, N. S. Keskar, and R. Socher, ``Regularizing and optimizing LSTM language models,'' in _International Conference on Learning Representations_ , 2018.
  * [7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Conference on Neural Information Processing Systems_ , 2017.
  * [8] J. Devlin, M. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' _North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019.
  * [9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai _et al._ , ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _International Conference on Learning Representations_ , 2021\.
  * [10] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang _et al._ , ``Swin transformer: Hierarchical vision transformer using shifted windows,'' in _International Conference on Computer Vision_ , 2021.
  * [11] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang _et al._ , ``Conformer: Convolution-augmented transformer for speech recognition,'' _Interspeech_ , 2020.
  * [12] Y. Peng, S. Dalmia, I. Lane, and S. Watanabe, ``Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding,'' in _International Conference on Machine Learning_ , 2022.
  * [13] K. Kim, F. Wu, Y. Peng, J. Pan, P. Sridhar, K. J. Han, and S. Watanabe, ``E-branchformer: Branchformer with enhanced merging for speech recognition,'' in _Spoken Language Technology Workshop_ , 2022.
  * [14] A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré, ``Combining recurrent, convolutional, and continuous-time models with linear state-space layers,'' _Conference on Neural Information Processing Systems_ , 2021.
  * [15] A. Voelker, I. Kajić, and C. Eliasmith, ``Legendre memory units: Continuous-time representation in recurrent neural networks,'' in _Conference on Neural Information Processing Systems_ , 2019.
  * [16] T. Lei, Y. Zhang, S. I. Wang, H. Dai, and Y. Artzi, ``Simple recurrent units for highly parallelizable recurrence,'' in _Empirical Methods in Natural Language Processing_ , 2018.
  * [17] T. Lei, ``When attention meets fast recurrence: Training language models with reduced compute,'' _arXiv:2102.12459_ , 2021.
  * [18] R. E. Kalman, ``A New Approach to Linear Filtering and Prediction Problems,'' _Journal of Basic Engineering_ , 1960.
  * [19] R. J. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder, _Forecasting with Exponential Smoothing: The State Space Approach_. Springer Berlin, Heidelberg, 2008.
  * [20] J. Durbin and S. J. Koopman, _Time Series Analysis by State Space Methods_. Oxford University Press, 05 2012\.
  * [21] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, ``Hippo: Recurrent memory with optimal polynomial projections,'' _Conference on Neural Information Processing Systems_ , 2020.
  * [22] A. Gu, K. Goel, and C. Ré, ``Efficiently modeling long sequences with structured state spaces,'' in _The International Conference on Learning Representations_ , 2022.
  * [23] H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur, ``Long range language modeling via gated state spaces,'' in _International Conference on Learning Representations_ , 2023.
  * [24] W. L. Brogan, _Modern control theory; 3rd ed._ Englewood Cliffs, NJ: Prentice-Hall, 1991. [Online]. Available: https://cds.cern.ch/record/226422
  * [25] N. R. Chilkuri and C. Eliasmith, ``Parallelizing legendre memory unit training,'' in _International Conference on Machine Learning_ , 2021.
  * [26] J. T. Smith, A. Warrington, and S. W. Linderman, ``Simplified state space layers for sequence modeling,'' _arXiv:2208.04933_ , 2022.
  * [27] A. Gu, I. Johnson, A. Timalsina, A. Rudra, and C. Ré, ``How to train your hippo: State space models with generalized basis projections,'' _arXiv:2206.12037_ , 2022.
  * [28] K. Goel, A. Gu, C. Donahue, and C. Ré, ``It's raw! audio generation with state-space models,'' _International Conference on Machine Learning_ , 2022\.
  * [29] N. Shazeer, Z. Lan, Y. Cheng, N. Ding, and L. Hou, ``Talking-heads attention,'' in _arXiv:2003.02436_ , 2020.
  * [30] A. Graves, ``Sequence transduction with recurrent neural networks,'' in _International Conference of Machine Learning Workshop on Representation Learning_ , 2012.
  * [31] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An asr corpus based on public domain audio books,'' in _International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [32] D. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. Cubuk, and Q. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' 2019.
  * [33] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, ``Audio augmentation for speech recognition,'' in _Sixteenth annual conference of the international speech communication association_ , 2015.
  * [34] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, ``Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,'' 2020.
  * [35] Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang _et al._ , ``Transformer-based acoustic modeling for hybrid speech recognition,'' in _International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [36] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng _et al._ , ``fairseq: A fast, extensible toolkit for sequence modeling,'' in _Proceedings of NAACL-HLT 2019: Demonstrations_ , 2019.
  * [37] T. Kudo and J. Richardson, ``Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,'' _arXiv:1808.06226_ , 2018.
  * [38] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, ``Going deeper with convolutions,'' in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [39] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' in _International Conference for Learning Representations_ , 2015.
  * [40] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu _et al._ , ``Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,'' _Interspeech_ , 2020.
  * [41] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, ``Pushing the limits of semi-supervised learning for automatic speech recognition,'' _2010.10504_ , 2020.
  * [42] C. Liu, F. Zhang, D. Le, S. Kim, Y. Saraf, and G. Zweig, ``Improving rnn transducer based asr with auxiliary tasks,'' in _Spoken Language Technology Workshop_ , 2021.