# Mamba in Speech: Towards an Alternative to Self-Attention

Xiangyu Zhang1*, Qiquan Zhang1*,Hexin Liu2*, Tianyi Xiao3,
Xinyuan Qian4,Beena Ahmed1, Eliathamby Ambikairajah1, Haizhou Li5,6, Julien
Epps1
1The University of New South Wales, 2 Nanyang Technological University,
3Beijing Institute of Technology 4University of Science and Technology Beijing
5National University of Singapore, 6The Chinese University of Hong Kong,
Shenzhen

###### Abstract

Transformer and its derivatives have achieved success in diverse tasks across
computer vision, natural language processing, and speech processing. To reduce
the complexity of computations within the multi-head self-attention mechanism
in Transformer, Selective State Space Models (i.e., Mamba) were proposed as an
alternative. Mamba exhibited its effectiveness in natural language processing
and computer vision tasks, but its superiority has rarely been investigated in
speech signal processing. This paper explores solutions for applying Mamba to
speech processing using two typical speech processing tasks: speech
recognition, which requires semantic and sequential information, and speech
enhancement, which focuses primarily on sequential patterns. The experimental
results exhibit the superiority of bidirectional Mamba (BiMamba) for speech
processing to vanilla Mamba. Moreover, experiments demonstrate the
effectiveness of BiMamba as an alternative to the self-attention module in
Transformer and its derivates, particularly for the semantic-aware task. The
crucial technologies for transferring Mamba to speech are then summarized in
ablation studies and the discussion section to offer insights for future
research. 111*Equal contribution, Our open accessed repository can be found
via https://github.com/Tonyyouyou/Mamba-in-Speech

##  1 Introduction

Transformer-based models [1] have shone brightly across various domains in
machine learning, including computer vision (CV) [2, 3, 4], natural language
processing (NLP) [5, 6, 7], and speech processing [8, 9]. This success is
linked to the multi-head self-attention (MHSA) module, which facilitates the
representation of intricate data structures within a specific context window.
However, the self-attention mechanism encounters a challenge with
computational complexity, which grows quadratically as the size of the context
window increases. In speech tasks, the window typically encompasses an entire
speech sample. This particularly yields an extensive context especially for
frame-level acoustic feature sequences, leading to a considerable increase in
computational complexity. Numerous efforts have been made to address this
challenge, with one notable approach being the utilization of a state space
model (SSM). SSM-based approaches [10, 11, 12, 13] have been developed to
handle sequential data across diverse tasks and modalities. By integrating a
time-varying mechanism into SSMs, a new model named Mamba [14] has been
proposed and shown outstanding performance in CV [15, 16] and NLP [17].

However, in the field of speech processing, despite some attempts to replace
transformers with Mamba [18, 19, 20], the results have not been as
satisfactory as expected. In [18], each Mamba is directly employed as a
substitute for a Transformer within a dual-path framework for speech
separation. [19] proposed SPMamba for speech separation, where Mamba is used
in conjunction with MHSA. Although these approaches achieve high performance
by employing a dual-path strategy or combining Mamba with attention to form a
new module, these methods negate the low time complexity of Mamba. In the
domain of multi-channel speech enhancement, Mamba was implemented to enhance a
SpatialNet from offline to online [20] yet underperformed the vanilla version.
Since different speech tasks focus on various characteristics of a speech
signal (e.g., speaker, language, emotion), they generally require different
levels of information. However, existing approaches have mostly investigated
speech enhancement and separation tasks, which focus primarily on the low-
level information within a speech signal. Therefore, it is still unclear how
to efficiently employ Mamba for other speech tasks, such as speech recognition
and spoken language understanding, which require high-level semantic
information within the speech signal.

In this paper, we provide solutions for applying Mamba to different speech
tasks based on their varying information requirements (in different
abstraction levels [21]) using speech recognition and speech enhancement as
examples. Our research first proposes and compares two bidirectional Mamba
(BiMamba) structures, being external BiMamba (ExtBiMamab) and inner BiMamba
(InnBiMamba). Experiments suggest that a bidirectional design can enhance the
capability of Mamba to model global dependencies within the features of a
speech signal. Mamba and BiMamba models are then evaluated independently or as
replacements for MHSA in Transformer and Conformer models across multiple
datasets. We demonstrate that the proposed BiMamba modules require additional
nonlinearity to learn high-level semantic information in speech tasks, serving
as an alternative to MHSA is thus an optimal approach to apply Mamba in this
scenario.

##  2 Preliminary

The State Space Model (SSM) based approaches, including the Structured State
Space Sequence Model (S4) [11] and Mamba [14], are derived from continuous
systems. S4 facilitates the transformation of a one-dimensional function or
sequence, $x(t)\in\mathbb{R}$ to $y(t)\in\mathbb{R}$, through an intermediate
state $h(t)$ in $\mathbb{R}^{n}$. The process leverages $A$ from
$\mathbb{R}^{n\times n}$ as the evolution matrix and $B$ from
$\mathbb{R}^{D\times N}$, $C$ from $\mathbb{R}^{D\times N}$ as the input and
output mapping matrices, respectively.

The fundamental equations are represented as:

$h^{\prime}(t)=Ah(t)+Bx(t),\quad y(t)=Ch(t)+Dx(t).$ (1)

In their discrete forms, parameter D can be represented as the residual
connection in Neural Network, S4 and Mamba introduce a scaling parameter
$\Delta$, transforming the continuous matrices $A,B$ into the discrete
matrices $\tilde{A},\tilde{B}$ respectively. This transformation commonly
utilizes the Zero-Order Hold (ZOH) method, defined by:

$\tilde{A}=\exp{\Delta A},\quad\tilde{B}=(\Delta A)^{-1}(\exp{\Delta
A}-I)\Delta B.$ (2)

Consequently, with the discretization into $\tilde{A},\tilde{B}$, the equation
(1) is adjusted for a discrete timestep $\Delta$, given as:

$h_{t}=\tilde{A}h_{t-1}+\tilde{B}x_{t},\quad y_{t}=Ch_{t}.$ (3)

The model then computes the final output from convolution via

$y=x*\bar{K},\quad\bar{K}=(C\tilde{B},C\tilde{AB},....C\tilde{A}^{L-1}\tilde{B})$
(4)

The $A$ matrix, as described above, is commonly initialized with a HiPPO
matrix [22] or a Diagonal matrix [12] to adeptly capture long-range
dependencies. Mamba enhances S4 by integrating a time-varying mechanism, which
enlarges the dimensions of matrices $B$ and $C$ to $\mathbb{R}^{B\times
L\times N}$, and modifies $\tilde{A}$ and $\tilde{B}$ to $\mathbb{R}^{B\times
L\times D\times N}$.

Although these modifications have enhanced model performance, granting it
"selective" capabilities, they do not alter the inherent nature of SSMs, which
operate in a unidirectional manner such as RNNs [23, 24]. This is not an issue
in training large language models, as many of these models are trained in an
autoregressive manner [25, 26]. However, for non-autoregressive speech models,
we require a module with causal capabilities similar to attention. Thus,
finding a suitable method to address this issue is essential. Moreover, the
observation that Mamba’s $A$ matrix randomization and real-valued diagonal
initialization perform equivalently suggests that Mamba’s ability to delineate
dependencies between inputs, compared to S4, needs further enhancement.

(a) InnBiMamba

(b) ExtBiMamba

Figure 1: The illustrations of (a) inner bidirectional Mamba (InnBiMamba)
[15], and (b) the external bidirectional Mamba (ExtBiMamba). $\sigma$ denotes
the SiLU activation.

##  3 Investigating Mamba in Speech Processing

###  3.1 Bidirecional processing

The original Mamba performs causal computations in a unidirectional manner,
using only historical information. However, in speech tasks, the model is
provided with the complete speech signal. Therefore, Mamba requires
bidirectional computations, as employed in the MHSA module, to capture global
dependencies within the features of the input signal. In this paper, we
explored two bidirectional strategies for Mamba in speech tasks, i.e., inner
bidirectional Mamba (InnBiMamba) [15] and external bidirectional Mamba
(ExtBiMamba) as shown in Figure 2.

Inner Bidirectional Layer (InnBiMamba). We first explore the inner
bidirectional Mamba (InnBiMamba) [15] for speech tasks as detailed in Figure
1(a). Here, two SSM modules share the same input and output projection layers.
The process feeds the input forward into one SSM module, while reversing the
input along the time dimension before feeding it into the other SSM module.
The output of the backward SSM module is reversed back before being combined
with the output of the forward SSM module. The combined output then passes
through the output projection layer.

External Bidirectional Layer (ExtBiMamba). In addition, we also propose a
simpler and more straight bidirectional modeling strategy, i.e., external
bidirectional Mamba (ExtBiMamba). Unlike the InnBiMamba layer, the ExtBiMamba
layer involves different input and output projection layers in forward and
backward Mamba layers, respectively, as illustrated in Figure 1(b). The input
is fed into the forward Mamba layer and the backward Mamba layer processes the
reversed input. The outputs of two Mamba layers are fused with addition
operation and a residual connection is applied around the ExtBiMamaba layer.
Please refer to Appendix A for detailed descriptions of two algorithms.

(a) Mamba model

(b) Transformer layer with Mamba

(c) Conformer layer with Mamba

Figure 2: Three applications of the Mamba layer in speech processing include:
(a) using stacked unidirectional/bidirectional Mamba layers as an alternative
to Transformer layers; (b) replacing causal and non-causal MHSA in Transformer
layer with unidirectional/bidirectional Mamba; and (c) replacing MHSA in
Conformer layer with the Mamba layer.

###  3.2 Task-aware model designs

Recent works have investigated Mamba in speech separation and speech
enhancement. These tasks primarily focus on low-level spectral information of
a speech signal [27]. In contrast, other speech tasks like speech recognition
and spoken language understanding require capturing high-level semantic
information. With reference to equation 3, SSM comprises mostly linear
computations. This implies that it has limited capability to capture high-
level information such as semantics and emotions. Although SiLU is used within
residual structures in practical implementations, this is primarily to
represent the parameter D in equation 1 for the state space model [14].
Therefore, adding more nonlinearity ability is crucial for Mamba to capture
high-level information.

To capture information of various abstraction levels, we progressively
explored three structures for increasing nonlinearity capability. As depicted
in Figure 2(a), the first strategy uses the Mamba/BiMamba layers independently
(i.e., as a direct replacement for the transformer layer) to construct the
Mamba/BiMamba model. The second approach employs the Mamba/BiMamba layer to
replace the MHSA modules within the Transformer, where the feed-forward net
(FFN) and layernorm modules are used to provide nonlinearity. The third
replaces the MHSA modules with Mamba/BiMamba layers in the Conformer, which is
a variant of the Transformer employing a convolutional layer after each MHSA
designed to additionally capture local information.

##  4 Experimental Setup

###  4.1 Speech Enhancement

Datasets. We follow the studies [28, 29] and employ the clean speech clips
from LibriSpeech train-clean-100 corpus [30] for training, comprising
$28\,539$ clips spoken by 251 speakers. The noise recordings are collected
from the following datasets [31], i.e., the noise data of the MUSAN datasets
[32], the RSG-10 dataset [33] (voice babble, F16, and factory welding are
excluded for testing), the Environmental Noise dataset [34, 35], the colored
noise set (with an $\alpha$ value ranging from -2 to 2 in increments of 0.25)
[36], the UrbanSound dataset [37] (street music recording no 26 270 is
excluded for testing), the QUT-NOISE dataset [38], and the Nonspeech dataset
[39]. Please refer to Appendix C.1 for the detailed experiment setup.

Model Configurations. In our experiments, we employ the same backbone network
architecture (a typical neural solution to speech enhancement) [31, 29, 40,
41], which comprises an input embedding layer, stacked feature transformation
layers (such as Mamba, Transformer, and Conformer layers), and an output
layer. To systematically study the Mamba networks, we use the standard
Transformer [31, 29] and Conformer [8] models as the baseline backbone
networks, across causal and non-causal configurations. Please refer to the
Appendix C.1 for detailed model configurations.

We evaluated enhanced speech with five commonly used assessment metrics, i.e,
perceptual evaluation of speech quality (PESQ) [42], extended short-time
objective intelligibility (ESTOI), and three composite metrics. For PESQ, both
wide- and narrow-band PSEQ were used to evaluate the speech quality, with the
score range of $[-0.5,\,4.5]$. The ESTOI [43] score is typically between 0 and
1. The three composite metrics [44] are used to predict the mean opinion
scores of the intrusiveness of background noise (CBAK), the signal distortion
(CSIG), and the overall signal quality (COVL), respectively, with the score
range of $[0,\,5]$. For all these five metrics, a higher score means better
performance.

###  4.2 Speech Recognition

Datasets. We evaluate our models on ASR with four datasets, i.e., LibriSpeech
[30], AN4 [45], SEAME [46], and ASRU [47], in which all speech signals are
sampled at 16 kHz. LibriSpeech (LibriSpeech960) containing approximately 1000
hours of audio recordings and their paired texts, in which a subset
LibriSpeech100 is used for ablation studies due to its higher recording
quality. The AN4 dataset contains approximately one hour of audio recordings
of primarily spoken alphanumeric strings, such as postal codes and telephone
numbers. It is employed to assess the model’s ability to perform with a small
dataset. Two English-Mandarin code-switching datasets SEAME and ASRU-CS-2019
(denoted as ASRU) are then used for a more challenging scenario compared to
monolingual. The SEAME dataset contains 200-hour spontaneous South-east Asian-
accented speech with intra- and inter-sentential code-switches, divided as
introduced in [48]. The ASRU dataset contains a 500-hour Mandarin and a
200-hour code-switching training sets recorded in mainland China, where only
the code-switching set is used for training, following [46, 47, 49].

Model Configurations. We employed the best-performing Conformer/Transformer
model configurations provided by official recipes in ESPnet or existing works.
We maintained the experimental setups of models related to Mamba the same as
Conformer/Transformer. The substitution of MHSA modules with Mamba/BiMamba is
merely implemented within the encoder layers of a joint CTC-Attention model
[50], where Transformer decoder layers serve as the ASR decoder. Detailed
model configurations are in Appendix B.

We employ word error rate (WER) and mixed word error rate (MER) to measure the
ASR performance for monolingual and code-switching ASR tasks, respectively,
where the MER considers the word error rate for English and the character
error rate for Mandarin. All experiments are performed using the ESPnet
toolkit [51].

##  5 Experimental Results and Analysis

Table 1: Comparison results of InnBiMamba and ExtBiMamba architectures in NB-PESQ, WB-PESQ, ESTOI (%), CSIG, CBAK, and COVL. The BiMamba models are denoted as “Inn/ExtBiMamba-the number of BiMamba layers”. Method | #Params | Causality | Metrics |
---|---|---|---|---
|  |  | NB-PESQ | WB-PESQ | ESTOI | CSIG | CBAK | COVL
Noisy | – | – | 1.88 | 1.24 | 56.12 | 2.26 | 1.80 | 1.67
InnBiMamba-9 | 4.48 M | ✗ | 2.84 | 2.14 | 75.74 | 3.39 | 2.59 | 2.74
ExtBiMamba-5 | 4.51 M | ✗ | 2.86 | 2.15 | 76.12 | 3.46 | 2.60 | 2.78
InnBiMamba-13 | 6.41 M | ✗ | 2.90 | 2.19 | 76.89 | 3.50 | 2.63 | 2.82
ExtBiMamba-7 | 6.26 M | ✗ | 2.90 | 2.20 | 77.04 | 3.54 | 2.64 | 2.84
Table 2: Evaluation results of InnBiMamba and ExtBiMamba in training speed (training time per step) and inference speed (RTF). | InnBiMamba-9 | ExtBiMamba-5 | InnBiMamba-13 | ExtBiMamba-7
---|---|---|---|---
sec/step | 0.159 | 0.122 | 0.212 | 0.159
RTF | $1.88\times 10^{-4}$ | $1.69\times 10^{-4}$ | $2.76\times 10^{-4}$ | $2.34\times 10^{-4}$

###  5.1 Speech Enhancement (Also c.f. Appendix C.2)

Table 3: Performance comparisons of Transformer and Mamba network architectures in NB-PESQ, WB-PESQ, ESTOI (%), CSIG, CBAK, and COVL, across causal and non-causal configurations. The Transformers with $4$ and $6$ stacked Transformer layers are denoted as Transformer-4 and Transformer-6, respectively.  Method | #Params | Causality | Metrics |
---|---|---|---|---
|  |  | NB-PESQ | WB-PESQ | ESTOI | CSIG | CBAK | COVL
Noisy | – | – | 1.88 | 1.24 | 56.12 | 2.26 | 1.80 | 1.67
Transformer-4 | 3.29 M | ✔ | 2.56 | 1.84 | 70.32 | 3.17 | 2.39 | 2.47
Mamba-4 | 1.88 M | ✔ | 2.60 | 1.87 | 70.99 | 3.17 | 2.41 | 2.48
Mamba-7 | 3.20 M | ✔ | 2.64 | 1.91 | 72.46 | 3.26 | 2.45 | 2.56
Transformer-4 | 3.29 M | ✗ | 2.74 | 2.01 | 73.44 | 3.31 | 2.50 | 2.63
ExtBiMamba-3 | 2.76 M | ✗ | 2.76 | 2.05 | 73.93 | 3.36 | 2.53 | 2.67
ExtBiMamba-4 | 3.64 M | ✗ | 2.83 | 2.11 | 75.43 | 3.46 | 2.57 | 2.75
Transformer-6 | 4.86 M | ✔ | 2.60 | 1.87 | 71.37 | 3.20 | 2.41 | 2.50
Mamba-6 | 2.76 M | ✔ | 2.63 | 1.91 | 71.94 | 3.23 | 2.44 | 2.54
Mamba-10 | 4.51 M | ✔ | 2.68 | 1.94 | 73.20 | 3.30 | 2.47 | 2.59
Transformer-6 | 4.86 M | ✗ | 2.78 | 2.05 | 74.56 | 3.38 | 2.52 | 2.69
ExtBiMamba-5 | 4.51 M | ✗ | 2.86 | 2.15 | 76.12 | 3.46 | 2.60 | 2.78
ExtBiMamba-6 | 5.39 M | ✗ | 2.88 | 2.17 | 76.69 | 3.50 | 2.62 | 2.82
Table 4: Training speed and inference speed of ExtBiMamba and non-causal Transformers. | Transformer-4 | ExtBiMamba-3 | ExtBiMamba-4 | Transformer-6 | ExtBiMamba-5 | ExtBiMamba-6
---|---|---|---|---|---|---
sec/step | 0.099 | 0.089 | 0.102 | 0.125 | 0.122 | 0.141
RTF | $1.48\times 10^{-4}$ | $1.08\times 10^{-4}$ | $1.38\times 10^{-4}$ | $2.12\times 10^{-4}$ | $1.71\times 10^{-4}$ | $2.04\times 10^{-4}$

InnBiMamba vs. ExtBiMamba. In Table 1, we illustrate the comparison results of
InnBiMamba and ExtBiMamba across different model sizes, in terms of six
metrics, i.e., NB-PESQ, WB-PESQ, ESTOI, CSIG, CBAK, and COVL. Overall,
ExtBiMamba consistently performs slightly better than InnBiMamba across the
model sizes. In addition, Table 2 compares the training speed (training time
per step) and inference speed (real-time factor [52]) of InnBiMamba and
ExtBiMamba, confirming the superiority of ExtBiMamba over InnBiMamba. Real-
time factor (RTF) is measured by dividing the time taken to process a speech
utterance by the duration of the speech. The RTFs were measured on an NVIDIA
Tesla V100 GPU and averaged over 20 executions. We used a batch size of 4
noisy mixtures, each with a duration of 10 seconds [52].

Mamba vs. Transformer. Table 3 compares Transformer and Mamba network
architectures. For Mamba models, we report the results of Mamba models with
the same number of layers and a similar model size to the Transformer. It can
be observed that the Mamba models consistently demonstrate obvious performance
superiority over the Transformer models with lower parameter overheads, across
causal and noncausal configurations. For instance, the Mamba-7 (3.20 M) and
ExtBiMamba-5 (4.51 M) improve on the causal Transformer-4 (3.29 M) and the
noncausal Transformer-6 (4.81 M) by 0.08 and 0.08, 0.07 and 0.1, 2.14% and
1.56%, 0.09 and 0.08, 0.06 and 0.08, and 0.09 and 0.09 in terms of NB-PESQ,
WB-PESQ, ESTOI, CSIG, CBAK, and COVL, respectively. In addition, ExtBiMamba
models provide substantial performance improvements over original
(unidirectional) Mamba models across all the metrics, which confirms the
effectiveness of the bidirectional modeling. ExtBiMamba-5 (4.51 M) provided
gains of 0.18 in NB-PESQ, 0.21 in WB-PESQ, 2.92% in ESTOI, 0.16 in CSIG, 0.13
in CBAK, and 0.19 in COVL over Mamba-10 (4.51 M), respectively.

Mamba vs. Conformer. Table 5 reports the comparative results of Conformer and
Mamba. We can see that original Mamba (causal) models outperform causal
Conformer models across all metrics while involving fewer parameters. For
instance, compared to causal Conformer-6 (9.26 M), Mamba-20 (8.89 M) improves
NB-PESQ by 0.04, WB-PESQ by 0.06, ESTOI by 0.68%, CSIG by 0.05, CBAK by 0.05,
and COVL by 0.06. It can also be seen that overall, ExtBiMamba models exhibit
slightly better or comparable performance to noncausal Conformer models.
Similarly, the substantial performance superiority of the bidirectional
modeling is observed from evaluation results of ExtBiMamba-6 (5.39) vs.
Mamba-13 (5.83M) and ExtBiMamba-10 (8.89 M) vs. Mamba-20 (8.89 M).

Table 5: Speech enhancement performance comparisons of Conformer and Mamba network architectures in NB-PESQ, WB-PESQ, ESTOI (%), CSIG, CBAK, and COVL, across causal and noncausal configurations. The Conformer models with $4$ and $6$ Conformer layers are denoted as Conformer-4 and Conformer-6, respectively. Method | #Params | Causality | Metrics |
---|---|---|---|---
|  |  | NB-PESQ | WB-PESQ | ESTOI | CSIG | CBAK | COVL
Noisy | – | – | 1.88 | 1.24 | 56.12 | 2.26 | 1.80 | 1.67
Conformer-4 | 6.22 M | ✔ | 2.67 | 1.94 | 72.78 | 3.30 | 2.46 | 2.59
Mamba-13 | 5.83 M | ✔ | 2.70 | 1.97 | 73.53 | 3.32 | 2.50 | 2.62
Conformer-4 | 6.22 M | ✗ | 2.88 | 2.17 | 76.68 | 3.51 | 2.61 | 2.82
ExtBiMamba-6 | 5.39 M | ✗ | 2.88 | 2.18 | 76.73 | 3.50 | 2.62 | 2.82
ExtBiMamba-7 | 6.26 M | ✗ | 2.90 | 2.20 | 77.04 | 3.54 | 2.64 | 2.84
Conformer-6 | 9.26 M | ✔ | 2.68 | 1.94 | 73.41 | 3.30 | 2.46 | 2.59
Mamba-20 | 8.89 M | ✔ | 2.72 | 2.00 | 74.09 | 3.35 | 2.51 | 2.65
Conformer-6 | 9.26 M | ✗ | 2.91 | 2.20 | 77.56 | 3.54 | 2.62 | 2.84
ExtBiMamba-10 | 8.89 M | ✗ | 2.91 | 2.20 | 77.64 | 3.59 | 2.65 | 2.87
Table 6: Training speed and inference speed of ExtBiMamba and non-causal Conformers. | Conformer-4 | ExtBiMamba-6 | ExtBiMamba-7 | Conformer-6 | ExtBiMamba-10
---|---|---|---|---|---
sec/step | 0.160 | 0.142 | 0.156 | 0.215 | 0.206
RTF | $2.27\times 10^{-4}$ | $2.07\times 10^{-4}$ | $2.37\times 10^{-4}$ | $3.27\times 10^{-4}$ | $3.32\times 10^{-4}$

Mamba vs. MHSA. We also explore replacing the MHSA with the Mamba layer in
Transformer and Conformer. Tables 12-13 present the evaluation results of
replacing the MHSA module in Transformer and Conformer with the Mamba layer.
Due to space constraints, Tables 12-13 are placed in Appendix C. It can be
observed that Transformers substantially benefit from using the Mamba and
BiMamba layers. TransMamba-4 and TransExtBiMamba-6 improve over causal
Transformer-4 and non-causal Transformer-6 by 0.09 and 0.13 in NB-PESQ, 0.09
and 0.16 in WB-PESQ, 1.96% and 4.04% in ESTOI, 0.07 and 0.22 in CSIG, 0.06 and
0.13 in CBAK, and 0.08 and 0.19 in COVL, respectively. In addition,
TransMamba-4 (3.99 M) and TransInnBiMamba-4 (4.17 M) outperform causal
Transformer-6 (4.86 M) and non-causal Transformer (4.86 M), respectively,
which further confirms the superiority of Mamba layer over MHSA. Among
InnBiMamba and ExtBiMamba, we observe that TransExtBiMamba performs slightly
better than TransInnBiMamba. With fewer parameters, TransExtBiMamba-4 (5.74 M)
achieves slightly higher scores in WB-PESQ, CSIG, CBAK, and COVL, but slightly
lower scores in NB-PESQ and ESTOI than TransInnBiMamba-6 (6.19 M).

As shown in Table 13, we observe that the Conformer architecture can benefit
from the use of Mamba and ExtBiMamba. The ConExtBiMamba-4 and ConExtBiMamba-6
outperform non-causal Conformer-4 and Conformer-6 by 0.04 and 0.02 in NB-PESQ,
0.03 and 0.03 in WB-PESQ, 1.06% and 0.65% in ESTOI, 0.06 and 0.06 in CSIG,
0.03 and 0.03 in CBAK, and 0.06 and 0.05 in COVL, respectively. In addition,
ConExtBiMamba-4 (8.67 M) also exhibits a slightly better performance than non-
causal Conformer-6 (9.26 M). Among InnBiMamba and ExtBiMamba, similarly,
ConExtBiMamba performs better than ConvInnBiMamba.

###  5.2 Speech Recognition

Independent vs. Substitute for MHSA. Table 7 reports the performance of Mamba
when used independently and as a replacement for MHSA across various datasets.
We can observe that independent Mamba and BiMamba models exhibit significantly
lower performance compared to the Transformer and Conformer models (with the
same number of layers). Since the Mamba has fewer parameters when configured
with the same number of layers as the Transformer and Conformer, we increased
its number of layers (shown in Appendix D.1) to match the number of parameters
with the Conformer to further evaluate Mamba for ASR. The performance of
Mamba, however, remained undesirable.

Table 7: ASR results for the different datasets without language model in WER/MER (%). “ESPnet” means the best result reported by ESPnet official or other paper experimented with ESPnet, where the SEAME and ASRU results come from [53, 54, 49]. Method |  | LibriSpeech-100 | LibriSpeech-960 | SEAME | ASRU
---|---|---|---|---|---
|  | dev | test | dev | test | man | sge | dev | test
ESPnet |  |  |  |  |  |  |  |  |
Conformer |  | 6.3 | 6.5 | 2.1 | 2.4 | 16.6 | 23.3 | - | 12.2
Branchformer |  | 6.1 | 6.3 | 2.2 | 2.4 | - | - | - | -
Reproduced Results |  |  |  |  |  |  |
Branchformer |  | 6.3 | 6.4 | 2.2 | 2.4 | 16.3 | 23.2 | 12.5 | 11.8
Mamba |  | 40.8 | 40 | 21.8 | 22.3 | 44.5 | 55.3 | 38.0 | 36.3
InnBiMamba |  | 39.6 | 38.2 | 21.8 | 22.5 | 44.3 | 55.4 | 38.4 | 37.9
ExtBiMamba |  | 38.5 | 37.7 | 21.6 | 22.1 | 44.4 | 55.2 | 38.2 | 36.8
Transformer |  | 8.0 | 8.4 | 2.8 | 3.2 | 17.7 | 24.5 | 13.7 | 13.1
$-$ MHSA $+$ Mamba |  | 10.9 | 11.2 | 3.2 | 3.5 | 20.7 | 29.5 | 24.2 | 23.1
$-$ MHSA $+$ InnBiMamba |  | 8.8 | 9.4 | 2.5 | 3.0 | 18.4 | 26.0 | 20.2 | 19.5
$-$ MHSA $+$ ExtBiMamba |  | 8.4 | 8.7 | 2.5 | 2.8 | 17.2 | 24.3 | 18.7 | 18.0
Conformer |  | 6.3 | 6.5 | 2.3 | 2.6 | 16.9 | 23.6 | 12.8 | 12.2
$-$ MHSA $+$ Mamba |  | 6.6 | 6.9 | 2.6 | 2.9 | 17.7 | 24.9 | 13.5 | 12.9
$-$ MHSA $+$ InnBiMamba |  | 6.0 | 6.4 | 2.1 | 2.3 | 17.1 | 23.8 | 12.7 | 12.2
$-$ MHSA $+$ ExtBiMamba |  | 5.9 | 6.0 | 2.0 | 2.3 | 16.6 | 23.4 | 12.3 | 11.5
Table 8: WER results on LibriSpeech (without external language model) across various parameter counts and frameworks. Method | Params (M) | dev | dev other | test | test other
---|---|---|---|---|---
LiberSpeech100 |  |
Transformer | 29.38 | 8.0 | 20.0 | 8.4 | 20.2
$-$ MHSA $+$ InnBiMamba | 33.32 | 8.8 | 23.3 | 9.4 | 23.6
$-$ MHSA $+$ ExtBiMamba | 40.42 | 8.4 | 22.4 | 8.7 | 23.1
Conformer Large | 42.17 | 6.2 | 17.4 | 6.4 | 17.3
Conformer | 34.23 | 6.3 | 17.4 | 6.5 | 17.3
$-$ MHSA $+$ InnBiMamba | 36.89 | 6.0 | 17.4 | 6.4 | 17.6
$-$ MHSA $+$ InnBiMamba Large | 49.90 | 6.1 | 17.3 | 6.3 | 17.3
$-$ MHSA $+$ ExtBiMamba | 41.59 | 5.9 | 17.1 | 6.0 | 17.2
LiberSpeech960 |  |
Transformer | 99.36 | 2.8 | 7.6 | 3.2 | 7.5
$-$ MHSA $+$ InnBiMamba | 103.35 | 2.5 | 7.4 | 3.0 | 7.3
$-$ MHSA $+$ ExtBiMamba | 110.04 | 2.5 | 6.9 | 2.8 | 6.9
Conformer | 116.15 | 2.3 | 5.5 | 2.6 | 5.6
$-$ MHSA $+$ InnBiMamba | 118.81 | 2.1 | 5.6 | 2.4 | 5.5
$-$ MHSA $+$ ExtBiMamba | 123.51 | 2.0 | 5.4 | 2.3 | 5.4

In contrast to the independent Mamba/BiMamba, we observe a significant
improvement in the ASR task when Mamba/BiMamba is used as a replacement for
MHSA. Specifically, replacing MHSA with ExtBiMamba in the Conformer (named
ConExtBiMamba) exhibits higher performance than the SOTA performance achieved
by Conformer and Branchformer across multiple datasets with the same training
setups. Additionally, ConExtBiMamba provides faster training and inference
speeds compared to the Conformer model as detailed in Appendix D.3 Table 18.
When we replaced MHSA with ExtBiMamba, the number of parameters exceeded that
of the original Conformer. To eliminate the possibility that the performance
improvement results from the higher number of parameters, we increase the size
of Conformer to a similar number of parameters as ConExtBiMamba, and present
the results in Table 8. We found that although the Conformer’s performance
improves, it still underperforms ConExtBiMamba and ConInnBiMamba, supporting
the effectiveness of our approach.

Unidirectional vs. Bidirectional. Tables 7-8 present the evaluation results of
unidirectional Mamba and two bidirectional Mamba modules (i.e., InnBiMamba,
and ExtBiMamba) with their corresponding model sizes. InnBiMamba and
ExtBiMamba provide substantial improvements over Mamba when used as a
replacement for the MHSA module within the Transformer and Conformer. This
confirms the effectiveness of the bidirectional modeling.

In addition, ExtBiMamba consistently outperforms InnBiMamba across various
frameworks and datasets. We next compare ExtBiMamba with Conformer-Large and
ConInnBiMamba-Large as shown in Table 8. The results demonstrate that the
performance improvement for ASR stems from factors beyond merely an increased
number of model parameters.

Ablation Study on ConExtBiMamba. We employ the ConExtBiMamba model, which
improves the SOTA result achieved by Branchformer in the ASR task, for
ablation studies. Firstly, we found that within the ConExtBiMamba framework,
variations in BiMamba’s hyperparameters had little impact on the performance
of the model. Secondly, introducing Gaussian Noise to the $A$ matrix within
Mamba in ConBiMamba enhances the model’s effectiveness (for a detailed
discussion, see Appendix E.1).

We further discovered that within the Conformer structure, the Swish
activation and Macaron-style feed-forward layers enhance the performance of
ConExtBiMamba. However, positional encoding and dropout had little effect on
the model (for a detailed discussion, see Appendix E.2).

We also found that the robustness of the ConExtBiMamba model was much stronger
than that of the Conformer, particularly on very small datasets where its
performance was more stable. In tests conducted on the AN4 dataset using five
random seeds, ConExtBiMamba outperformed the Conformer in every trial. The
mean WER for the Conformer was 10.2, while for ConExtBiMamba it was just 4.54,
with a WER variance of only 1.1104 compared with the Conformer’s significantly
higher variance of 77.71 (for a detailed discussion, see Appendix E.3).

(a) BiMamba

(b) BiMamba with FFN

(c) BiMamba

(d) BiMamba with FFN

Figure 3: Decision Boundaries for BiMamba and BiMamaba with Feed-forward
layer(FFN)

##  6 Discussion

As demonstrated in Section 5.2, independent Mamba or BiMamba models exhibit
low performance in the ASR task, while replacing MHSA with BiMamba layers
demonstrates impressive performance and outperforms the vanilla Transformer
and Conformer models. Since the latter approach achieves significantly higher
ASR performance by additionally employing a feed-forward network (FFN) and a
residual connection compared to the independent Mamba and BiMamba models, we
then explore the factors contributing to this performance improvement via
ablation studies in Table 16.

We assume that extracting high-abstraction-level information requires greater
nonlinearity than capturing low-abstraction-level information. Specifically,
ASR models transcribe speech signals by understanding the context of acoustic
features and aligning speech to tokens, corresponding to low-level sequential
and high-level semantic information, respectively. Since a speech enhancement
model focuses on low-abstraction-level spectral information, an ASR model may
need higher nonlinear ability than a speech enhancement model. As illustrated
in Section 2, we consider a BiMamba layer as a weakly nonlinear module similar
to MHSA [55]. In Figure. 3, we use the BiMamba model to find the decision
boundary for data with simple and complex distribution, respectively. We
observe that BiMamba struggles to find the decision boundary for data of a
complex distribution without the aid of an FFN (with ReLU activation similar
to that in Transformer), which validates our assumption.

In addition, results in Table 16 indicate the effectiveness of the FFN and
residual connection in a Transformer model for ASR. Similar to independent
ExtBiMamba, removing the residual connection and FFN from a Transformer model
leads to gradient vanishing and decreases nonlinearity, resulting in
significant performance degradation for ASR. This further underpins that using
BiMamba layers as a replacement for MHSA is more appropriate for speech tasks
which require models to learn high-abstraction-level information than
employing it independently.

##  7 Conclusion

In this paper, we explore the use of Mamba in speech processing, for tasks
requiring information from low to high abstraction levels. We first compared
two bidirectional designs for Mamba and next employed them independently or as
a replacement for MHSA in Transformer and Conformer models. While independent
BiMamba models exhibited high performance in the speech enhancement task with
the ability to capture low-abstraction-level spectral information, it can not
well achieve speech recognition, which requires semantic information within
the speech signal. In contrast, using BiMamba as a replacement of MHSA in
Conformer (i.e., ConExtBiMamba) matched or exceeded the performance of the
SOTA Branchformer across multiple datasets. Ablation studies suggest that
using BiMamba to replace MHSA is more appropriate for tasks requiring high-
abstract-level information due to greater nonlinearity compared to independent
BiMamba.

## Limitation and Broader Impacts

There are numerous datasets for Speech Recognition and Speech Enhancement, and
we cannot guarantee that our model will outperform the Conformer across all
datasets. However, we have conducted experiments on a variety of datasets,
including laboratory speech, multilingual speech, multilingual recordings from
real-world environments, and extremely small datasets. These cover most
scenarios and sufficiently demonstrate the effectiveness of our model.

Our research enhances the performance of speech recognition and speech
enhancement, but more importantly, it provides guidance on how to use Mamba in
the speech domain, especially in areas requiring semantic information.
Potential negative impacts include further displacement of manual
transcription due to improved model performance. However, currently, there are
few people engaged in manual annotation, and our model also has the potential
to create more job opportunities in the field of artificial intelligence.

## Acknowledgement

This work was supported by the Australian Research Council Discovery Project
DP230101184.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  * [4] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [5] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [6] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  Gpt understands, too.  AI Open, 2023.
  * [7] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [8] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In Proc. Interspeech 2020, pages 5036–5040, 2020.
  * [9] Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe.  Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding.  In International Conference on Machine Learning, pages 17627–17643. PMLR, 2022.
  * [10] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  Advances in neural information processing systems, 34:572–585, 2021.
  * [11] Albert Gu, Karan Goel, and Christopher Re.  Efficiently modeling long sequences with structured state spaces.  In International Conference on Learning Representations, 2021.
  * [12] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  Advances in Neural Information Processing Systems, 35:22982–22994, 2022.
  * [13] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré.  On the parameterization and initialization of diagonal state space models.  Advances in Neural Information Processing Systems, 35:35971–35983, 2022.
  * [14] Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  arXiv preprint arXiv:2312.00752, 2023.
  * [15] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.  Vision mamba: Efficient visual representation learning with bidirectional state space model.  arXiv preprint arXiv:2401.09417, 2024.
  * [16] Yijun Yang, Zhaohu Xing, and Lei Zhu.  Vivim: a video vision mamba for medical video object segmentation.  arXiv preprint arXiv:2401.14168, 2024.
  * [17] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al.  Jamba: A hybrid transformer-mamba language model.  arXiv preprint arXiv:2403.19887, 2024.
  * [18] Xilin Jiang, Cong Han, and Nima Mesgarani.  Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation.  arXiv preprint arXiv:2403.18257, 2024.
  * [19] Kai Li and Guo Chen.  Spmamba: State-space model is all you need in speech separation.  arXiv preprint arXiv:2404.02063, 2024.
  * [20] Changsheng Quan and Xiaofei Li.  Multichannel long-term streaming neural speech enhancement for static and moving speakers.  arXiv preprint arXiv:2403.07675, 2024.
  * [21] Haizhou Li, Bin Ma, and Kong Aik Lee.  Spoken language recognition: from fundamentals to practice.  Proc. IEEE, 101(5):1136–1159, 2013.
  * [22] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.  Hippo: Recurrent memory with optimal polynomial projections.  Advances in neural information processing systems, 33:1474–1487, 2020.
  * [23] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986.  Biometrika, 71:599–607, 1986.
  * [24] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [25] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training, 2018.
  * [26] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019.
  * [27] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022.
  * [28] Qiquan Zhang, Xinyuan Qian, Zhaoheng Ni, Aaron Nicolson, Eliathamby Ambikairajah, and Haizhou Li.  A time-frequency attention module for neural speech enhancement.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:462–475, 2023.
  * [29] Qiquan Zhang, Meng Ge, Hongxu Zhu, Eliathamby Ambikairajah, Qi Song, Zhaoheng Ni, and Haizhou Li.  An empirical study on the impact of positional encoding in transformer-based monaural speech enhancement.  In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1001–1005, 2024.
  * [30] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In Proc. ICASSP, pages 5206–5210, 2015.
  * [31] Qiquan Zhang, Xinyuan Qian, Zhaoheng Ni, Aaron Nicolson, Eliathamby Ambikairajah, and Haizhou Li.  A time-frequency attention module for neural speech enhancement.  IEEE/ACM TASLP, 31:462–475, 2023.
  * [32] David Snyder, Guoguo Chen, and Daniel Povey.  MUSAN: A music, speech, and noise corpus.  arXiv preprint arXiv:1510.08484, 2015.
  * [33] Herman JM Steeneken and Frank WM Geurtsen.  Description of the RSG-10 noise database.  report IZF, 3:1988, 1988.
  * [34] Fatemeh Saki and Nasser Kehtarnavaz.  Automatic switching between noise classification and speech enhancement for hearing aid devices.  In in Proc. EMBC, pages 736–739, 2016.
  * [35] Fatemeh Saki, Abhishek Sehgal, Issa Panahi, and Nasser Kehtarnavaz.  Smartphone-based real-time classification of noise signals using subband features and random forest classifier.  In Proc. ICASSP, pages 2204–2208, 2016.
  * [36] Qiquan Zhang, Aaron Nicolson, Mingjiang Wang, Kuldip K Paliwal, and Chenxu Wang.  DeepMMSE: A deep learning approach to mmse-based noise power spectral density estimation.  IEEE/ACM TASLP, 28:1404–1415, 2020.
  * [37] J. Salamon, C. Jacoby, and J. P. Bello.  A dataset and taxonomy for urban sound research.  In Proc. ACM-MM, pages 1041–1044, 2014.
  * [38] David B Dean, Sridha Sridharan, Robert J Vogt, and Michael W Mason.  The QUT-NOISE-TIMIT corpus for the evaluation of voice activity detection algorithms.  In Proc. INTERSPEECH, 2010.
  * [39] Guoning Hu and DeLiang Wang.  A tandem algorithm for pitch estimation and voiced speech segregation.  IEEE Transactions on Audio, Speech, and Language Processing, 18(8):2067–2079, 2010.
  * [40] Qiquan Zhang, Hongxu Zhu, Qi Song, Xinyuan Qian, Zhaoheng Ni, and Haizhou Li.  Ripple sparse self-attention for monaural speech enhancement.  In Proc. ICASSP, pages 1–5, 2023.
  * [41] Aaron Nicolson and Kuldip K Paliwal.  Masked multi-head self-attention for causal speech enhancement.  Speech Com., 125:80–96, 2020.
  * [42] Recommendation ITU-T P ITU.  862.2: Wideband extension to recommendation P. 862 for the assessment of wideband telephone networks and speech codecs. ITU-Telecommunication standardization sector, 2007.
  * [43] Jesper Jensen and Cees H Taal.  An algorithm for predicting the intelligibility of speech masked by modulated noise maskers.  IEEE/ACM TASLP, 24(11):2009–2022, 2016.
  * [44] Yi Hu and Philipos C Loizou.  Evaluation of objective quality measures for speech enhancement.  IEEE Trans. Audio, Speech, Lang. process., 16(1):229–238, 2007.
  * [45] Alejandro Acero and Richard M Stern.  Environmental robustness in automatic speech recognition.  In International Conference on Acoustics, Speech, and Signal Processing, pages 849–852. IEEE, 1990.
  * [46] Dau-Cheng Lyu, Tien-Ping Tan, Eng Siong Chng, and Haizhou Li.  Seame: a mandarin-english code-switching speech corpus in south-east asia.  In Eleventh Annual Conference of the International Speech Communication Association, 2010.
  * [47] Xian Shi, Qiangze Feng, and Lei Xie.  The asru 2019 mandarin-english code-switching speech recognition challenge: Open datasets, tracks, methods and results.  arXiv preprint arXiv:2007.05916, 2020.
  * [48] Zhiping Zeng, Yerbolat Khassanov, Van Tung Pham, Haihua Xu, Eng Siong Chng, and Haizhou Li.  On the end-to-end solution to mandarin-english code-switching speech recognition.  In Proc. Interspeech, pages 2165–2169, 2019.
  * [49] Hexin Liu, Xiangyu Zhang, Leibny Paola Garcia, Andy WH Khong, Eng Siong Chng, and Shinji Watanabe.  Aligning speech to languages to enhance code-switching speech recognition.  arXiv preprint arXiv:2403.05887, 2024.
  * [50] Suyoun Kim, Takaaki Hori, and Shinji Watanabe.  Joint ctc-attention based end-to-end speech recognition using multi-task learning.  In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4835–4839. IEEE, 2017.
  * [51] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai.  ESPnet: End-to-end speech processing toolkit.  In Proceedings of Interspeech, pages 2207–2211, 2018.
  * [52] Zhifeng Kong, Wei Ping, Ambrish Dantrey, and Bryan Catanzaro.  Speech denoising in the waveform domain with self-attention.  In Proc. ICASSP, pages 7867–7871, 2022.
  * [53] Hexin Liu, Haihua Xu, Leibny Paola Garcia, Andy W. H. Khong, Yi He, and Sanjeev Khudanpur.  Reducing language confusion for code-switching speech recognition with token-level language diarization.  In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pages 1–5, 2023.
  * [54] Hexin Liu, Leibny Paola Garcia, Xiangyu Zhang, Andy WH Khong, and Sanjeev Khudanpur.  Enhancing code-switching speech recognition with interactive language biases.  In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pages 10886–10890. IEEE, 2024.
  * [55] Christopher Manning.  Lecture 8: Transformers.  CS224N: Natural Language Processing with Deep Learning, Stanford University, 2024.  Accessed: 2024-05-14, Slide 32.
  * [56] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  arXiv preprint arXiv:1804.03619, 2018.
  * [57] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [58] Philipp Koehn.  Neural machine translation.  Cambridge University Press, 2020.

##  Appendix A Algorithms of Bidirectional Mamba (InnBiMamba and ExtBiMamba)

Algorithm 1 InnBiMamba Layer Workflow

0: $\mathbf{H}_{l-1}$: (B,L,D)

0: $\mathbf{H}_{l}$: (B,L,D)

1:
$\mathbf{H}_{l-1}^{\prime}:\texttt{(B,L,D)}\leftarrow\textbf{Norm}(\mathbf{H}_{l-1})$

2:
$\mathbf{x}:\texttt{(B,L,E)}\leftarrow\textbf{Linear}^{\mathbf{x}}(\mathbf{H}_{l-1}^{\prime})$

3:
$\mathbf{z}:\texttt{(B,L,E)}\leftarrow\textbf{Linear}^{\mathbf{z}}(\mathbf{H}_{l-1}^{\prime})$

4: for $o\in\\{\text{forward},\text{backward}\\}$ do

5:
$\mathbf{x}_{o}^{\prime}:\texttt{(B,L,E)}\leftarrow\textbf{SiLU}(\textbf{Conv1d}_{o}(\mathbf{x}))$

6:
$\mathbf{B}_{o}:\texttt{(B,L,N)}\leftarrow\textbf{Linear}_{o}^{\mathbf{B}}(\mathbf{x}_{o}^{\prime})$

7:
$\mathbf{C}_{o}:\texttt{(B,L,N)}\leftarrow\textbf{Linear}_{o}^{\mathbf{C}}\left({\mathbf{x}_{o}}^{\prime}\right)$

8:
$\boldsymbol{\Delta}_{o}:\texttt{(B,L,E)}\leftarrow\log\left(1+\exp\left(\right.\right.\textbf{Linear}_{o}^{\Delta}\left(\mathbf{x}_{o}^{\prime}\right)+\textbf{Parameter}_{o}^{\Delta}))$

9:
$\overline{\mathbf{A}}_{o}:\texttt{(B,L,E,N)}\leftarrow\boldsymbol{\Delta}_{o}\boldsymbol{\bigotimes}\textbf{
Parameter}_{o}^{\mathbf{A}}$

10:
$\overline{\mathbf{B}}_{o}:\texttt{(B,L,E,N)}\leftarrow\boldsymbol{\Delta}_{o}\boldsymbol{\bigotimes}\,\mathbf{B}_{o}$

11:
$\mathbf{y}_{o}:\texttt{(B,L,E)}\leftarrow\mathbf{SSM}\left(\overline{\mathbf{A}}_{o},\overline{\mathbf{B}}_{o},\mathbf{C}_{o}\right)\left(\mathbf{x}_{o}^{\prime}\right)$

12: end for

13: $y^{\prime}_{\text{forward}}:\texttt{(B,L,E)}\leftarrow
y_{\text{forward}}\odot\textbf{SiLU}(\mathbf{z})$

14: $y^{\prime}_{\text{backward}}:\texttt{(B,L,E)}\leftarrow
y_{\text{backward}}\odot\textbf{SiLU}(\mathbf{z})$

15:
$\mathbf{H}_{l}:\texttt{(B,L,D)}\leftarrow\textbf{Linear}^{\mathbf{H}}(y^{\prime}_{\text{forward}}+y^{\prime}_{\text{backward}})+\mathbf{H}_{l-1}$

16: return $\mathbf{H}_{l}$

Algorithm 2 ExtBiMamba Layer Workflow

0: $\mathbf{H}_{l-1}$: (B,L,D)

0: $\mathbf{H}_{l}$: (B,L,D)

1:
$\mathbf{H}_{l-1}^{\prime}:\texttt{(B,L,D)}\leftarrow\textbf{Norm}(\mathbf{H}_{l-1})$

2: for $o\in\\{\text{forward},\text{backward}\\}$ do

3:
$\mathbf{x}_{o}:\texttt{(B,L,E)}\leftarrow\textbf{Linear}_{o}^{\mathbf{x}}(\mathbf{H}_{l-1}^{\prime})$

4:
$\mathbf{z}_{o}:\texttt{(B,L,E)}\leftarrow\textbf{Linear}_{o}^{\mathbf{z}}(\mathbf{H}_{l-1}^{\prime})$

5:
$\mathbf{x}_{o}^{\prime}:\texttt{(B,L,E)}\leftarrow\textbf{SiLU}(\textbf{Conv1d}_{o}(\mathbf{x}_{o}))$

6:
$\mathbf{B}_{o}:\texttt{(B,L,N)}\leftarrow\textbf{Linear}_{o}^{\mathbf{B}}(\mathbf{x}_{o}^{\prime})$

7:
$\mathbf{C}_{o}:\texttt{(B,L,N)}\leftarrow\textbf{Linear}_{o}^{\mathbf{C}}\left({\mathbf{x}_{o}}^{\prime}\right)$

8:
$\boldsymbol{\Delta}_{o}:\texttt{(B,L,E)}\leftarrow\log\left(1+\exp\left(\right.\right.\textbf{Linear}_{o}^{\Delta}\left(\mathbf{x}_{o}^{\prime}\right)+\textbf{Parameter}_{o}^{\Delta}))$

9:
$\overline{\mathbf{A}}_{o}:\texttt{(B,L,E,N)}\leftarrow\boldsymbol{\Delta}_{o}\boldsymbol{\bigotimes}\textbf{
Parameter}_{o}^{\mathbf{A}}$

10:
$\overline{\mathbf{B}}_{o}:\texttt{(B,L,E,N)}\leftarrow\boldsymbol{\Delta}_{o}\boldsymbol{\bigotimes}\,\mathbf{B}_{o}$

11:
$\mathbf{y}_{o}:\texttt{(B,L,E)}\leftarrow\mathbf{SSM}\left(\overline{\mathbf{A}}_{o},\overline{\mathbf{B}}_{o},\mathbf{C}_{o}\right)\left(\mathbf{x}_{o}^{\prime}\right)$

12:
$\mathbf{y}_{o}^{\prime}:\texttt{(B,L,D)}\leftarrow\textbf{Linear}_{o}^{\mathbf{H}}(\mathbf{y}_{o}\bigodot\textbf{SiLU}(\mathbf{z}_{o}))$

13: end for

14:
$\mathbf{H}_{l}:\texttt{(B,L,D)}\leftarrow(y^{\prime}_{\text{forward}}+y^{\prime}_{\text{backward}})+\textbf{H}_{l-1}$

15: return $\mathbf{H}_{l}$

Algorithms 1 and 2 illustrate the workflows of InnBiMamba and ExtBiMamba,
respectively. The main differences are highlighted in violet. For the
InnBiMamba layer, the linear input projections ($\textbf{Linear}^{\mathbf{x}}$
and $\textbf{Linear}^{\mathbf{z}}$) and the linear output projection
$\textbf{Linear}^{\mathbf{H}}$ are shared across forward and backward
operations. In contrast, the ExtBiMamba layer uses different input linear
projections ($\textbf{Linear}_{o}^{\mathbf{x}}$ and
$\textbf{Linear}_{o}^{\mathbf{z}}$) and output linear projections
($\textbf{Linear}_{o}^{\mathbf{H}}$) across forward and backward operations.

##  Appendix B Model Configurations for ASR

For Mamba and BiMamba models, we employ the default hyper-parameters [14]: the
state dimension $d_{\text{state}}\\!=\\!16$, convolution dimension
$d_{\text{conv}}\\!=\\!4$, and the expansion factor was set to $2$. To keep
the same batch size as in the official implementation of ESPnet, the
experiments on the LibriSpeech-960 dataset were performed on an NVIDIA A100
80GB. All the other experiments were run on an NVIDIA V100 32GB. The detailed
model configurations for each dataset are given in Tables 9, 10 and 11.

Table 9: Implementation details in different tasks and datasets. We show the best Transformer configurations reported in ESPnet Official | LibriSpeech100 | LibriSpeech960 | ASRU | SEAME
---|---|---|---|---
Frontend
window length | 400 | 400 | 400 | 400
hop length | 160 | 128 | 160 | 160
SpecAug
time warp window | 5 | 5 | 5 | 5
num of freq masks | 2 | 2 | 2 | 2
freq mask width | (0, 27) | (0, 30) | (0, 27) | (0, 27)
num of time masks | 2 | 2 | 2 | 2
time mask width or ratio range | (0, 0.05) | (0, 40) | (0, 0.05) | (0, 0.05)
Architecture
feature size $d$ | 256 | 512 | 256 | 256
hidden size $d_{\text{hidden}}$ | 1024 | 2048 | 1024 | 2048
attention heads $h$ | 4 | 8 | 4 | 4
num of encoder layers | 18 | 18 | 18 | 24
depth-wise conv kernel | 31 | 31 | 31 | 31
Training
epochs | 70 | 100 | 70 | 70
learning rate | 2e-3 | 2e-3 | 2e-3 | 2e-3
warmup steps | 15k | 25k | 15k | 15k
weight decay | 1e-6 | 1e-6 | 1e-6 | 1e-6
dropout rate | 0.1 | 0.1 | 0.1 | 0.1
ctc weight | 0.3 | 0.3 | 0.3 | 0.3
label smoothing weight | 0.1 | 0.1 | 0.1 | 0.1
Table 10: Implementation details in different tasks and datasets. We show the best Conformer configurations reported in ESPnet Official  | LibriSpeech100 | LibriSpeech960 | ASRU | SEAME
---|---|---|---|---
Frontend
window length | 400 | 400 | 400 | 400
hop length | 160 | 160 | 160 | 160
SpecAug
time warp window | 5 | 5 | 5 | 5
num of freq masks | 2 | 2 | 2 | 2
freq mask width | (0, 27) | (0, 27) | (0, 30) | (0, 30)
num of time masks | 2 | 2 | 2 | 2
time mask width or ratio range | (0, 0.05) | (0, 0.05) | (0, 40) | (0, 40)
Architecture
feature size $d$ | 256 | 512 | 256 | 256
hidden size $d_{\text{hidden}}$ | 1024 | 2048 | 2048 | 2048
attention heads $h$ | 4 | 8 | 4 | 4
num of encoder layers | 12 | 12 | 12 | 12
depth-wise conv kernel | 31 | 31 | 31 | 31
Training
epochs | 120 | 50 | 70 | 70
learning rate | 2e-3 | 2.5e-3 | 1e-3 | 1e-3
warmup steps | 15k | 40k | 25k | 25k
weight decay | 1e-6 | 1e-6 | 1e-6 | 1e-6
dropout rate | 0.1 | 0.1 | 0.1 | 0.1
ctc weight | 0.3 | 0.3 | 0.3 | 0.3
label smoothing weight | 0.1 | 0.1 | 0.1 | 0.1
Table 11: Implementation details in different tasks and datasets. We show the best branchformer configurations reported in ESPnet Official  | LibriSpeech100 | LibriSpeech960 | ASRU | SEAME
---|---|---|---|---
Frontend
window length | 400 | 400 | 400 | 400
hop length | 160 | 160 | 160 | 160
SpecAug
time warp window | 5 | 5 | 5 | 5
num of freq masks | 2 | 2 | 2 | 2
freq mask width | (0, 27) | (0, 27) | (0, 27) | (0, 27)
num of time masks | 2 | 2 | 2 | 2
time mask width or ratio range | (0, 0.05) | (0, 0.05) | (0, 0.05) | (0, 0.05)
Architecture
feature size $d$ | 256 | 512 | 256 | 256
hidden size $d_{\text{hidden}}$ | 1024 | 2048 | 2048 | 1024
attention heads $h$ | 4 | 8 | 4 | 4
num of encoder layers | 12 | 18 | 12 | 12
depth-wise conv kernel | 31 | 31 | 31 | 31
Training
epochs | 70 | 70 | 70 | 70
learning rate | 2e-3 | 2.5e-3 | 2e-3 | 2e-3
warmup steps | 15k | 40k | 15k | 15k
weight decay | 1e-6 | 1e-6 | 1e-6 | 1e-6
dropout rate | 0.1 | 0.1 | 0.1 | 0.1
ctc weight | 0.3 | 0.3 | 0.3 | 0.3
label smoothing weight | 0.1 | 0.1 | 0.1 | 0.1

##  Appendix C Experiments on Speech Enhancement

###  C.1 Detailed Experimental Setup

Data Generation. Noise recordings that exceeded 30 seconds in duration were
split into clips of $30$ seconds or less. This yielded $6\,809$ noise clips,
with each clip less than or equal to $30$ seconds in duration. For validation
experiments, $1\,000$ clean speech and noise clips (without replacement) were
randomly drawn from the aforementioned clean speech and noise sets and mixed
to generate a validation set of $1\,000$ noisy clips, where each clean speech
clip was degraded by a random section of one noise clip at a random SNR level
(sampled between $-10$ and $20$ dB, in 1 dB steps). For evaluation
experiments, we employed four real-world noise sources (excluded from the
training set) including two non-stationary and two colored ones. The two non-
stationary noise sources were the voice babble from the RSG-10 noise dataset
[33] and street music from the Urban Sound dataset [37]. The two colored noise
sources were F16 and factory welding from RSG-10 noise dataset [33]. For each
of the four noises, we randomly picked twenty clean speech clips (without
replacement) from the test-clean-100 of LibriSpeech corpus [30] and degraded
each clip with a random section of the noise clip at the five SNR levels,
i.e., $\left\\{-5\text{ dB},0\text{ dB},5\text{ dB},10\text{ dB},15\text{
dB}\right\\}$. This generated $400$ noisy mixtures for evaluation.

Feature Extraction. All audio signals were sampled at a rate of 16 kHz. We
employed a 512-sample (32 ms) long square-root-Hann window with a hop length
of 256 samples (16 ms), to extract a 257-point single-sided STFT spectral
magnitude as the input to the neural models [31].

Model Configurations. To perform extensive comparison studies across different
model sizes, we use the Transformer and Conformer architectures comprising
$N\\!\in\\!\\{4,\,6\\}$ stacked Transformer and Conformer layers respectively
[31]. For the Transformer speech enhancement backbone, we follow the
configuration in [29, 31]: the layer dimension $d_{\text{model}}\\!=\\!256$,
the number of attention heads $H=8$, and the inner-layer size of the feed-
forward network (FFN) $d_{f\\!f}=1024$. For the Conformer backbone, we adopt
the parameter configurations in [31]: the layer dimension
$d_{\text{model}}\\!=\\!256$, the number of attention heads $H=8$, the kernel
size of convolution $32$, the expansion factor for convolution module $2$, and
the inner-layer size of FFN $d_{f\\!f}=1024$. For the Mamba model, we use the
parameter configurations in [14]: the model dimension $D=256$, the SSM state
dimension $N=16$, the local convolution width $d_{\text{conv}}=4$, and the
expansion factor $E=2$. All the experiments were run on an NVIDIA Tesla
V100-SXM2-32GB GPU.

Implementation Details. All the models were implemented in PyTorch. We used
mean-square error (MSE) on the power-law compressed spectral magnitude as the
objective loss function [56]. The noisy mixtures were dynamically generated at
training time. For each minibatch, we randomly picked 10 clean speech clips
and degrade each clip by a random section of a random noise clip at a random
SNR level sampled from $-10$ to $20$ dB (in $1$ dB steps). The Adam algorithm
[57] was used for gradient optimization, with parameters as in [1], i.e.,
$\beta_{1}=0.9$, $\beta_{2}=0.98$, and $\epsilon=10^{-9}$. We utilized the
gradient clipping technique to cut the gradient values to a range between $-1$
and $1$. All the models were trained for 150 epochs for fair comparison. The
warm-up strategy was adopted to adjust the learning rate:
$lr=d_{model}^{-0.5}\cdot\textrm{min}\left(n\\_step^{-0.5},n\\_step\cdot
w\\_steps^{-1.5}\right)$, where $n\\_step$ and $w\\_steps$ denote the
iteration steps and the warm-up iteration steps, respectively. We followed the
study [31] and set $w\\_steps$ as $40000$.

###  C.2 Experiments on Mamba layer vs. MHSA

Table 12: Evaluation results for replacing the MHSA with Mamba/BiMamba layer in Transformer network, across causal and noncausal configurations. Method | #Params | Causality | Metrics |
---|---|---|---|---
|  |  | NB-PESQ | WB-PESQ | ESTOI | CSIG | CBAK | COVL
Noisy | – | – | 1.88 | 1.24 | 56.12 | 2.26 | 1.80 | 1.67
Transformer-4 | 3.29 M | ✔ | 2.56 | 1.84 | 70.32 | 3.17 | 2.39 | 2.47
$-$ MHSA $+$ Mamba | 3.99 M | ✔ | 2.65 | 1.93 | 72.28 | 3.24 | 2.45 | 2.55
Transformer-4 | 3.29 M | ✗ | 2.74 | 2.01 | 73.44 | 3.31 | 2.50 | 2.63
$-$ MHSA $+$ InnBiMamba | 4.17 M | ✗ | 2.86 | 2.15 | 76.24 | 3.44 | 2.61 | 2.78
$-$ MHSA $+$ ExtBiMamba | 5.74 M | ✗ | 2.88 | 2.18 | 76.75 | 3.54 | 2.63 | 2.84
Transformer-6 | 4.86 M | ✔ | 2.60 | 1.87 | 71.37 | 3.20 | 2.41 | 2.50
$-$ MHSA $+$ Mamba | 5.92 M | ✔ | 2.68 | 1.95 | 73.04 | 3.31 | 2.48 | 2.60
Transformer-6 | 4.86 M | ✗ | 2.78 | 2.05 | 74.56 | 3.38 | 2.52 | 2.69
$-$ MHSA $+$ InnBiMamba | 6.19 M | ✗ | 2.89 | 2.17 | 77.22 | 3.49 | 2.61 | 2.80
$-$ MHSA $+$ ExtBiMamba | 8.54 M | ✗ | 2.91 | 2.21 | 77.60 | 3.60 | 2.65 | 2.88
Table 13: Evaluation results for replacing the MHSA with Mamba/BiMamba layer in Conformer network, across causal and noncausal configurations. Method | #Params | Causality | Metrics |
---|---|---|---|---
|  |  | NB-PESQ | WB-PESQ | ESTOI | CSIG | CBAK | COVL
Noisy | – | – | 1.88 | 1.24 | 56.12 | 2.26 | 1.80 | 1.67
Conformer-4 | 6.22 M | ✔ | 2.67 | 1.94 | 72.78 | 3.29 | 2.45 | 2.58
$-$ MHSA $+$ Mamba | 6.92 M | ✔ | 2.69 | 1.95 | 73.27 | 3.29 | 2.48 | 2.59
Conformer-4 | 6.22 M | ✗ | 2.88 | 2.17 | 76.68 | 3.51 | 2.61 | 2.82
$-$ MHSA $+$ InnBiMamba | 7.10 M | ✗ | 2.89 | 2.17 | 77.40 | 3.51 | 2.62 | 2.81
$-$ MHSA $+$ ExtBiMamba | 8.67 M | ✗ | 2.92 | 2.20 | 77.74 | 3.57 | 2.64 | 2.88
Conformer-6 | 9.26 M | ✔ | 2.68 | 1.94 | 73.41 | 3.30 | 2.46 | 2.59
$-$ MHSA $+$ Mamba | 10.32 M | ✔ | 2.71 | 1.95 | 73.69 | 3.34 | 2.49 | 2.62
Conformer-6 | 9.26 M | ✗ | 2.91 | 2.20 | 77.56 | 3.54 | 2.62 | 2.84
$-$ MHSA $+$ InnBiMamba | 10.59 M | ✗ | 2.92 | 2.19 | 77.85 | 3.52 | 2.64 | 2.83
$-$ MHSA $+$ ExtBiMamba | 12.94 M | ✗ | 2.93 | 2.23 | 78.21 | 3.60 | 2.65 | 2.89
Table 14: Training speed and inference speed of Transformer and TranBiMamba, and Conformer and ConBiMamba. | Transformer-6 | TransInnBiMamba-6 | TransExtBiMamba-6 | Conformer-6 | ConInnBiMamba-6 | ConExtBiMamba-6
---|---|---|---|---|---|---
sec/step | 0.125 | 0.155 | 0.161 | 0.215 | 0.206 | 0.214
RTF | $2.12\times 10^{-4}$ | $1.81\times 10^{-4}$ | $1.95\times 10^{-4}$ | $3.27\times 10^{-4}$ | $2.85\times 10^{-4}$ | $2.99\times 10^{-4}$

##  Appendix D Further Explanation of Mamba used in ASR

###  D.1 Performance Across Different Mamba/BiMamba Sizes

Table 15 reports the performance of Mamba and BiMamba on LibriSpeech100 and
SEAME, across different model sizes. With comparative model sizes,
Mamba/BiMamba still performs poorly than the Conformer.

Table 15: WER/MER results (%) for LibriSpeech100 and SEAME: the left part displays results from LibriSpeech100, while the right part shows results from SEAME. Method | #Params (M) | dev | test | #Params (M) | man | sge
---|---|---|---|---|---|---
Conformer | 34.23 | 6.3 | 6.5 | 47.27 | 16.9 | 23.6
Transformer | 29.38 | 8.0 | 8.4 | 29.38 | 17.7 | 24.5
Mamba-based Models |  |  |
Mamba | 20.41 | 40.8 | 40 | 20.41 | 44.5 | 55.3
Mamba Large | 32.52 | 38.8 | 39.2 | 28.78 | 45.5 | 55
ExtBiMamba | 25.79 | 38.5 | 37.7 | 25.79 | 44.4 | 55.2
ExtBiMamba Large | 33.52 | 37.8 | 37.2 | 33.52 | 43.6 | 54.3

###  D.2 Ablation study of FFN and Residual for ExtBiMamba

Table 16: Ablation Studies on LibriSpeech100 for Transformer and ConExtBiMamba by WER. Method | dev | test
---|---|---
Transformer | 8.0 | 8.4
$-$ Residual | 45.7 | 54.8
$-$ FFN | 23.2 | 25.4
ExtBiMamba | 38.5 | 37.7
$+$ Residual | 42.1 | 41.7
$+$ FFN | 34.9 | 36.1

###  D.3 InnBiMamba vs. ExtBiMamba

In Tables 8 and 17, we present the performance of InnBiMamba and ExtBiMamba in
Transformer and Conformer frameworks. We found that ExtBiMamba consistently
outperforms InnBiMamba across all datasets. Given that InnBiMamba typically
has fewer parameters than ExtBiMamba, we increase the number of layers with
InnBiMamba to match ExtBiMamba for fair comparison. Again, ExtBiMamba performs
better than InnBiMamba.

Table 17: InnBiMamba vs. ExtBiMamba: MER Results (%) on ASRU and SEAME, Left results are from ASRU, Right results are from SEAME Method | Params (M) | dev | test | Params (M) | man | sge
---|---|---|---|---|---|---
Transformer | 30.86 | 13.7 | 13.1 | 29.86 | 17.7 | 24.5
$-$ MHSA $+$ InnBiMamba | 34.80 | 20.2 | 19.5 | 33.81 | 18.4 | 26.0
$-$ MHSA $+$ ExtBiMamba | 44.24 | 18.7 | 18 | 43.25 | 17.2 | 24.3
Conformer | 48.27 | 12.8 | 12.2 | 47.27 | 16.9 | 23.6
$-$ MHSA $+$ InnBiMamba | 50.10 | 12.7 | 12.2 | 49.11 | 17.1 | 23.8
$-$ MHSA $+$ ExtBiMamba | 56.40 | 12.3 | 11.5 | 55.40 | 16.6 | 23.4
Table 18: Training speed (mins/epoch) and Inference speed (RTF) for ASR in LibriSpeech100 on NIVDA Tesla V100.  | Conformer | ConInnBiMamba | ConExtBiMamba
---|---|---|---
mins/epoch | 21.4 | 18.3 | 19.8
RTF | 0.179 | 0.174 | 0.177

Table 18 compares the training and inference speeds of Conformer,
ConInnBiMamba, and ConExtBiMamba. The results show that ConInnBiMamba and
ConExtBiMamba train and infer faster than the Conformer, confirming the
training and inference efficiency of the ConBiMamba architectures. Among them,
ConInnBiMamba is slightly faster than ConExtBiMamba.

##  Appendix E Ablation Study on ConExtBiMamba for ASR Task

###  E.1 Effect of Hyper-Parameter for BiMamba in ConExtBiMamba

The default hyper-parameters for ConExtBiMamba are defined as: the SSM state
dimension $d_{\text{state}}=16$, the state expansion factor $E=2$, and the
local convolution width $d_{\text{conv}}=4$. From Table 19, we find that
increasing the state expansion factor or decreasing the local convolution
width has little impact on the performance of ConExtBiMamba.

We further investigate the impact of the $A$ matrix on performance. The $A$
matrix plays a crucial role in both S4 and Mamba. In the original Mamba paper,
it is initialized using a real-valued diagonal matrix. However, ensuring
randomness is important in deep learning [58]. To explore this, we examine
three types of $A$ matrices: a real-valued diagonal matrix, a completely
randomized matrix, and a real-valued diagonal matrix initialized by
multiplying each element with noise generated from a Gaussian distribution.
Through our experiments, we discovered that the diagonal matrix with Gaussian
noise yielded the best results.

Table 19: Ablation Study on LibriSpeech100 for ConExtBiMamba by employing WER (%) Method | dev (↓) | dev other (↓) | test (↓) | test other (↓)
---|---|---|---|---
ConvExtBiMamba with Gaussian Noise | 5.9 | 17.1 | 6.0 | 17.3
$\rightarrow$ $d_{\text{state}}=32$ | 5.9 | 17.1 | 6.0 | 17.3
$\rightarrow$ $d_{\text{conv}}=2$ | 6.0 | 17.1 | 6.2 | 17.3
$\rightarrow$ $E=4$ | 5.9 | 17.1 | 6.0 | 17.3
$\rightarrow$ Random Noise A matrix | 6.0 | 17.2 | 6.2 | 17.4
$\rightarrow$ default A matrix | 6.1 | 17.2 | 6.2 | 17.5

###  E.2 Effect of Hyper-Parameters for ConExtBiMamba

We conduct ablation experiments on Swish Activation, Macaron-style feed-
forward layers, positional encoding, and dropout. From Table 20, it is
apparent that Swish Activation and Macaron-style feed-forward layers enhance
the model’s performance, while positional encoding and dropout do not have a
significant impact. We suggest that the reason positional encoding, a crucial
component in Transformer-based models, does not significantly affect the
ConExtBiMamba model is twofold. Firstly, Mamba independently models each
input, which suggests that the model may better differentiate between
different positions. Secondly, because Mamba operates similarly to an RNN, the
dependencies between inputs inherently embed some positional information into
the model, thus diminishing the impact of positional encoding.

Table 20: Ablation Studies for ConExtBiMamba on LibriSpeech100. Method | dev (↓) | dev other (↓) | test (↓) | test other (↓)
---|---|---|---|---
ConExtBiMamba | 5.9 | 17.1 | 6.0 | 17.3
$-$ Macaron | 6.1 | 17.7 | 6.3 | 17.9
$-$ Swish | 6.2 | 17.7 | 6.4 | 18.1
$-$ Dropout | 6.0 | 17.1 | 6.0 | 17.4
$+$ Positional Encoding | 6.0 | 17.1 | 6.0 | 17.3

###  E.3 Performance on Extremely Small Datasets

To evaluate model robustness, we conduct tests on the AN4 dataset—a notably
small dataset—using five randomly selected seeds for both the Conformer and
ConExtBiMamba models under identical hyperparameters. Our results show that
ConExtBiMamba consistently outperforms the Conformer across all seeds.
Specifically, the mean WER for the Conformer was 10.02, while it was
significantly lower for ConExtBiMamba at just 4.54. Remarkably, ConExtBiMamba
also demonstrated exceptional robustness, with a WER variance of only 1.11
compared to the Conformer’s substantially higher variance of 77.71. Observing
the loss metrics, we noted that ConBiMamba is less prone to overfitting on
small datasets compared with the Conformer. This suggests that ConBiMamba,
like RNNs, benefits from its ability to process information recursively over
time, updating only a small set of parameters with each iteration, which helps
prevent overfitting in scenarios with limited data. Additionally, this
framework also avoids the gradient vanishing and explosion issues typically
associated with RNNs. Combining these observations with our main results, we
can conclude that ConExtBiMamba effectively merges the strengths of self-
attention-based models and RNN models.

Table 21: Performance on extremely small dataset AN4 by employing WER (%) Method/Seed | 2048 | 233 | 666 | 1024 | 3407 | Average | Variance
---|---|---|---|---|---|---|---
Conformer | 4.0 | 8.2 | 6.1 | 4.4 | 27.4 | 10.02 | 77.71
ConExtBiMamba | 3.8 | 3.8 | 4.8 | 3.8 | 6.5 | 4.54 | 1.11