Feature Attribution Explanations for Spiking Neural Networks

Elisa Nguyen
University of Twente

Enschede, Netherlands

Meike Nauta
University of Twente

Enschede, Netherlands

Gwenn Englebienne
University of Twente

Enschede, Netherlands

Christin Seifert
University of Marburg

Marburg, Germany

Third-generation artificial neural networks, Spiking Neural Networks (SNNs), can be efficiently implemented on hardware. Their implementation on neuromorphic chips opens a broad range of applications, such as machine learning-based autonomous control and intelligent biomedical devices. In critical applications, however, insight into the reasoning of SNNs is important, thus SNNs need to be equipped with the ability to explain how decisions are reached. We present Temporal Spike Attribution (TSA), a local explanation method for SNNs. To compute the explanation, we aggregate all information available in model-internal variables: spike times and model weights. We evaluate TSA on artificial and real-world time series data and measure explanation quality w.r.t. multiple quantitative criteria. We find that TSA correctly identifies a small subset of input features relevant to the decision (i.e., is output-complete and compact) and generates similar explanations for similar inputs (i.e., is continuous). Further, our experiments show that incorporating the notion of absent spikes improves explanation quality. Our work can serve as a starting point for explainable SNNs, with future implementations on hardware yielding not only predictions but also explanations in a broad range of application scenarios. Source code is available at <https://github.com/ElisaNguyen/tsa-explanations>.

Explainability, feature attribution, spiking neural network

§ INTRODUCTION

Spiking neural networks (SNNs), also known as third-generation artificial neural networks [1], consist of spiking neurons.
Spiking neurons emit spikes at certain points in time to transmit information, similar to action potentials in biological neurons and are thus close to biological reality [2]. SNNs are at least as powerful as deep artificial neural networks with continuous activation functions (ANNs) [1]. Their applicability to supervised, unsupervised and reinforcement learning are active research areas [3]. However, the predictive performance of SNNs is not yet on par with ANNs due to the non-differentiability of spikes, making SNN optimization an active research field [4]. Nonetheless, SNNs are interesting as they yield the potential to be implemented in neuromorphic hardware, which is energy- and memory-efficient [5].
Moreover, studies show improved adversarial robustness of SNNs [6]. Their inherent temporal nature also lends itself naturally to processing temporal data making them suitable for critical domains relying on sensor data such as autonomous control [7] and applications using biomedical signals [8].

Critical domains, for example medical applications, pose specific requirements to machine learning models. In addition to high predictive performance, models should make predictions based on the right reasons and be transparent about their decision-making process [9]. Exposing important information of machine learning models is the focus of research on EXplainable Artificial Intelligence (XAI) [10, 11]. Model explanations address the requirement for algorithm transparency and provide methods to inspect model behavior [12].
While various explanation methods exist for second-generation artificial neural networks (ANNs) [11, 10], to the best of our knowledge, the current body of work in explaining SNNs only comprises two major works, namely [13, 14].
XAI for SNNS is thus yet a sparsely studied research area.
If left unaddressed, this research gap could lead to situations where SNNs are methodologically mature for real-world deployment but remain unused because they lack transparency.

We contribute to the field of XAI for SNNs by presenting Temporal Spike Attributions (TSA), an SNN-specific explanation method. The resulting explanations are local, i.e., explain a particular prediction and answer the question: `Why did the model make this decision?' [10].
We build on the explanation method of [14] which uses the model's spike trains. We additionally include the SNN's weights to consider all model-internal variables and regard the absence of spikes to be informative as it also impacts the resulting spike patterns.
TSA results in more complete and correct explanations due to the utilization of comprehensive model-internal information. We demonstrate TSA on time series data. In contrast to anecdotal evidence which is mainly used to evaluate XAI methods [15], we systematically evaluate TSA  quantitatively w.r.t. multiple aspects relevant to explanation quality: The correctness of the explanations (correctness), the explanation's ability to capture the complete model behavior (output-completeness), sensitivity to input changes (continuity), and explanation size (compactness).
In summary, our contributions are as follows:

* We present Temporal Spike Attribution (TSA), a local feature attribution method for SNNs inferred from all model internal variables.

* We apply Kim & Panda's [14] explanation method, which uses only spike train information, on time series data as a baseline to show the impact of incorporating all model internal information in TSA.

* We thoroughly validate TSA's explanation performance using a multi-faceted quantitative evaluation of feature attribution explanations for SNNs, evaluating correctness, completeness, continuity and compactness.

Because SNNs are more popular in neuroscience than machine learning, we briefly introduce SNNs in Section <ref>. Section <ref> reviews related works on SNN explainability. We reflect on the effect of SNN-internal variables on a prediction and present our explanation method, TSA, capturing these effects in Section <ref>. The multi-faceted evaluation in Sections <ref> and <ref> shows the improved explanation performance of TSA. We discuss results in Section <ref> and conclude in Section <ref>.

§ SPIKING NEURAL NETWORKS

This section introduces SNNs and their components.
SNNs are characterized by their computational units, spiking neurons [2]. Analogous to traditional ANNs, SNNs are networks of spiking neurons with weighted connections.
SNNs process information in the form of spike trains, i.e., spikes over time. Neuron models differ in the spike generation mechanisms. Our proposed method is independent of the chosen spike generation mechanism. Without loss of generality, we employ the commonly used leaky integrate-and-fire (LIF) neuron model (cf. Figure <ref>). The membrane potential $u$ of a LIF neuron can be modeled with a linear differential equation:

\begin{equation}
\label{eq:lifintegrate}
\tau_{m}\frac{d u}{d t} = -[u(t)-u_{\textrm{rest}}] + I(t),
\end{equation}

where $I(t)$ describes the amount by which $u$ changes to external input. $\tau_m$ is the time constant of the neuron, which dictates the decay of $u$ in time.
A LIF neuron fires when $u(t)$ crosses a threshold $\theta$ from below. Upon firing a spike, the membrane potentials of the postsynaptic neurons are changed by the weight value, as the spikes are propagated forward in the network. The sign of the weight defines the synapses' nature (i.e., inhibitory or excitatory) and the weight value defines the strength of the postsynaptic potential.
After firing, $u$ is reset to a low reset potential $u_{r}$, and then slowly increases back to its resting value $u_{\textrm{rest}}$.

SNNs internally process spike trains and thus require input data in the form of spikes. The translation of non-spiking to spiking data is called neural coding. Different neural codes exist with temporal and rate coding being the most common.
Temporal coding is biologically more plausible than rate coding because it emphasizes exact spike times as information carriers [2].

Schematic overview of an SNN with LIF neurons. Input spikes $x_i$ are fed to the SNN. Internally, it transmits information as spike patterns $s_i$, which are propagated forward with weights $w_{j,i}^{(l)}$ to determine the postsynaptic potential (PSP). The state of the postsynaptic neuron $u_j$ is changed by the PSP.

§ RELATED WORK

Our work is positioned in the field of EXplainable Artificial Intelligence (XAI) for SNNs.
XAI researches methods to address the black-box nature of machine learning models and explain their reasoning to laypersons and experts  [12].
Machine learning models can either be explained globally by providing an overview of the whole model, or locally by explaining single predictions. Global explanations aim at providing a global understanding of how input relates to an outcome distribution addressing the model explanation and model inspection problem [10, 11]. Examples of global explanation methods are [16] and [17].
The complexity of global explanations increases with the number of input features and model parameters and is therefore a challenging problem in XAI.
Local explanations target individual model predictions and address the outcome explanation problem [10, 11], i.e. they explain the relation between a specific model input and output. Two prominent examples of local explanations are LIME [18] and SHAP [19].
We aim to explain predictions of SNNs and develop a local explanation method.

§.§ Explaining Spiking Neural Networks

Comparison of our method (TSA) to related work FSF [13] and SAM [14] in terms of XAI taxonomy [10] and evaluation [15]. Completeness refers to output-completeness.

FSF [13]     SAM [14]     TSA (Ours)

XAI Method     Post-hoc     Post-hoc     Post-hoc

Scope     Global     Local    Local

Data type     Tabular     Images     Time-series

Model      MC-SEFRON     Convolutional SNNs     Specific to SNNs

Correctness     $\checkmark$     $\times$     $\checkmark$

Completeness     $\times$     $\times$     $\checkmark$

Continuity     $\times$     $\times$     $\checkmark$

Compactness     $\times$     $\times$     $\checkmark$

Coherence     $\times$     $\checkmark$     $\times$

Few works have studied explaining SNNs, which we introduce in the following.
Jeyasothy et al. [13] present feature strength functions (FSFs) to explain a specific SNN model architecture MC-SEFRON with a population encoding layer, no hidden layers, and time-dependent weights. FSFs invert the population coding scheme to link the explanation back to input features and extract interpretable knowledge.
FSFs are functions of the input, i.e. in a human-understandable domain rather than the temporal domain of spike trains. The FSFs are a global and model-specific explanation method, which addresses the model inspection problem [11]. In contrast, we target local explanations to explain model decisions that are applicable to a wider range of SNN models.

Kim & Panda [14] present Spike Activation Map (SAM), a local explanation method for SNNs.
SAM generates visual heatmaps based on a calculation of input feature importance (in the image classification case, these are pixels) and was studied on deep convolutional SNNs with LIF neurons on image data.
SAM is inspired by the biological observation that short inter-spike intervals are deemed to carry information because they likely cause a postsynaptic spike. The authors define contribution scores of single spikes and aggregate them to represent the spike train contribution in the neuronal contribution score (NCS).
The final activation map is computed at time $t$ by a forward pass in the network by multiplying  NCS's at $t$ and summarizing NCS's across the channel axis of convolutional layers.
In contrast to the model-specific FSFs [13], NCSs are model-agnostic because they are solely based on spike information which is part of all SNNs.

Our explanation method TSA is model-agnostic but not model-independent because TSA also takes the model's weights into account, which represent what the SNN has learned.
Furthermore, we aim to cater to the intrinsic temporal design of SNNs and therefore designed TSA for explaining predictions of a time series classification task. We look at time series data as opposed to image data to better fit the temporal nature of SNNs. Thus, we contribute to local explanations for SNNs and compare TSA to SAM.

Table <ref> presents a concise comparison of the related work and our explanation method. We do not compare to model-agnostic methods for ANNs (e.g., LIME [18]) because their application to SNNs is not trivial. Moreover, ANN-based explanations do not rely on SNN model internals and hence might not capture the true model behavior [20]. Our aim is
an SNN-specific explanation method.

§.§ Evaluating SNN Explanations

In contrast to evaluating the predictive performance of models with quasi-standard evaluation metrics (e.g., F-score, AUC), evaluating explanations is an ongoing research topic.
Since the recipients of explanations are humans and are usually context-dependent, there is no standard evaluation protocol [12].
In addition, a good explanation fulfills several different properties, e.g. the correctness or faithfulness in explaining the model behavior and human-comprehensibility among others [15, 21, 22].
Moreover, evaluating explanations is challenging because the ground truth (what the model actually learned) is rarely known. To overcome this issue, one could apply the “Controlled Synthetic Data Check” [15] where a model is applied to (structured) synthetic datasets such that the true data distribution is known, e.g. [23]. We apply this method by constructing a synthetic data set for a classification problem on two input sensors (cf. Section <ref>).

The evaluations of FSFs and SAM were each focused on one aspect of explanation quality: Jeyasothy et al. [13] evaluated “reliability" by using FSFs instead of model weights in the same prediction task, testing how correctly the FSFs capture the global model behavior. Kim & Panda [14] tested the “accuracy" of SAM by comparison with an existing heatmap explanation on ANNs, i.e. how coherent and aligned SAM explanations are to other explanations.
In our work, we perform a multi-faceted evaluation based on the Co-12 framework for evaluating XAI [15] on a synthetic and a real-world data set. More specifically, we evaluate correctness, output-completeness, continuity, and compactness as defined by Nauta et al. [15]. We chose this subset of Co-12 properties as we focus on studying the content of TSA explanations first before considering presentation- and user-related properties.

§ TEMPORAL SPIKE ATTRIBUTION (TSA)

SNNs learn internal weights during training and process data as spike trains (cf. Section <ref>.
Whereas the Spike Activation Map (SAM) [14]
considers only spike trains to generate explanations, TSA captures all information available in the model for a prediction $\hat{y}$ at time $T$ of one $D$-dimensional input $x^{D\times T}$. These information are (i) spike times $S^{(l)}$, (ii) learned weights $W$, and (iii) membrane potentials at the output layer $U^{(L)}$. Each component has an influence on the output $\hat{y}$ and therefore should be included in the explanation.
We explain the single components in Sections <ref> to <ref> and describe their integration to a feature attribution explanation in Section <ref>.

§.§ Influence of Hidden Neuron Spike Times

In temporal coding, the information about the data is assumed to be in the exact spike times of a neuron [2]. The spike times indicate the attribution of neurons $I$ to their downstream neurons $J$ and represent the model's activation in a prediction. Hence, the spike times influence the prediction.
The relationship between the spike times of $I$ and their attribution to $J$ is captured in [14]'s neuronal contribution score (NCS). The NCS is characterized by $\gamma$, which specifies the steepness of the exponential decay over time. We define the decay at the same rate as the decay of the LIF neuron's membrane potential $u$ to reflect the dynamics of the model.

While we build on the NCS, our spike time component $N_{i}(t)$ additionally considers the absence of spikes as information carriers. In a fully connected SNN, each neuron of layer $l$ is connected with each neuron of layer $l+1$. The weighted sum of the neuron's spiking behavior in $l$ determines the amount by which the membrane potentials of neurons in $l+1$ are changed. Absent spikes do not contribute to this sum. Hence, if a neuron $i$ does not spike at time $t$, it does not contribute actively to a change in $J$, allowing a natural decay. Absent spikes can thus be understood in two ways: (i) an absent spike does not affect $u_j$ (cf. Eq. <ref>) or (ii) an absent spike affects $u_j$ by not changing $u_j$ and letting it decay naturally (cf. Eq. <ref>).
In the second case, the attribution of absent spikes to the postsynaptic neuron is negative. However,
absent spike attribution should not weigh as much as spikes because their effect is highly dependent on other incoming synapses. We weigh the contribution of absent spikes by $\frac{1}{B}$ as an approximation of their attribution factor, with $B$ being the size of the preceding layer. This approximation is simple and reflects the relative magnitude of a non-spiking neuron's attribution. Formally, we calculate the spike time component $N_{i}(t)$ as follows:

\begin{align}
\small
\label{eq:ingredient_spikes_s}
N^{S}_{i}(t) &= \sum_{t'=0}^{t}
\begin{cases}
\exp (-\gamma |t - t'|) & \text{if } x_{i, t'}=1\\
0 & \text{Otherwise}
\end{cases}
\end{align}

only considering the presence of spikes, and

\begin{align}
\small
\label{eq:ingredient_spikes_ns}
N^{NS}_{i}(t) &= \sum_{t'=0}^{t}
\begin{cases}
\exp (-\gamma |t - t'|) & \text{if } x_{i, t'}=1\\
-\frac{1}{B}\exp (-\gamma |t - t'|) & \text{Otherwise}
\end{cases}
\end{align}

when including the information about absent spikes.

In our experiments, we compare both
variants as TSAS (spikes only according to Eq. <ref>), and TSANS (non spikes included, according to Eq. <ref>).

§.§ Influence of Model Weights

The weights $W$ represent the strengths of connections of an SNN and have so far not been considered in feature attribution explanations for SNNs. We specifically include weights to capture the contribution of connections to the predictions of an SNN.
Weights determine the impact of spikes to downstream neurons $J$ directly, where the weight value indicates the weight's attribution to a neuron $j \in J$, and the sign specifies whether the synapse is excitatory or inhibitory and leads to an in- or decrease of $u_j$.
Since weights are a property of the model, i.e. independent of the input, the weight contribution obtains its meaning in combination with other components.

§.§ Influence of Output Layer

The output layer is the last computational layer, i.e., the basis for prediction. The output layer consists of spiking neurons, thus the prediction is dictated either by spike patterns $S^{(L)}$ or membrane potentials $U^{(L)}$. In the first case, the influence of the output layer can be captured by the NCS.
The SNN models in our work follow [24]'s architecture and make predictions based on the latter, i.e. the largest $u_i^{(L)}$ determines the predicted class. We capture the effect of the output layer on a model prediction by considering the classification softmax probability $P(t)$ in the computation of an explanation.

\begin{equation}
\Vec{P}(t) = \text{softmax}(U^{(L)})
\end{equation}

§.§ Computation of TSA

TSA combines neuron spike times, model weights, and output layer information in a forward pass into a final score as shown in Algorithm <ref>.
A neuron $i$ generates spike train $s_i$ to the downstream computational layers. The neuron is connected to the next layer $l+1$ via the weight matrix $W^{(l)}$.
Spike time information is captured by $\Vec{N}^{(l)}(t)$, model weights are contained in $W$, and output layer information, i.e. membrane potentials are encoded in $\Vec{P}(t)$.
The spike times and the weights are combined by multiplying the diagonal matrix of $\Vec{N}^{(l)}(t_c)$ with $W^{(l)}$ per layer (line 7 in Algorithm <ref>).
The resulting matrix can be computed for the input layer and all hidden layers. The result is a set of matrices consisting of scores
that represent how the presynaptic neurons contribute to the postsynaptic neurons under direct consideration of the weights.
The values are aggregated in a forward pass through the model. This represents how the input influences the model's neurons and is captured by the input contribution $C_I(t)$ (Line 8).
The final feature attribution $A(x, t) \in \mathbb{R}^{O\times D \times t}$ is computed by multiplying $C_I(t)$ with the softmax probabilities (Line 10). The absence of spikes can be understood as either not affecting (Eq. <ref>) or affecting downstream neurons (Eq. <ref>). We compare both
variants as TSAS (spikes only), and TSANS (non spikes included).
Thus, the computation in line 6 of Algorithm <ref> differs respectively.

Temporal Spike Attribution
Let $x\in\mathbb{R}^{D\times T}$ be an input to SNN $f$ with $L$ layers, $S^{(l)}$ a layer's spike trains, $U^{(L)}$ the output layer's membrane potentials, $W^{(l)}$ the weight matrix connecting layers $l$ and $l+1$, and $t$ the explanation time.
[1]
$S^{(1)}, ..., S^{(L-1)}, U^{(L)}\gets f(x)$
$\Vec{P}(t) \gets \text{softmax}(U^{(L)})$
$t' = 0, 1, 2, ..., t$
Initialize $C_I(t') = I \in \mathbb{R}^{DxD}$
$l = 1, 2, ..., L-1$
Compute $\Vec{N}^{(l)}(t')$
$N_W^{(l)}(t') \gets \text{diag}[\Vec{N}^{(l)}(t')]\cdot W^{(l)}$
$C_I(t') \gets C_I(t') \cdot N_W^{(l)}(t')$
$A(x, t') \gets C_I(t')\cdot \text{diag}(\Vec{P}(t'))$
Concatenate $A(x, t')$ to attribution map $A(x, t)$.

§ EXPERIMENTAL SETUP

We demonstrate TSAS and TSANS on both synthetic and real-world data using fully connected SNNs of different depths. Additionally, we compare the quality of the extracted explanations to SAM [14].

§.§ Data Sets

Synthetic data sets are commonly used in XAI research as the data's true distribution is known [23].
For the synthetic data set, we chose a simple task of classifying two-dimensional, binary time series data of varying length, i.e., $x_{i,t} \in \{0,1\}$, into one of four classes equivalent to a logical OR. An example is shown in Figure <ref>.
The underlying idea of such a simple, synthetic dataset is that the SNN should learn and use the same reasoning as the data generation process, which is known a priori. We can then evaluate whether the explanation shows the expected reasoning.
We generate the data by sampling both the duration and activation of $x_i$ at random. The maximum duration for an activity is set at 600 time steps, and we generate 900,000 time steps sequentially as the entire data set. Once the data is generated, we add labels per time step according to the data.
We perform a 70%-30% sequential train-test split, i.e. the first 70% of the data constitute the training and the remaining 30% the test set.

Label      $x_1$     $x_2$

A     0     0

B     1     1

C     0     1

D     1     0

Synthetic data set. Class label assignment (top) and example time series (bottom).

For a real-world scenario, we use the “Activities of Daily Living Recognition using Binary Sensors” data set (ADL) [25]. ADL is an imbalanced multivariate time series data set that can be used for multi-class classification. The data was collected from a wireless binary sensor network installed in the homes of two subjects over 35 days at a time granularity of one second. The data set includes 10 classes specifying different activities of the subjects, e.g. Sleeping or Toileting. ADL is openly available in the UCI machine learning repository[1]. The sensors are human-interpretable, e.g., activation of the Bed sensor means that the subject is lying in their bed (cf. Figure <ref>). Since the data is human-understandable, feature attributions in this input space are easily understandable (e.g. “The most attributing feature is the bed sensor activation at time $t$. Hence, it is important for the model that the subject lies in bed at $t$ for predicting the activity “Sleeping” at $t$.").
The SNN models are trained to predict the activity at each time step. The input data is converted to spikes, and we apply no other data transformations. Thus, the neural coding is a direct mapping of spike times. As a preprocessing step, we add a constantly spiking sensor as a bias input to the data.
Gaps between activities were filled with inactivity in all sensors. We introduce the Other class for these time gaps.[2]
We treat the data as one long time series per subject and split the data set sequentially into a training (60%), validation (20%), and test (20%) set to preserve temporal dependencies.

[2]We found two cases where the activity end precedes the start (Index 78, 80 of subject A). We excluded these activities from the data set as the data collection or labeling was faulty and filled their time with inactivity.

Data sample of subject A from the ADL dataset with either the Basin or the Seat sensor active.

§.§ SNN Models

We train three SNNs with 1, 2 and 3 hidden layers denoted SNN-1L, SNN-2L and SNN-3L to evaluate the effect of network depth on explanation quality, where the respective size of the hidden layers is determined by hyperparameter tuning.

The models are implemented as recurrent networks with binary activations with discretized formulas of the model dynamics as in [24]. We train the models in a similar fashion, i.e. using backpropagation with surrogate gradients.
We adapt the training procedure to our data set that exhibits a temporal order (i.e., activities follow one after another in time). The membrane potential state $u(t)$ is retained between data samples to reflect the temporal dependencies. So, the model state is initialized with the last state of the last simulation.

While the main focus of our work is explaining SNNs and not their optimization, the models should perform reasonably well, so that the models have learned something worth explaining. Therefore, we do a hyperparameter search in a greedy optimization process for 20 epochs under the assumption of independence of hyperparameters for the ADL task(Table <ref>). Due to the simplicity of the synthetic task, we omit hyperparameter tuning.

Results of greedy hyperparameter search. Tested ranges are: {0.01, 0.001, 0.0001} for $\Delta t$ and learning rate, {0.1, 0.01, 0.001} for $\tau_{syn}$ and $\tau_{mem}$, {128, 256, 512} for batch size, {25, 50, 100, 200} for hidden layer sizes.

3cADL     Synthetic

Hyperparameter     SNN-1L     SNN-2L     SNN-3L     All SNNs
$\Delta t$     0.001     0.001     0.001     0.001

$\tau_{syn}$                    0.01               0.01             0.01        0.01

$\tau_{mem}$                    0.01              0.001              0.01            0.001

Learning rate              0.01               0.001              0.001             0.001

Batch size                 128            256             512       128

Hidden size 1     -                 100           50          10

Hidden size 2     -               -               25           10

The final models were fully retrained on the training set with early stopping with a patience of 10 epochs, monitoring the validation loss.
We compare the SNN models against the majority baseline, i.e., a classifier that always predicts the class most represented in the training set in terms of balanced accuracy [26] at the 95% confidence interval (CI):

\begin{equation}
\small
\textrm{CI} = 1.96\sqrt{\frac{\text{Balanced Accuracy}*(1-\text{Balanced Accuracy})}{N}}
\end{equation}

where  $N$ is the number of samples. The synthetic data set is balanced with four classes, thus the majority baseline has a balanced accuracy of .25, while in ADL the majority class (Sleeping) leads to a balanced accuracy of .09.

SNN model performance for the synthetic and ADL task in balanced accuracy ()% at 95% CI. Results are based on one training run. The synthetic data set contains no validation split (n.a.). The baseline is a majority vote.

Data     Model     Test     Train     Validation

SNN-1L     98.3 $\pm$ 0.0     98.3 $\pm$ 0.0     n.a.

SNN-2L     95.6 $\pm$ 0.1     95.5 $\pm$ 0.1     n.a.

SNN-3L    93.3 $\pm$ 0.1     93.2 $\pm$ 0.1    n.a.

Baseline     25.0     25.0     n.a.
SNN-1L     51.6 $\pm$ 0.1     50.6 $\pm$ 0.1     53.6 $\pm$ 0.1

SNN-2L     51.7 $\pm$ 0.1     51.5 $\pm$ 0.1     54.9 $\pm$ 0.1

SNN-3L     50.0 $\pm$ 0.1    49.0 $\pm$ 0.1    52.0 $\pm$ 0.1

Baseline     9.1    9.1    9.1

Table <ref> reports model performance for both data sets.
In the synthetic task, all models achieve high accuracy ($>$ 0.9), where model depth is correlated with lower performance. Since learning in SNNs is an active research area itself [4], the reason for this phenomenon is unclear. Still, all models are able to solve the synthetic task well and are therefore useful for our analysis of well-performing SNNs.
All models learn the ADL task similarly well, significantly outperforming the baseline (selecting the majority class).
while not overfitting (cf. Table <ref>). While other work reports higher performance on ADL with complex ANNs [27], our SNN models are sufficiently accurate for evaluating TSA in a real-world setting.

§.§ Evaluation

Because explanation quality is a multidimensional property (e.g., [15]), we quantify the explanation quality of TSAS and TSANS objectively using mainly content-related properties of the Co-12 framework [15].
We use SAM as a baseline to investigate whether the incorporation of model weights improves explanations[3].
To the best of our knowledge, we are the first to apply such quantitative evaluation measures to explanations for SNNs.

[3]We note that [14] developed SAM to explain image data and did not claim that these maps are applicable to other data types. Still, we believe it is applicable as it uses only spike times, and applied it to our time series data sets.

§.§.§ Evaluation Setup

Because we simulate our SNN models in our experiments as recurrent models with binary activations [24], i.e. in a non-neuromorphic environment, calculating TSA on all test data is computationally not tractable.
For an accurate and balanced evaluation of our explanations, we sub-sample the test data, choosing the same number of samples per class, to identify an explanation evaluation set. We assume that relevant features for the prediction are contained within a fixed time window of ${[t-1000, t]}$ where $t$ is the prediction time.
For the synthetic data, we randomly select 25 samples per class (i.e., 100 overall).
For each ADL subject, nine samples across the start, middle, and end of an activity are chosen per class in the test set.
The start and end of the activity are defined as the first and last minute of this activity, respectively. Given these constraints, we sample $t$ at random, resulting in 180 samples (81 and 99 for subjects A and B).

§.§.§ Evaluation Measures

Correctness refers to whether the explanation reflects the true behavior of the model, which is universally desirable.
To measure correctness, we incrementally delete ranked feature segments (i.e., most attributing first) and record model prediction performance [15]. The area under the curve of the resulting graph represents explanation correctness as explanation selectivity [28]. A low score is desirable, as the model performance is expected to drop significantly when highly attributing feature segments are deleted.
We define feature segments as a number of continuous, strictly positively or negatively attributing time steps within one input dimension $d$ of $x$ that is at most 10 seconds long. This duration is assumed to capture the temporal dependencies at an appropriate information coarseness for both classification tasks posed by the synthetic and ADL data sets.
Moreover, the attribution values are not expected to vary strongly within 10 seconds if they are either positive or negative. We implement this incremental feature deletion in temporal data as an inversion of the input spike train, similar to the perturbations proposed by [29] for the evaluation of XAI methods on time series data.

Output-completeness assesses to which extent the set of identified important features $F$ is sufficient for prediction $\hat{y}$, i.e., $F\rightarrow\hat{y}$.
A perfect output-complete explanation covers all important features relevant to the prediction but might include more features than necessary (cf. criterion compactness).
We evaluate output-completeness by deleting unimportant feature segments and reporting the model performance upon deletion [15]. We define unimportant feature segments as having zero attribution because TSA produces unscaled feature attribution explanations for which an importance threshold is difficult to define.
Contrary to the evaluation of correctness, we do not invert the unimportant feature segments to delete them. For correctness, we incrementally change small parts of the data so that the perturbation is not large in the beginning but accumulates. For evaluating output-completeness, however, we delete many features at once.
If we inverted all the unimportant features, the data would become unrealistic and not reflect the original data distribution (e.g., multiple sensors active around the house while the subject cannot be in multiple locations at once) since we delete all unimportant features at once.
Instead, we shuffle the unimportant features randomly in the time domain to delete any effect the unimportant features have at their original time. This resembles the permutation importance method for structured data [30] where the importance of features is measured by how much a score, in our case the model performance, changes upon random permutation of the features. If feature segments are truly unimportant, the model predictions should not change.

Continuity refers to an explanation method's capability to generalize. An explanation method is continuous if it generates similar explanations for similar input. As non-continuous behavior is difficult for a user to understand, continuity is desirable.
We measure continuity by inspecting the stability for slight variations in the input data. We use max-sensitivity defined as the maximum Frobenius norm of the difference between the explanations on original and slightly varied data [31]:

\begin{equation}
\label{eq: max-sensitivity}
\textrm{Sens}_{\textrm{max}}(e, f, x, t, r) = \max_{||x'-x|| \leq r}||e(f,x', t)-e(f,x, t)||
\end{equation}

where $e$ refers to the explanation, $f$ to the model, $x$ the input data, $t$ the current timestep and $r$ defines the neighborhood region in which perturbed data $x'$ is still considered as similar to $x$.
We vary the data randomly by perturbing the duration of active sensors in the range of 10% of the original duration.
Such perturbations are realistic given the task of activity prediction where the task duration is not rigid and different instances of the same activity can have a different pace, e.g. taking a bit more time for the activity “shower".

Compactness refers to the size of an explanation, where a compact, in our case, sparse, attribution map is desirable. We measure the mean size of all extracted explanations as the sum of absolute attribution values for compactness:

\begin{equation}
\textrm{Compactness}(A) = \frac{1}{N}\sum_i^D\sum_j^t |a_{i,j}|
\end{equation}

where $A$ is the matrix of dimension $D\times t$, i.e. input dimensionality $D \times$ the timestep to be explained. $N$ denotes the total number of explanations extracted for the experiment.

§ RESULTS

In addition to the quantitative analysis in Section <ref>, we present visual examples of explanations in Section <ref> to give an impression of TSA and analyze its coherence [15].

§.§ Quantitative Results

Quantitative evaluation results at 95% CI of our TSAS and TSANS explanations compared to SAM [14] on Synthetic and ADL. Arrows indicate the direction of better performance. Continuity measured as max-sensitivity (no CI).

Measure     TSAS    TSANS     Baseline (SAM)             TSAS    TSANS     Baseline (SAM)

(r)1-5 (r)6-9
4@lCorrectness $\downarrow$     16*[origin=c]90ADL

SNN-1L        0.404 $\pm$ 0.096     0.115 $\pm$ 0.063     0.628 $\pm$ 0.095         0.086 $\pm$ 0.041        0.006 $\pm$ 0.011           0.329 $\pm$ 0.069

SNN-2L        0.809 $\pm$ 0.077     0.426 $\pm$ 0.097     0.762 $\pm$ 0.083         0.633 $\pm$  0.070        0.170 $\pm$ 0.055           0.655  $\pm$ 0.069

SNN-3L        0.813 $\pm$ 0.076      0.520 $\pm$ 0.098     0.822 $\pm$ 0.075          0.505 $\pm$ 0.073        0.154 $\pm$ 0.053           0.377 $\pm$ 0.071
(r)2-2 (r)3-5 (r)7-9
4@lOutput-completeness $\uparrow$

SNN-1L        0.726 $\pm$ 0.087     1.000 $\pm$ 0.000     0.737 $\pm$ 0.086           0.996  $\pm$  0.009        1.000 $\pm$ 0.000            0.624 $\pm$ 0.071

SNN-2L        0.692 $\pm$ 0.090      1.000 $\pm$ 0.000     0.724 $\pm$ 0.088         0.658 $\pm$ 0.069        0.957 $\pm$ 0.030          0.649 $\pm$ 0.070

SNN-3L        0.600 $\pm$ 0.096     0.989 $\pm$ 0.020     0.680 $\pm$ 0.091         0.631 $\pm$ 0.070         0.991 $\pm$ 0.014                 0.553 $\pm$ 0.073
(r)2-2 (r)3-5 (r)7-9
4@lContinuity $\downarrow$

SNN-1L        0.282     0.334     0.679          0.652          0.474          2.715

SNN-2L        0.021     0.023     0.863          0.011          0.009          4.338

SNN-3L        0.001     0.002     0.676           0.002        0.002          53.287
(r)2-2 (r)3-5 (r)7-9
4@lCompactness $\downarrow$

SNN-1L        0.344 $\pm$ 0.009     0.411 $\pm$ 0.005     1.143 $\pm$ 0.029            0.730 $\pm$ 0.109         1.651 $\pm$   0.138          9.634 $\pm$ 5.337

SNN-2L        0.007 $\pm$ 0.000     0.011 $\pm$ 0.000     0.760 $\pm$ 0.031            0.002 $\pm$ 0.000        0.003 $\pm$   0.000        2.183 $\pm$ 0.633

SNN-3L        0.000 $\pm$ 0.000[3]     0.001 $\pm$ 0.000     0.321 $\pm$ 0.011          0.001 $\pm$ 0.000        0.001 $\pm$ 0.000        49.197 $\pm$ 293.630
9@l$^\mathrm{3}$While very small, the explanation size is $>0$. This value is rounded.

The results of the quantitative analysis are presented in Table <ref>. Similar explanation performance trends can be observed from the synthetic and ADL experiments:
In terms of correctness and output-completeness, TSANS is clearly superior to TSAS and SAM. This shows that the SNN's output is also based on the absence of spikes and that considering the absence of spikes is relevant for the explanation (cf. detailed discussion in Section <ref>). TSAS, however, does not show a clear improvement to SAM in correctness, implying that the additional consideration of weights to spike times is not significant for capturing the model's true behavior.

SAM is slightly better than TSAS in terms of output-completeness in the synthetic task, while TSAS is better in the ADL experiments. Investigating the extent to which the inclusion of model weights in the computation of a feature attribution explanation for SNNs improves the explanation's completeness in finding all relevant parts of the model input is left for future work.
The results in the evaluation of continuity show that TSAS and TSANS both are more stable than SAM in producing similar explanations for similar, but slightly different input data. Also in terms of explanation size, TSAS and TSANS generate smaller explanations than SAM where TSAS produces the most compact explanations.

§.§ Visual Inspection

TSA generates feature attribution explanations, which can be visualized and overlaid with the input data. We visually inspect TSA explanations in the case of correct prediction, misclassification, and deep SNN models. In all examples, we display only the last 7 (synthetic) and 40 seconds (ADL) of the sample due to space constraints. The visualizations show positive and negative class attributions in red and blue, respectively. White corresponds to an attribution value of zero. The color scale is explanation-specific and dictated by the largest absolute attribution value. The $y$-axis shows the input dimensions, i.e. the sensors of the data set. Sensor activation is visualized by spikes (vertical lines) at a time step.

§.§.§ Correct Predictions

Example visualizations for correct predictions of SNN-1L in the synthetic task are shown in Figure <ref>. Examples of the ADL task are shown in Figure <ref>.

TSANS (left), TSAS (center), and SAM (right) activation of class C for SNN-1L's correct prediction of $y=$C for the synthetic task. Best viewed in color.

TSAS breakfast class activation.

TSANS breakfast class activation.

SAM breakfast class activation.

Attribution maps of our method compared to SAM [14] for the same example of a correct prediction of SNN-1L ($y=$Breakfast). Best viewed in color.

Overall, the explanations generated by TSANS, TSAS, and SAM for SNN-1L seem quite similar: Recent time steps attribute stronger than time steps further in the past. All explanations also recognize the spiking input to be the important part, while TSANS also assigns attribution to non-spiking parts of the data. Both TSAS and TSANS show different attribution strengths between different input dimensions for the same time step, while SAM seems to assign similar non-zero values to the same time steps regardless of the input dimension. Explanations generated by SAM do not differentiate between positive and negative attributions.

§.§.§ Misclassifications

In Figure <ref>, explanations for an incorrect prediction of the model are displayed for the true class (Breakfast) as well as the predicted class (Lunch). While both maps look very similar, there are subtle differences. In both cases, the Cupboard sensor activation attributes positively to the classes. This makes sense because a kitchen sensor is likely to be connected to meal-related classes. Still, the map shows that the model connects this sensor's activation with the Lunch class rather than the Breakfast class at this point in the data. It is also noticeable that the model bias is negatively attributing to the prediction in both cases, with the negative attribution in the last time step being larger for Lunch. However, in the Breakfast class case, the (negative) bias is slightly stronger across time, suggesting that the model is biased against Breakfast. Given the stronger positive attribution of the Cupboard sensor activation and the slight bias against Breakfast, Lunch, a likely similar-looking class, is predicted. This example highlights the informativeness of negative and positive class attribution, as SAM would be unable to distinguish this.

TSAS lunch class activation.

TSAS breakfast class activation.

Visualizations of TSAS for an incorrect prediction. The numerical values at time steps 3577 and 3600 were added manually.´The attribution of the cupboard is stronger for the lunch class than the breakfast class. Best viewed in color.

§.§.§ Deep SNNs

Figure <ref> shows examples from TSANS explanations extracted from SNN-2L and SNN-3L on the same sample as Figure <ref>.
The examples show that the decay rate $\gamma$ is important for SNN models, as it dictates how far into the past spikes can influence the model prediction at time $t$. SNN-2L and SNN-3L have different $\gamma$. SNN-2L, with a steeper decay, can only consider the last two to three time steps while SNN-3L can look further into the past.

Furthermore, the examples also show a limitation of TSA for deep models. Attribution values tend to alternate in an input dimension between positive and negative values, which could be a result of the repeated multiplication of values in $[-1, 1]$ during the aggregation of attribution scores across layers. This indicates a need to further study TSA on deep SNNs.

TSANS breakfast class activation for SNN-2L.

TSANS breakfast class activation for SNN-3L.

TSANS explanations from SNN-2L and SNN-3L on the same example of a correct prediction. Best viewed in color.

§ DISCUSSION

The quantitative evaluation shows that TSA achieves a significant improvement over SAM in both the synthetic and ADL data sets. More specifically, TSANS outperforms SAM in all tested properties while TSAS is better at explaining SNN-1L and roughly on par with SAM for explaining SNN-2L and SNN-3L. Overall,  no local explanation method outperforms all others in all tested properties, which demonstrates the multi-dimensionality of explanation quality.

In comparison to SAM, TSAS and TSANS both consider the model weights in the computation of attribution directly. Since the weights are an essential part of a neural network, we hypothesized that their direct consideration would improve explanation quality. Our results show that this is largely the case, especially for explanations of SNN-1L. For this model, both TSAS and TSANS showed significant improvements to SAM in all tested properties except for output-completeness in the synthetic task.
The continuity of TSA could also be positively impacted by considering weights $W$, as $W$ is a constant factor across time, which scales the NCS.
Our SNN models exhibit weights $W$ with weight values $|<1|$, hence the attribution values are kept small, leading to more compact explanations. In cases where weight values are large, weights could be normalized for explanation compactness.
Additionally, TSA distinguishes between positive and negative attribution because we consider the excitatory and inhibitory nature of $W$, whereas SAM is unable to make this distinction. This can also be observed in the examples shown (Figure <ref>), where TSAS and TSANS both assign a negative bias attribution while SAM marks the bias as positively attributing.
Nevertheless, $W$ could potentially cause the decreased explanation performance for TSA explanations of deeper SNNs. Signs may cancel each other out as weighted NCS are multiplied across layers.
Furthermore, $W$ could lead to vanishing attribution for deep models in our case (e.g., indicated by the compactness of deep SNNs), due to repeated multiplication of values $|<1|$. This can be investigated in future work.

The results show that absent spikes are relevant. We extended the definition of the NCS by the contribution score for absent spikes in TSANS.
The quantitative results clearly show that inactive input dimensions are relevant to the model prediction. While the models mainly use spikes for their prediction, the inactivity of other neurons is also relevant. Our synthetic data set with logical OR is designed such that the absence of spikes is decisive for the class label, and such reasoning should be correctly reflected in the explanation. In the real-world ADL data set, the active Toaster sensor is for example important to the model at the same time as the bathroom sensors' inactivity to classify Breakfast. As both correctness and output-completeness of TSANS present significant improvements to SAM and TSAS, it is likely that the model has learned the connection between non-spikes and classes. Particularly the experiments with synthetic data verify that this observation is not specific to the models trained on the ADL task, but also holds for accurate temporally coded SNN models.
Since SAM and TSAS do not consider absent spikes, they generally perform worse in the evaluation for content-related measures.
The evaluation of continuity and compactness uses absolute values to determine the explanation's performance. As SAM and TSAS are restricted to defining attribution only for spikes, the amount of change in the explanation is also limited. Therefore, TSAS explanations are more compact than TSANS. However, both do not exhibit a large difference in continuity, which indicates that TSA itself is continuous.

§.§ Limitations

While the evaluation results indicate that TSA improves upon SAM in terms of explanation quality, there are limitations to consider.
First, TSA, like SAM, is a post-hoc explanation method. This means that it is applied to trained SNN models. Any unreasonable-looking or unexpected explanation could therefore either be rooted in erroneous model behavior or in errors of the explanation method [32].
To ensure that the latter is not the case, we systematically and quantitatively evaluated TSA in our work, where we found that the applicability of TSA to deep SNNs likely requires further research.
Second, we tested only SNNs based on LIF. While TSA only relies on spike times, membrane potentials and weights and is therefore applicable to any spike generation mechanism, the computation of the NCS requires a specified decay parameter $\gamma$. With LIF and other integrate-and-fire models, the choice of $\gamma$ is straightforward. For more complex models that do not specify $\gamma$ directly (e.g. Hodgkin-Huxley neurons [33]), it must be defined first before using TSA to explain.
Third, the evaluation does not consider user-related tasks since the technical feasibility of the method was in focus first. We did not explicitly test for comprehensibility, but it is an important property of explanations [11]. Hence, TSA as an explanation method for SNNs does not have sufficient maturity for usage yet. Instead, it offers an effective starting point for further research on explanations for SNNs.

§ CONCLUSION

We present a local explanation method to address the outcome explanation problem for SNNs. We define Temporal Spike Attribution (TSA), a feature attribution explanation that uses model-internal variables to explain the prediction of time series classification.
The two variants of TSA differ in the consideration of absent spike contribution, namely TSAS and TSANS. We demonstrate TSA with three SNNs of different depths with temporal data and evaluated it in a multi-faceted quantitative analysis.
We found that TSANS is superior in correctness and output-completeness, which shows the importance of considering absent spikes. There is no substantial difference between the TSA variants in terms of continuity and compactness. Besides, a decrease in the quality of TSA is observed for deep SNNs, which we attribute to the aggregation across layers.
We find that it is advantageous to consider all available information for explaining SNN predictions. Future studies could focus on the explanation of deep SNNs and human-comprehensibility based on TSA.

[1]
W. Maass, “Networks of spiking neurons: The third generation of neural network models,” Neural Networks, vol. 10, no. 9, pp. 1659–1671, 1997.

[2]
W. Gerstner, W. M. Kistler, R. Naud, and L. Paninski, Neuronal dynamics: From single neurons to networks and models of cognition / Wulfram Gerstner, Werner M. Kistler, Richard Naud, Liam Paninski.
Cambridge: Cambridge University Press, 2014.

[3]
F. Ponulak and A. Kasinski, “Introduction to spiking neural networks: Information processing, learning and applications,” Acta Neurobiol. Exp. (Wars.), vol. 71, no. 4, pp. 409–433, 2011.

[4]
X. Wang, X. Lin, and X. Dang, “Supervised learning in spiking neural networks: A review of algorithms and evaluations,” Neural networks : the official journal of the International Neural Network Society, vol. 125, pp. 258–280, 2020.

[5]
A. F. Murray, Pulse-Based Computation in VLSI Neural Networks, pp. 87––109.
Cambridge, MA, USA: MIT Press, 1999.

[6]
S. Sharmin, P. Panda, S. S. Sarwar, C. Lee, W. Ponghiran, and K. Roy, “A comprehensive analysis on adversarial robustness of spiking neural networks,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2019.

[7]
Z. Bing, C. Meschede, F. Röhrbein, K. Huang, and A. C. Knoll, “A survey of robotics control based on learning-inspired spiking neural networks,” Frontiers in Neurorobotics, vol. 12, 2018.

[8]
M. R. Azghadi, C. Lammie, J. K. Eshraghian, M. Payvand, E. Donati, B. Linares-Barranco, and G. Indiveri, “Hardware implementation of deep network accelerators towards healthcare and biomedical applications,” IEEE Transactions on Biomedical Circuits and Systems, vol. 14, no. 6, pp. 1138–1159, 2020.

[9]
J. He, S. L. Baxter, J. Xu, J. Xu, X. Zhou, and K. Zhang, “The practical implementation of artificial intelligence technologies in medicine,” Nature medicine, vol. 25, no. 1, pp. 30–36, 2019.

[10]
A. Adadi and M. Berrada, “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 6, pp. 52138–52160, 2018.

[11]
R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, “A survey of methods for explaining black box models,” ACM Computing Surveys, vol. 51, no. 5, pp. 1–42, 2019.

[12]
C. Molnar, G. Casalicchio, and B. Bischl, “Interpretable machine learning – a brief history, state-of-the-art and challenges,” in ECML PKDD 2020 Workshops, (Cham), pp. 417–431, Springer International Publishing, 2020.

[13]
A. Jeyasothy, S. Sundaram, S. Ramasamy, and N. Sundararajan, “A novel method for extracting interpretable knowledge from a spiking neural classifier with time-varying synaptic weights,” CoRR, vol. abs/1904.11367, 2019.

[14]
Y. Kim and P. Panda, “Visual explanations from spiking neural networks using inter-spike intervals,” Scientific Reports, vol. 11, no. 1, 2021.

[15]
M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, J. Schlötterer, M. van Keulen, and C. Seifert, “From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai,” ACM Comput. Surv., feb 2023.
Just Accepted.

[16]
B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. sayres, “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV),” in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, pp. 2668–2677, PMLR, 10–15 Jul 2018.

[17]
M. Nauta, R. van Bree, and C. Seifert, “Neural prototype trees for interpretable fine-grained image recognition,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14928–14938, 2021.

[18]
M. T. Ribeiro, S. Singh, and C. Guestrin, “"why should i trust you?": Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, (New York, NY, USA), p. 1135–1144, Association for Computing Machinery, 2016.

[19]
S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, (Red Hook, NY, USA), p. 4768–4777, Curran Associates Inc., 2017.

[20]
R. Poyiadzi, X. Renard, T. Laugel, R. Santos-Rodríguez, and M. Detyniecki, “On the overlooked issue of defining explanation objectives for local-surrogate explainers,” ArXiv, vol. abs/2106.05810, 2021.

[21]
U. Bhatt, A. Weller, and J. M. F. Moura, “Evaluating and aggregating feature-based model explanations,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 (C. Bessiere, ed.), pp. 3016–3022, ijcai.org, 2020.

[22]
R. Srinivasan and A. Chander, “Explanation perspectives from the cognitive sciences - A survey,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 (C. Bessiere, ed.), pp. 4812–4818, ijcai.org, 2020.

[23]
Y. Liu, S. Khandagale, S. Khandagale, C. White, and W. Neiswanger, “Synthetic Benchmarks for Scientific Research in Explainable Machine Learning,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (J. Vanschoren and S. Yeung, eds.), vol. 1, 2021.

[24]
E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019.

[25]
F. J. Ordóñez, P. de Toledo, and A. Sanchis, “Activity recognition using hybrid generative/discriminative models on home environments using binary sensors,” Sensors (Basel, Switzerland), vol. 13, no. 5, pp. 5460–5477, 2013.

[26]
L. Mosley, A balanced approach to the multi-class imbalance problem.

[27]
R. A. Hamad, M. Kimura, L. Yang, W. L. Woo, and B. Wei, “Dilated causal convolution with multi-head self attention for sensor human activity recognition,” Neural Computing and Applications, 2021.

[28]
G. Montavon, W. Samek, and K. Müller, “Methods for interpreting and understanding deep neural networks,” Digit. Signal Process., vol. 73, pp. 1–15, 2018.

[29]
U. Schlegel, H. Arnout, M. El-Assady, D. Oelke, and D. A. Keim, “Towards a rigorous evaluation of xai methods on time series,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 4197–4201, 2019.

[30]
L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp. 5–32, 2001.

[31]
C.-K. Yeh, C.-Y. Hsieh, A. Suggala, D. I. Inouye, and P. K. Ravikumar, “On the (in)fidelity and sensitivity of explanations,” in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), vol. 32, Curran Associates, Inc., 2019.

[32]
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, “Explaining explanations: An overview of interpretability of machine learning,” in 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80–89, 2018.

[33]
A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” The Journal of physiology, vol. 117, no. 4, pp. 500–544, 1952.

[34]
F. Zenke and S. Ganguli, “Superspike: Supervised learning in multilayer spiking neural networks,” Neural computation, vol. 30, no. 6, pp. 1514–1541, 2018.

§ DETAILS ON DATA SETS

All code for synthetic data set generation as well as preprocessing of real-world data is available in our Github repository <https://github.com/ElisaNguyen/tsa-explanations>.

§.§ Synthetic Data

The synthetic data set is generated as a smaller and simpler version of the real-world data, which shall be easy to learn. It consists of two-dimensional time series data with four classes. The data is binary (i.e., $x_{i,t} \in \{0,1\}$). We generate the data by sampling both the duration and activation of $x_i$ at random. The maximum duration for an activity is set at 600 time steps, and we generate 900 000 time steps sequentially as the entire data set. Once the data is generated, we add labels per time step according to the data, as specified in Figure <ref> of the main paper. There are no misclassified time steps.

§.§ Real-world Data

The real-world data set we use is the Binary ADL data set [25], which is openly available in the UCI machine learning repository. The data set consists of the start and end times of activities and their corresponding labels across two subjects, A and B. Given this information, we generate continuous time series across the complete duration of the data set. Gaps between activities were filled with inactivity in all sensors. We introduce the Other class for these time gaps.
We found two cases where the activity end precedes the start (Index 78, 80 of subject A). We excluded these activities from the data set as either the data collection or labeling is faulty and filled their time with inactivity.

§ SNN MODEL DEFINITION AND TRAINING

The SNN models (SNN-1L, SNN-2L and SNN-3L) were built as fully connected recurrent networks with binary activations, using discretized formulas of the network dynamics, in accordance with [24]. Likewise, the models are also trained with surrogate gradient descent using a fast sigmoid surrogate. Moreover, we adapt the training procedure to our data set that exhibits a temporal order (i.e., activities follow one after another in time). The membrane potential state $u(t)$ is retained between data samples to reflect the temporal dependencies. So, state variables of the model are initialized with the last states of the last simulation.
The maximum membrane potential at the output layer at each time step $\Delta t$ determines the prediction. This allows the use of regular loss functions for optimization. Similar to [34], the models are optimized with negative log-likelihood.
While the main focus of our work is explaining SNNs and not their optimization, the models should demonstrate a clear improvement in predictive performance to pure chance and perform reasonably well, so that the models have learned something worth explaining.

Due to the simplicity of the synthetic classification task, we omit hyperparameter tuning. The model hyperparameters were determined across all models beforehand (Table <ref>)

Hyperparameter     Choice
$\Delta t$     0.001

$\tau_{syn}$                    0.01

$\tau_{mem}$                    0.001

Learning rate              0.001

Batch size                 128

Size of hidden layer       10

Hyperparameters used for model building with synthetic data set.

Standard hyperparameters are not sufficient in the ADL task. Therefore, the hyperparameters of the networks are tuned in a greedy optimization process for 20 epochs under the assumption of independence for this task.
With the tuned hyperparameters, the final models were fully retrained on the training set. As regularization, early stopping with a patience of 10 epochs was used, monitoring the validation loss. The final hyperparameters are shown in Table <ref>.

Hyperparameter     SNN-1L     SNN-2L     SNN-3L
$\Delta t$     0.001     0.001     0.001

$\tau_{syn}$                    0.01               0.01             0.01

$\tau_{mem}$                    0.01              0.001              0.01

Learning rate              0.01               0.001              0.001

Batch size                 128            256             512

Size of hidden layer 1     -                 100           50

Size of hidden layer 2     -               -               25

Results of greedy hyperparameter optimization for all models on the binary ADL task. Tested ranges are: {0.01, 0.001, 0.0001} for $\Delta t$ and learning rate, {0.1, 0.01, 0.001} for $\tau_{syn}$ and $\tau_{mem}$, {128, 256, 512} for batch size, {25, 50, 100, 200} for hidden layer sizes.

The scripts used for hyperparameter optimization and model training are provided in our GitHub repository <https://github.com/ElisaNguyen/tsa-explanations>.

§ EVALUATION MEASURE COMPUTATION

In this section, pseudocode and formulas are given for this work's quantitative evaluation of explainability performance.
The scripts used in our experiments are also published in our GitHub repository ().

Correctness is computed in explanation selectivity [28], shown in algorithm <ref>. The deletion of feature segments is performed as an inversion of the time series, in line with [29].

Explanation selectivity

Let $e$ be the explanation function that results in feature attribution map $A(x, t)$ describing the attributions to the predicted class, $f(x, t)$ denote the model's prediction on an input $x\in X$ at time $t$. Let $R$ be the total number of feature segments of $x$, $N$ the size of the test set $X$ and $Y$ be the corresponding ground-truth labels for $X$.

$x\in X$
$t = 1,2,...,T$ with $T$ being the duration of $x$
$A(x, t) \gets e(f, x, t)$
Define $R$ feature segments.
Sort the feature segments in descending order by their mean attribution values.

rank $r=0,1,...,R$
$x^{\text{inv@}r} \gets$ Invert feature segment $x^{(r)}$ so that $x^{(r)}=|x^{(r)}-1|$.
$\hat{y}^{\text{inv@}r}\gets f(x^{\text{inv@}r}, t)$
Let $X^{\text{inv@}r}$ denote $X$ with feature segments up to rank $r$ inverted.
rank $r=0,1,...,R$
Compute Balanced Accuracy of $\hat{Y}^{\text{inv@}r}, Y$.
Compute explanation selectivity as the AUC of the graph resulting from the performance of the model depending on the amount of feature segments inverted.

Output-completeness is measured with a model preservation check upon deletion of unimportant feature segments, specified in algorithm <ref>.

Model preservation check

Let $f(x, t)$ be a SNN model's prediction on input $x\in X$ at time $t$, $e$ be the explanation function which results in attribution map $A(x, t)$ that describes the attribution to the predicted class. Let $\epsilon$ be the threshold for feature importance (in our experiments $\epsilon = 0$).
$x\in X$
$t = 1,2,...,T$ with $T$ being the duration of $x$
$A(x, t) \gets e(f, x, t)$.
Mask $A$ where $|a|>\epsilon$.
$x_p\gets$ Perturb unmasked area of $A$.
$\hat{y}_p \gets f(x_p, t)$
Compute balanced accuracy of $\hat{Y}_p, \hat{Y}$ as a model preservation check.

Continuity is measured in max-sensitivity[31], denoted as Sens$_{\textrm{max}}$:

\begin{equation}
\label{eq: max-sensitivity}
\textrm{Sens}_{\textrm{max}}(e, f, x, t, r) = \max_{||x'-x|| \leq r}||e(f,x', t)-e(f,x, t)||
\end{equation}

where $e$ refers to the explanation, $f$ to the model, $x$ the input data, $t$ the current timestep and $r$ defines the neighborhood region in which perturbed data $x'$ is still considered as similar to $x$.

Compactness is computed as follows:

\begin{equation}
\textrm{Compactness}(A) = \frac{1}{N}\sum_i^D\sum_j^t |a_{i,j}|
\end{equation}

where $A$ is the matrix of dimension $D\times t$, i.e. input dimensionality $D \times$ the timestep to be explained. $N$ denotes the total number of explanations extracted for the experiment.


# Improving dual-microphone speech enhancement by learning cross-channel
features with multi-head attention

###### Abstract

Hand-crafted spatial features, such as inter-channel intensity difference
(IID) and inter-channel phase difference (IPD), play a fundamental role in
recent deep learning based dual-microphone speech enhancement (DMSE) systems.
However, learning the mutual relationship between artificially designed
spatial and spectral features is hard in the end-to-end DMSE. In this work, a
novel architecture for DMSE using a multi-head cross-attention based
convolutional recurrent network (MHCA-CRN) is presented. The proposed MHCA-CRN
model includes a channel-wise encoding structure for preserving intra-channel
features and a multi-head cross-attention mechanism for fully exploiting
cross-channel features. In addition, the proposed approach specifically
formulates the decoder with an extra SNR estimator to estimate frame-level SNR
under a multi-task learning framework, which is expected to avoid speech
distortion led by end-to-end DMSE module. Finally, a spectral gain function is
adopted to further suppress the unnatural residual noise. Experiment results
demonstrated superior performance of the proposed model against several state-
of-the-art models.

Index Terms—  dual-microphone speech enhancement, multi-head cross-attention,
SNR estimator, spatial cues extraction, channel-independent encoding

##  1 Introduction

Speech communication function of mobile devices has been well-designed and
widely used as a convenient tool for contacting others due to its portable
characteristics. The quality and intelligibility of the received speech can be
severely degraded by background noise if the far-end talker is in an adverse
acoustic environment. To attenuate background noise, a two-channel microphone
array is typically deployed, where a primary microphone is placed on the
bottom of a mobile phone and a secondary microphone on the top.

Conventional dual-channel speech enhancement approaches are based on signal
processing, and can be divided into two categories, the blind source
separation (BSS) approaches [1, 2], and the beamforming [3, 4] approaches.
Although these conventional approaches are fast and lightweight, their
performance and robustness are not reliable in a complex acoustic environment.

Recently, with the success of deep learning based single-channel speech
enhancement [5, 6], dual-channel speech enhancement works have been developed
on exploring deep learning approaches with conventional speech enhancement
methods. One is that the deep neural network (DNN) is used to enhance each
microphone signal separately, after which a beamformer is used to linearly
integrate the dual-channel signals [7, 8]. Experimental results show that the
DNN-beamformer approach gives better results than conventional approaches and
shows robustness in terms of various noisy types and SNR ranges. However, the
DNN only learns temporal-spectral features of each channel while ignoring the
spatial features of the target speech in the DNN-beamformer approach. In order
to sufficiently leverage the spatial features of dual-channel data, several
approaches have been proposed, which apply the spatial features, such as
interaural phase or intensity difference (IPD, IID) [9], as additional inputs
for improving objective intelligibility and perceptual quality. Despite
performance improvement by learning spectral features together with spatial
features, the mutual relationship between spatial and spectral information is
difficult to learn by a simple DNN, which may cause the under-utilization of
spatial information.

Fig. 1: Schematic diagram of MHCA-CRN. The feature maps generated by the
encoding layer are interchanged between channels after processing each
convolutional block, which is shown in the gray box. Furthermore, MHCA denotes
multi-head cross-attention.

To overcome the aforementioned limitation, this paper is motivated to design a
convolutional recurrent network (CRN), which can separately process each
channel for preserving intra-channel features while interchanging information
between encoded channels for fully exploiting the spatial information of dual-
channel data. For this purpose, this paper proposes a channel-wise encoding
structure to process each input channel independently for preserving intra-
channel features, and a multi-head cross-attention (MHCA) mechanism to boost
network performance by effectively aggregating cross-channel spatial
information. In addition, to maintain superior speech quality, the proposed
model formulates the decoder as a multi-task learning framework with an
auxiliary task of SNR estimation which has proven to be beneficial to the
perceived speech quality [10].

The rest of this paper is organized as follows: the model architecture is
presented in Section 2. Section 3 is the dataset and experimental settings.
Section 4 demonstrates the results and analysis, and a conclusion is shown in
Section 5.

##  2 Proposed MHCA-CRN Model

The proposed MHCA based CRN model (MHCA-CRN) treats the dual-microphone
enhancement as a supervised learning task, as shown in Fig. 1. First, the
proposed model separately encodes the extracted feature from each channel of
the noisy signal and interchanges information between encoded channels by
using MHCA after each downsampling block. Then the encoded features of both
channels are concatenated and fed to LSTM blocks for aggregating temporal
contexts. The output of the LSTM blocks is separately fed to the SNR estimator
and decoder blocks under a multi-task learning framework. Finally, the output
of the SNR estimator block is used to compute a frame-level spectral gain
function to remove the residual noise in the estimated spectrum.

###  2.1 Encoder-decoder structure

The encoder processes each channel independently, in order to preserve the
intra-channel feature of dual-channel data and to explicitly utilize the
cross-channel relationship. Each encoder contains several stacked 2-D
convolutional layers, each of which is followed by batch normalization [11]
and exponential linear unit (ELU) [12]. The dilation is applied to the layers
along the frequency axis.

The generated feature maps from the encoder of each channel are used as inputs
to the MHCA block, which are then interchanged between these two channels. The
main objective of the cross-channel attention block is to derive the
relationship between two channels.

The decoder is the mirror representation of each encoder except all the
convolution layers are replaced with deconvolution layers. Skip connections
are introduced to compensate for information loss during the encoding process
of the primary microphone.

###  2.2 Multi-head cross-attention

The MHCA module (shown in Fig 2) is designed for synchronizing time delay
between two channels, which holds the spatial information of the target
speaker. The MHCA takes transformed feature maps corresponding to the encoder
of primary channel, $\textbf{X}_{1}$, processed a $1\times 1$-convolution
block to form the query, and takes transformed feature maps of the encoder of
reference channel, $\textbf{X}_{2}$, to form the key-value pair by using two
$1\times 1$-convolution blocks. The proposed MHCA first computes the query,
and key-value pair for obtaining the attention component A, given by

$\textbf{A}=\text{softmax}(\textbf{Q}\textbf{K}^{\top})\textbf{V}$ (1)

where Q, K, and V denote the query, key, and value, respectively. Intuitively,
the multiplication operation between Q and K emphasizes the regions which are
slowly varying in time and have high power. What is more, the output of MHCA
block Z is computed by:

$\textbf{Z}=\text{sigmoid}(\textbf{X}_{1}+\textbf{A})$ (2)

Consequently, the Z is weight value that are re-scaled between 0 and 1 through
a sigmoid activation function.

Fig. 2: Multi-head cross-attention module.

###  2.3 SNR estimator

Previous researches have proved that directly training DNN models may
inevitably cause a certain amount of speech distortion [13]. To this end, we
propose to utilize SNR estimator to estimate frame-level SNR under a multi-
task learning framework for maintaining speech quality while reducing noise.

The input of the SNR estimator is the feature maps obtained by two LSTM
layers, then it is fed to a convolution layer with sigmoid activation to
estimate the frame-level SNR. The training target for the SNR estimator is the
mapped SNR [14], which is a mapped version of instantaneous SNR. The
definition of instantaneous SNR is as follows:

$\xi_{dB}(t,f)=20\log_{10}(|S(t,f)|/|N_{1}(t,f)|)$ (3)

where $t$ and $f$ denote the time and frame index, $\xi_{dB}(t,f)$ is scaled
in [0, 1] and can be viewed as a priori SNR, $S(t,f)$ and $N_{1}(t,f)$ are
respectively the clean and noise spectrum of the primary channel. In addition,
it is assumed that $\xi_{dB}(t,f)$ is distributed normally with mean,
$\mu_{f}$, and variance, $\sigma^{2}_{f}$:
$\xi_{dB}(t,f)\sim\mathcal{N}(\mu_{f},\sigma^{2}_{f})$. The mapped SNR is
given by

$\bar{\xi}(t,f)=1+\Big{[}\frac{1}{2}+\text{erf}\Big{(}\frac{\xi_{dB}(t,f)-\mu_{f}}{\sigma_{f}\sqrt{2}}\Big{)}\Big{]}$
(4)

where “erf” is the error function.

During inference (shown in Fig 3), the SNR estimate, $\hat{\xi}(t,f)$ is
computed by

$\hat{\xi}(t,f)=10^{((\sigma_{f}\sqrt{2}\text{erf}^{-1}(2\hat{\bar{\xi}}(t,f)-1)+\mu_{f})/10)}$
(5)

where $\hat{\xi}(t,f)$ is output from SNR estimator.

###  2.4 Loss function

Since proposed MHCA-CRN model formulates the decoder with an extra SNR
estimator under a multi-task learning framework, the proposed MHCA-CRN model
is trained by a combination of two losses. First, the MSE loss to guide the
learning of SNR estimator,

$L_{SNR}=\text{MSE}(\xi(t,f),\hat{\xi}(t,f))$ (6)

Second, the loss [15] for target speech spectrum reconstruction is given by:

$\begin{split}L_{f}=\frac{1}{T\times
F}\sum_{t=1}^{T}\sum_{f=1}^{F}|(|S(t,f)|)-(|\hat{S}(t,f)|)|\end{split}$ (7)

Finally, the total loss is

$L=L_{f}+\alpha L_{SNR}$ (8)

Since both loss values are not on the same scale, we empirically set $\alpha$
to 10.

###  2.5 Target speech reconstruction

For further suppressing the residual noise, the proposed model adopts the
spectral gain function , $G^{SNR}(t,f)$, for dual-microphone speech
enhancement, which is represented as

$G^{SNR}(t,f)=\frac{\hat{\xi}(t,f)}{\hat{\xi}(t,f)+1}$ (9)

where $\hat{\xi}(t,f)$ is the estimated SNR obtained by SNR estimator.

The computed final gain $G^{SNR}(t,f)$ is then multiplied to the estimated
spectrum $\hat{S}(t,f)$ to suppress the residual noise, which is then combined
with the noisy phase to resynthesize the time-domain waveform of the enhanced
speech, as shown in Fig 3.

Fig. 3: Diagram of target speech reconstruction. Table 1: PESQ and STOI comparison for the different models. A higher score means better performance where the BOLD text indicates the best performance for each metric. Test SNR | Channel | -5 dB | 0 dB | 5 dB | 10 dB
---|---|---|---|---|---
Meric | - | STOI(%) | PESQ | STOI(%) | PESQ | STOI(%) | PESQ | STOI(%) | PESQ
Unprocessed | Dual | 57.06 | 1.37 | 69.33 | 1.88 | 80.59 | 2.12 | 87.63 | 2.49
DeepXi [14] | Single | 77.48 | 1.88 | 90.27 | 2.36 | 92.17 | 2.66 | 95.74 | 3.11
CB-NR [16] | Dual | 54.42 | 1.46 | 68.31 | 2.03 | 77.57 | 2.38 | 88.10 | 2.74
CRN-PSM [9] | Dual | 78.20 | 1.76 | 87.30 | 2.17 | 92.76 | 2.59 | 95.76 | 2.99
DC-CRN [17] | Dual | 86.54 | 2.48 | 92.64 | 2.94 | 95.88 | 3.20 | 97.47 | 3.43
MHCA-CRN | Dual | 86.58 | 2.51 | 92.83 | 3.03 | 95.96 | 3.24 | 97.52 | 3.46
-without spectral mapping | Dual | 84.98 | 2.17 | 89.80 | 2.57 | 93.82 | 2.92 | 95.47 | 3.19
-without SNR estimator | Dual | 85.76 | 2.39 | 92.34 | 2.88 | 95.47 | 3.11 | 97.02 | 3.38
-without MHCA blocks | Dual | 82.93 | 2.08 | 90.06 | 2.64 | 94.16 | 2.98 | 95.84 | 3.16

##  3 Experimental setup

###  3.1 Data preparation

29 hours and 1 hour of speech are selected from Librispeech corpus as training
and validation sets, respectively. The noises are from the DEMAND dataset. In
addition, this paper simulate room impulse response (RIR) by the IMAGE method
[18]. Specifically, two microphones with 2cm interval are placed at the center
of a $5m$(length) $\times$ $5m$(width) $\times$ $3m$(height) room, and noise
source which are placed at 1.5m away from the center of the two microphones
and ranged from $0^{\circ}$ to $360^{\circ}$ spaced by $10^{\circ}$.

For each mixture, a speech and a slice of noise are randomly chosen and are
placed at two different positions, and the speech and noise are mixed under
the randomly SNR levels ranging from -5dB to 10dB. In addition, the frame
length is 32 ms and the hop size is 16 ms. The Hanning window is used as the
analysis window. The sampling rate is 16 kHz. A 512-point discrete Fourier
transform is used to extract complex short-time Fourier transform (STFT)
spectrograms.

###  3.2 Baselines and training details

The proposed MHCA-CRN model has been compared with four other baselines: (1)
CB-NR [16]: A coherence-based dual-channel noise reduction algorithm; (2)
DeepXi [14]: A minimum mean-square error (MMSE) approach for single-channel
speech enhancement by using deep learning; (3) CRN-PSM [9]: a CRN approach to
predict phase sensitive mask (PSM) for dual-microphone speech enhancement; (4)
DC-CRN [17]: a densely-connected CRN approach for mobile communication based
on dual-channel complex spectral mapping.

To better validate the proposed structure and strategies, we add three
ablation experiments. Firstly, we remove the SNR estimator and keep
deconvolution layers to predict the clean speech spectrum directly. Secondly,
we keep the SNR estimator only and remove the deconvolution layers for
comparing the performance between single channel deep learning based MMSE
approach, i.e. DeepXi, and dual-channel deep learning based MMSE approach.
Finally, we remove the MHCA blocks to evaluate the effectiveness of MHCA.

For the training step, all models are trained with Adam optimizer for
stochastic gradient descent (SGD) based optimization. The learning rate is set
to 0.001. All training samples are zero-padded to have the same number of time
steps as the longest sample.

Fig. 4: From left to right: Visualization of MHCA mask at layer 1, layer 3,
and layer 5.

##  4 Results and analysis

The speech enhancement systems are evaluated using Perceptual Evaluation
Speech Quality (PESQ) and Short Term Objective Intelligibility (STOI).
Experimental results are summarized in Table 1.

According to Table 1, we have the following observations: Firstly, the deep
learning based methods significantly improve both STOI and PESQ metrics, and
outperform the conventional approach, i.e., CB-NR. Secondly, the MHCA-CRN
without spectral mapping performs better than DeepXi in terms of PESQ and
STOI, which indicates that the dual-channel based model show more robustness
than single-channel based model. Finally, apart from that, the proposed MHCA-
CRN achieves better results than MHCA-CRN without SNR estimator, the proposed
MHCA-CRN consistently outperforms the state-of-the-art model, i.e., DC-CRN, in
both metrics. This indicate the effectiveness of the SNR estimator.

Table 1 furthermore shows that the MHCA has significantly improved network
performance. The visualization of MHCA masks at different convolution layers
is as shown in Fig 4. In early layers, the masks pay more attention to certain
feature map channels, for example, the $4^{th}$ and $9^{th}$ channels of the
feature map are highlighted by the mask at the first layer. What is more, when
the layer is going deeper, the shape of the MHCA mask is changing in order to
synchronize the time delay between two channels. Take, for example, mask at
$3^{rd}$ layer highlights several channels but at different time frames, while
mask at $7^{th}$ highlights certain channels at all time frames. This is
demonstrated that the spatial cues between dual-channel can be implicitly
exploited by MHCA.

##  5 Conclusion

In this paper, we propose an MHCA-CRN for dual-microphone speech enhancement,
aiming to straightforwardly and efficiently exploit spatial information. The
model adopts a channel-wise encoding structure to process each input channel
independently for preserving intra-channel features and uses the MHCA
mechanism to aggregate cross-channel spatial information. Furthermore, an SNR
estimator is adopted along with the decoder to estimate frame-level SNR under
a multi-task learning framework for further improving the speech quality.
Finally, a spectral gain function is adopted to remove unnatural residual
noise. Experimental results show that our proposed method can suppress the
noise meanwhile maintaining better intelligibility.

## References

  * [1] Mohamed Djendi and Rédha Bendoumia,  “A new adaptive filtering subband algorithm for two-channel acoustic noise reduction and speech enhancement,”  Computers & Electrical Engineering, vol. 39, no. 8, pp. 2531–2550, 2013.

  * [2] Rahima Henni, Mohamed Djendi, and Mustapha Djebari,  “A new efficient two-channel fast transversal adaptive filtering algorithm for blind speech enhancement and acoustic noise reduction,”  Computers & Electrical Engineering, vol. 73, pp. 349–368, 2019\.
  * [3] S Applebaum and Dean Chapman,  “Adaptive arrays with main beam constraints,”  IEEE Transactions on Antennas and Propagation, vol. 24, no. 5, pp. 650–662, 1976.
  * [4] K Buckley and L Griffiths,  “An adaptive generalized sidelobe canceller with derivative constraints,”  IEEE Transactions on antennas and propagation, vol. 34, no. 3, pp. 311–319, 1986.
  * [5] J Rouat,  “Computational auditory scene analysis: Principles, algorithms, and applications (wang, d. and brown, gj, eds.; 2006)[book review],”  IEEE Transactions on Neural Networks, vol. 19, no. 1, pp. 199–199, 2008.
  * [6] Xinmeng Xu, Yang Wang, Dongxiang Xu, Yiyuan Peng, Cong Zhang, Jie Jia, and Binbin Chen,  “Multi-stage progressive speech enhancement network,”  Proc. Interspeech 2021, pp. 2691–2695, 2021.
  * [7] Xiong Xiao, Shinji Watanabe, Hakan Erdogan, Liang Lu, John Hershey, Michael L Seltzer, Guoguo Chen, Yu Zhang, Michael Mandel, and Dong Yu,  “Deep beamforming networks for multi-channel speech recognition,”  in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5745–5749.
  * [8] Hakan Erdogan, John R Hershey, Shinji Watanabe, Michael I Mandel, and Jonathan Le Roux,  “Improved mvdr beamforming using single-channel mask prediction networks.,”  in Interspeech, 2016, pp. 1981–1985.
  * [9] Ke Tan, Xueliang Zhang, and DeLiang Wang,  “Real-time speech enhancement using an efficient convolutional recurrent network for dual-microphone mobile phones in close-talk scenarios,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5751–5755.
  * [10] Aaron Nicolson and Kuldip K Paliwal,  “Masked multi-head self-attention for causal speech enhancement,”  Speech Communication, vol. 125, pp. 80–96, 2020.
  * [11] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  in International conference on machine learning. PMLR, 2015, pp. 448–456.
  * [12] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter,  “Fast and accurate deep network learning by exponential linear units (elus),”  arXiv preprint arXiv:1511.07289, 2015.
  * [13] Chengyu Zheng, Xiulian Peng, Yuan Zhang, Sriram Srinivasan, and Yan Lu,  “Interactive speech and noise modeling for speech enhancement,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 14549–14557.
  * [14] Aaron Nicolson and Kuldip K Paliwal,  “Deep Xi as a front-end for robust automatic speech recognition,”  in 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE). IEEE, 2020, pp. 1–6.
  * [15] Ashutosh Pandey and Deliang Wang,  “On adversarial training and loss functions for speech enhancement,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5414–5418.
  * [16] Youna Ji, Jun Byun, and Young-cheol Park,  “Coherence-based dual-channel noise reduction algorithm in a complex noisy environment.,”  in INTERSPEECH, 2017, pp. 2670–2674.
  * [17] Ke Tan, Xueliang Zhang, and DeLiang Wang,  “Real-time speech enhancement for mobile communication based on dual-channel complex spectral mapping,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6134–6138.
  * [18] Jont B Allen and David A Berkley,  “Image method for efficiently simulating small-room acoustics,”  The Journal of the Acoustical Society of America, vol. 65, no. 4, pp. 943–950, 1979.