# Explaining Speech Classification Model Predictions

Eliana Pastor♣, Alkis Koudounas♣, Giuseppe Attanasio♡, Dirk Hovy♡, Elena
Baralis♣

♣ Politecnico di Torino, Turin, Italy
♡ Bocconi University, Milan, Italy

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# Explaining Speech Classification Models via Input Perturbation

Eliana Pastor♣, Alkis Koudounas♣, Giuseppe Attanasio♡, Dirk Hovy♡, Elena
Baralis♣

♣ Politecnico di Torino, Turin, Italy
♡ Bocconi University, Milan, Italy

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# What Did You Just Say? Explaining Speech Classification Models via Input
Perturbation

Eliana Pastor♣, Alkis Koudounas♣, Giuseppe Attanasio♡, Dirk Hovy♡, Elena
Baralis♣

♣ Politecnico di Torino, Turin, Italy
♡ Bocconi University, Milan, Italy

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# Explaining Speech Classification Models via Word-Level
Audio Segments and Paralinguistic Features

Eliana Pastor♣, Alkis Koudounas♣, Giuseppe Attanasio♡, Dirk Hovy♡, Elena
Baralis♣

♣ Politecnico di Torino, Turin, Italy
♡ Bocconi University, Milan, Italy

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recent advances in eXplainable AI (XAI) have provided new insights into how
models for vision, language, and tabular data operate. However, few approaches
exist for understanding speech models. Existing work focuses on a few spoken
language understanding (SLU) tasks, and explanations are difficult to
interpret for most users. We introduce a new approach to explain speech
classification models. We generate easy-to-interpret explanations via input
perturbation on two information levels. 1) Word-level explanations reveal how
each word-related audio segment impacts the outcome. 2) Paralinguistic
features (e.g., prosody and background noise) answer the counterfactual: “What
would the model prediction be if we edited the audio signal in this way?” We
validate our approach by explaining two state-of-the-art SLU models on two
speech classification tasks in English and Italian. Our findings demonstrate
that the explanations are faithful to the model’s inner workings and plausible
to humans. Our method and findings pave the way for future research on
interpreting speech models.
Note: This preprint documents our approach and preliminary results. We are
working on expanding the evaluations and discussions.

##  1 Introduction

Recently, several eXplainable AI (XAI) techniques have been proposed to gain
insights into how models get to their outputs. Seminal work in computer vision
used gradients (Simonyan et al., 2013; Sundararajan et al., 2017; Selvaraju et
al., 2022, inter alia) or input perturbation (Zeiler and Fergus, 2013) to
build input saliency maps, i.e., visual artifacts to highlight the most
relevant parts for the prediction. Similar solutions have also been proposed
to explain language (Ribeiro et al., 2016; Sanyal and Ren, 2021; Jacovi et
al., 2021, inter alia) and tabular (Lundberg and Lee, 2017) models.

Figure 1: Explanation with word-level and paralinguistic attributes for a
sample in Fluent Speech Commands (Lugosch et al., 2019). Word-level audio-
transcript alignment represented through color. Word-level attributions to
explain the Increase (green, left boxes) and Bedroom (orange, right) target
classes.

And while there is significant progress in explaining model predictions for
image, text, and structured data models, explanations for Spoken Language
Understanding (SLU) models remain largely unexplored. Speech data consists of
both explicit content and discrete words, but also acoustic features,
linguistic variations, and paralinguistic cues, making it more complex to
decipher each element’s contribution to the model predictions. Existing
approaches use frequency features, e.g., spectrogram segments (Becker et al.,
2018; Frommholz et al., 2023). However, spectrograms are difficult to
interpret for most humans. Wu et al. (2023a) have instead proposed identifying
time segments, e.g., those corresponding to relevant phonemes. However,
meaningful, phoneme-level explanations are fine-grained and only serve a
limited number of tasks like Automatic Speech Recognition (ASR) or Phoneme
Recognition. They fail to capture more interpretable word-level attribution
needed for semantically-intensive tasks such as Speech Classification.
Moreover, these methods entirely overlook any paralinguistic aspects, e.g.,
prosody or channel noise, which carry information.

We propose a new approach to explaining speech models, producing easy-to-
interpret explanations including paralinguistic features. We base our approach
on input perturbation, an established XAI method. Our explanations provide
insights on two different but complementary levels: The uttered content and
paralinguistic features.

To quantify the contribution of each part of the utterance, we compute word-
level attribution scores as follows. First, we align the audio signal to its
transcript and get word-level timestamps. Then, we use these timestamps to
iteratively mask audio segments. Finally, we estimate word-level contributions
as the difference in the model’s output between the original signal and the
masked one. We follow a similar perturbation-based approach to measure the
contribution of paralinguistic aspects. Given an input utterance, we transform
the raw audio signal and measure the effect on the model’s prediction. We
perturb pitch to account for prosody, and audio stretching, background noise,
and reverb levels for channel-related aspects. Figure 1 shows a sample
explanation.

We test our approach by explaining wav2vec-2.0 (Baevski et al., 2020) and
XLS-R (Babu et al., 2022), two state-of-the-art SLU models, on two datasets
for Intent Classification and one for Emotion Recognition in English and
Italian. We assess the quality of our explanations under the faithfulness and
plausibility paradigms (Jacovi and Goldberg, 2020). Our experimental results
demonstrate that the explanations are faithful to the model’s inner workings
and plausible to humans.

#### Contributions.

We introduce a new method for explaining speech classification models. Using
word-level audio segments and paralinguistic features, it generates easy-to-
interpret visualizations that are faithful and plausible across two models,
languages, and tasks. We release the code at
https://github.com/elianap/SpeechXAI to encourage future research at the
intersection of SLU and interpretability.

##  2 Methodology

We generate explanations by assigning a single numerical attribution score to
each uttered word (§2.1) and paralinguistic feature (§2.2). Each score is
generated via input perturbation and quantifies the contribution the entity
(either a word or a paralinguistic feature) had in predicting a given target
class.

###  2.1 Word-level Audio Segment Attribution

We compute word-level contribution in two steps. First, we perform a word-
level audio-transcript alignment. In practice, we extract beginning and ending
timestamps for each uttered word. If no transcript or timestamp is available,
we use WhisperX (Bain et al., 2023) to generate it along with the word-level
timestamps. The resulting timestamps define a set of audio segments
corresponding to words in the time domain. See Figure 1 (top) for an example.

Second, we compute each segment’s contribution by masking it and measuring how
the model’s prediction changes. More formally, let $x$ be an utterance and let
$\\{x_{1},..,x_{n}\\}$ the set of $n$ word-level audio segments within.
Consider a speech classification model $f$ applied for tasks such as intent
classification or emotion recognition. Let $f(y=k|x)$ be the output
probability of the model $f$ for class $k$ given the input utterance $x$. We
define the relevance $r(x_{i})\in\mathbb{R}$ of each segment $x_{i}$ to the
model’s prediction for a target class $k$ as:

$r(x_{i})=f(y=k|x)-f(y=k|x\setminus x_{i})$ (1)

where $x\setminus x_{i}$ refers to the utterance when the segment $x_{i}$ is
masked. Following Wu et al. (2023a), we mask out segments by zeroing the
corresponding samples in the time domain.

Higher values for $r(x_{i})$ indicate greater relevance of the segments to the
prediction. A positive score indicates that the segment contributes positively
to the probability of belonging to a specific class, while a negative score
suggests that the segment may “push” the prediction toward another class. See
Figure 1 (middle) for an example.

###  2.2 Paralinguistic Attributions

Speech includes not only the semantic information conveyed by words but also
additional paralinguistic information communicated through the speaker’s voice
or from external conditions, such as pitch, speaking rate, and background
noise levels. We investigate the relevance of paralinguistic features by
introducing ad hoc perturbations of the utterances and studying the resulting
changes in class prediction probabilities.

Let $p(x)$ be a paralinguistic feature of interest of utterance $x$. For
example, it can correspond to the pitch of the utterance. We transform $x$
into $\widetilde{x}$ such that the value of feature $p(\widetilde{x})$ varies
from $p(x)$. Rather than a random perturbation, we control the induced
transformation so that it is interpretable, and we can trace back the impact
to feature $p$. For instance, we may increase the pitch.

We consider a series of transformation $\widetilde{X}_{p}$ =
{$\widetilde{x}_{1},..,\widetilde{x}_{t}$} to study the impact of changing the
paralinguistic feature $p$ on the model’s predictions. We compute the
relevance of $p(x)$ as follows.

$r(p(x))=f(y=k|x)-\frac{1}{|X|}\sum_{\widetilde{x}\in\widetilde{X}}f(y=k|\widetilde{x})$
(2)

The term
$\frac{1}{|X|}\sum_{\widetilde{x}\in\widetilde{X}}f(y=k|\widetilde{x})$
represents the average change in the prediction probability when perturbing
$p(x)$. In addition, we visualize the terms $f(y=k|x)-f(y=k|\widetilde{x})$ in
a heatmap representation to visualize the impact of each perturbation.
Heatmaps provide an intuitive way to observe the changes in prediction
probabilities as we vary the paralinguistic features.

##  3 Experiments

###  3.1 Experimental Setting

#### Paralinguistic Features.

In the experiments, we consider transformations of the pitch, time stretching,
the introduction of background white noise, and of reverberation. We describe
the libraries adopted for the transformations in our repository.

#### Datasets.

We evaluate our explanation on three publicly available datasets and two
tasks: Fluent Speech Commands (FSC; Lugosch et al., 2019) and the Italian
Intent Classification Dataset (ITALIC; Koudounas et al., 2023) datasets for
Intent Classification (IC) task and the IEMOCAP (Busso et al., 2008) for
Emotion Recognition (ER). FSC is a widely utilized benchmark dataset for the
IC task. Its test set comprises 3793 audio samples, each characterized by
three slots — action, object, and location — whose combination defines the
intent. ITALIC is an intent classification dataset for the Italian language.
The dataset includes 60 intents, and the test set consists of 1441 samples. We
use the “Speaker” setup, wherein the utterances of each speaker belong to a
single set among the train, validation, and test. IEMOCAP is a dataset for the
ER task annotated with emotion labels (i.e., happiness, anger, sadness,
frustration, and neutral state). It consists of recorded interactions between
pairs of actors engaged in scripted scenarios involving ten actors. Among its
five sessions, we consider Session ‘1’, consisting of 942 utterances.

#### Models.

We consider the monolingual wav2vec 2.0 base (Baevski et al., 2020) for FSC
and IEMOCAP. We use the public fine-tuned checkpoints Yang et al. (2021). We
use the multilingual XLS-R (Babu et al., 2022) for ITALIC and its fine-tuned
checkpoints (Koudounas et al., 2023).

###  3.2 Qualitative evaluation

In this section, we show how our explanation method reveals the reasons behind
a model prediction from the perspective of an individual prediction and
globally across the entire dataset.

#### Individual level.

Consider the FSC dataset and wav2vec 2.0 base fine-tuned-model. For a specific
utterance with transcription ‘Turn up the bedroom heat’, the model correctly
predicts increase as the action, heat as the object, and bedroom as the
location, fully identifying the intent. We may wonder: Is it correct for the
right reasons? Which are the paralinguistic features whose change would impact
the predictions? Our approach answers these questions.

| Turn | up | the | bedroom | heat.
---|---|---|---|---|---
act=increase | 0.250 | 0.545 | 0.260 | 0.139 | 0.021
obj=heat | 0 | 0 | 0 | 0.014 | 0.550
loc=bedroom | 0.002 | 0.006 | 0.087 | 0.997 | 0.323

Table 1: Example of word-level audio segment explanation; FSC dataset. The
higher the value, the more the audio segment is relevant for the prediction.

| pitch | stretch |  |
---|---|---|---|---
| down | up | down | up | reverb | noise
act=increase | 0 | 0.01 | 0.19 | 0.04 | 0.74 | 0.54
obj=heat | 0 | 0 | 0 | 0 | 0 | 0.86
loc=bedroom | 0.02 | 0 | 0.03 | 0.01 | 0.20 | 0.97

Table 2: Example of paralinguistic explanation, FSC dataset, instance in Table
1. The higher the value, the more the perturbations on the paralinguistic
feature impact the prediction.

Table 1 shows the word-level audio segment explanation for this utterance
computed with respect to the predicted class for each intent slot. For each
segment, we report its importance for the prediction. We visualize only the
word-level transcriptions for convenience and visualization constraints.
However, recall that our approach works end-to-end at the audio level, and
importance scores relate to audio segments. The explanation reveals that the
segment associated with the word ‘up’ is the most relevant term for the action
increase. Spoken words ‘heat’ and ‘bedroom’ are associated with the target
object heat and the target location bedroom. Hence, we can say that the
explanation is plausible and trust the model for this prediction.

Figure 2: Heatmap of the prediction differences when varying the
paralinguistic information. The higher the value, the more the paralinguistic
changes impact the prediction.

Table 2 shows the paralinguistic explanation. The prediction for this instance
is greatly affected by the introduction of noise. The reverberation impacts
the prediction for the slot action and slightly for the location; on the other
hand, the object prediction is not affected. The pitch transformation we
introduce does not impact the predictions, both when increasing (‘up’) and
lowering (‘down’) the pitch. Finally, we reveal that shrinking the utterance
duration (time ‘stretch down’) and hence increasing the utterance speed
impacts only the action increase.

We can further inspect the impact of paralinguistic transformations on
predictions by visualizing the prediction difference for each individual
transformation via heatmaps. Figure 2 shows the prediction difference when
stretching the audio and introducing reverberation. Note that ‘1’ and ‘0’ are
the reference values for time stretching and reverberation, respectively, and
hence correspond to the original utterance. We observe no impact when
extending the utterance duration (values $\geq$1.05). At the same time, we
note that the prediction probability of the action increase highly changes
when increasing the utterance speed (which corresponds to values 0.55-0.7).

Our approach reveals the relevant factors for individual predictions, and it
is, hence, a tool for model understanding. We include further examples of
explanations in our repository.

Figure 3: Summary plot of average importance of word-level audio segments,
separately for each predicted class. Top-15 segments, action label of FSC
dataset.

| pitch | stretch |  |
---|---|---|---|---
| down | up | down | up | reverb | noise
action | 0.04 | 0.03 | 0.13 | 0.09 | 0.27 | 0.59
object | 0.02 | 0.01 | 0.07 | 0.05 | 0.17 | 0.69
location | 0.01 | 0.01 | 0.06 | 0.04 | 0.11 | 0.35

Table 3: Average paralinguistic attributions for the FSC dataset. The higher
the score, the more the corresponding change in the paralinguistic feature
impacts the prediction probability.

|  | FSC | ITALIC | IEMOCAP
---|---|---|---|---
|  | action | object | location | intent | emotion
WA-L1O | Comprehensiveness | 0.619 | 0.623 | 0.465 | 0.693 | 0.508
random | 0.294±0.005 | 0.246±0.003 | 0.195±0.006 | 0.324±0.005 | 0.273±0.005
WA-L1O | Sufficiency | 0.158 | 0.083 | 0.065 | 0.164 | 0.311
random | 0.474±0.004 | 0.444±0.008 | 0.339±0.006 | 0.557±0.004 | 0.450±0.002

Table 4: Comprehensiveness and Sufficiency results for our word attribution
explanation via leave-one-out (WA-L1O) and random attribution for the FSC,
ITALIC, and IEMOCAP datasets, separately for each label. For
comprehensiveness, the closer to one, the better. For sufficiency, the closer
to zero, the better.

#### Global level.

We can also analyze model behavior across the entire dataset. We aggregate the
importance scores of word audio segments or paralinguistic levels to
investigate the global influence of each component.

Figure 3 shows a summary plot for the word-level audio segment explanations of
wav2vec 2.0 predictions on FSC test set for the label ‘action’. We first
compute the explanations for the predicted classes. Then, we aggregate audio
segments corresponding to the same transcripted word after basic processing
(i.e., lowercase and punctuation removal). We report the top 15 segments with
the highest average importance. Each term represents the average importance
scores separately for each class.

The summary plot reveals which spoken words are associated with a predicted
class. From Figure 3, we infer that the importance scores for some spoken
words such as ‘language’, ‘newspaper’, and ‘cooler’ across the entire test set
are associated with a single class value. Each class corresponds to a
plausible value (‘change language’, ‘bring’, and ‘decrease’), making the
explanations plausible. In cases where a term is associated with multiple
labels, the summary plot can serve as a debugging tool. For instance, the
spoken word ‘pause’ is correctly linked to the predicted action ‘deactivate’
but erroneously connected to ‘decrease’. Similar considerations apply to the
other two labels we include in the repository.

Table 3 shows the average importance score of paralinguistic explanations
aggregated for each label. The results reveal that adding background noise
globally impacts the model prediction. The reverberation affects more the
predictions of the action label than the ones of the location. We observe
higher average importance scores for the action label for the time stretching
component, specifically when compressing the utterance duration (‘stretch
down’) and, therefore, increasing the audio speed. Conversely, the pitch
transformation we introduce generally does not impact the predictions.

###  3.3 Quantitative evaluation

In this section, we quantitatively evaluate the quality of our explanations. A
critical requirement for explanations is their faithfulness to the model.
Faithfulness measures evaluate how accurately the explanation reflects the
model’s inner workings (Jacovi and Goldberg, 2020).

#### Metrics.

We generalize two widely adopted measures for the XAI literature:
comprehensiveness and sufficiency (DeYoung et al., 2020). These notions were
originally designed for token-level explanations for text classification,
where explainers assign a relevance score to each token. This scenario is
close to our word-level audio segment explanations. Intuitively, we consider
audio segments rather than tokens. Comprehensiveness evaluates whether the
explanation captures the audio segments the model used to make the prediction.
We measure it by progressively masking the audio segments highlighted by the
explanation, observing the change in probability, and finally averaging the
results. A high value of comprehensiveness indicates that the audio segments
highlighted by the explanations are relevant to the prediction. Conversely,
sufficiency captures if the audio segments in the explanation are sufficient
for the model to make the prediction. Opposed to comprehensiveness, we
preserve only the relevant audio segments and compute the prediction
difference. A low score indicates that the audio segments in the explanations
indeed drive the prediction. We include the extended description of the two
metrics in our repository.

#### Baseline.

We assess the quality of explanations compared to a random explainer. The
random explainer assigns a random score in the range [-1, 1] to each word
audio level segment.

#### Results.

Table 4 shows the comprehensiveness and sufficiency results on the FSC,
ITALIC, and IEMOCAP datasets, separately for each label. We generate our word-
level explanations with respect to the predicted class. For the random
baseline, we consider five rounds of generations, and we report average and
standard deviation. The results show that our word-level audio segment
explanations computed by leaving one out audio segments (WA-L1O in Table 4)
highly outperform the random baseline for both metrics.

##  4 Related Work

###  4.1 Interpretability for Speech Models

Multiple studies have adopted Layer-wise Relevance Propagation (LRP) (Bach et
al., 2015), initially proposed for image classification explanations, to
explain prediction across diverse audio analysis tasks. Most of these works
represent explanations as time-frequency heatmaps over spectrograms, such as
Becker et al. (2018) for gender and digit audio classification, Frommholz et
al. (2023) for audio event classification, and Colussi and Ntalampiras (2021)
for the task of urban sound classification. Wang et al. (2023) used heatmaps
over ad-hoc terms (carrier and modulation frequency) for the specific task of
audio classification of playing techniques (e.g., vibrato, trill, tremolo) in
the context of music signal analysis. While experts can find spectrograms a
familiar tool for understanding audio data, these visual representations can
be challenging for laypersons to interpret.

Becker et al. (2018) also adopt the LRP method to derive the relevance score
of individual samples with respect to the input waveform in the time domain.
Interpreting explanations as sets of individual samples can pose challenges,
such as a lack of abstraction and context of isolated data points. We advocate
for prioritizing a more user-friendly and intuitive approach to explanation.
In this line of intent, rather than samples, Wu et al. (2023b) assign
relevance scores to audio frames, i.e., raw data bins in time dimension of
predefined size. The work generalizes two XAI techniques from image
classification and explains Automatic Speech Recognition (ASR) systems. Mishra
et al. (2017) propose to describe the data to explain via interpretable
representations. Their method involves segmenting the data into equal-width
segments within the time, frequency, or time-frequency domains. Subsequently,
they apply the LIME explanation method (Ribeiro et al., 2016) to these
interpretable representations. However, these temporal explanations may be
affected by the size of the audio segments chosen for analysis. Moreover, they
are not grounded in spoken words or paralinguistic information, hindering
interpretability for semantically intensive contexts such as speech
classification.

The work by Wu et al. (2023a) aligns with our direction, as it not only tests
fixed-width audio segments but also audio segments aligned with phonemes.
However, the approach requires phoneme-level annotations, and therefore, it is
limited to evaluation purposes when such labeling is available. Moreover, the
method is suitable for the phoneme recognition task. In contrast, our approach
offers a more generalized solution to any Speech Language Understanding (SLU)
classification model and data. We automatically derive audio segments at the
word level, coupled with their transcriptions, via state-of-the-art speech
transcription systems. Furthermore, our approach stands out as the first to
offer explanations that study the impact of paralinguistic features on
predictions, presenting these insights in an interpretable form.

###  4.2 Explanation by Occlusion

Removing parts of input data to understand their impact is a well-established
strategy in explainability (Covert et al., 2021). Different domains use
various techniques for removing or masking parts of the data. Standard
techniques for image data include noise addition, blurring, or replacing via a
grey area. Using a special mask token or directly removing words is often
employed in text analysis. For structured data, analyzing the effects based on
average values is a typical approach (Covert et al., 2021). For speech data,
Wu et al. (2023a) have applied a similar technique to phonemes, using signal
zeroing for masking. However, the masking is adopted for generating
perturbation used by LIME explanation method (Ribeiro et al., 2016), and they
are at the phoneme level.

##  5 Conclusion

We propose a novel perturbation-based explanation method that explains the
predictions of speech classification models regarding word-level audio
segments and paralinguistic features. Our results show that our explanations
can be a tool for model understanding.

## Limitations

Our work has some technical and design limitations. From the technical
perspective, word-level segment attributions are computed by masking one-word
segment at a time, thus not considering the intersectional effect of multiple
masked words. We plan to experiment with different masking strategies.
Moreover, word-level explanations might not be the most helpful explanation in
specific speech classification tasks, e.g., spoken language identification or
speaker identification. We are accounting for this limitation by including
paralinguistic explanations, but we will also explore new methods. We will
also investigate the impact of the perturbation techniques and third-party
speech libraries on paralinguistic explanations. From the experimental design
perspective, we are currently reporting self-evaluation for plausibility. We
will conduct a comprehensive user study to evaluate it thoroughly.

## References

  * Babu et al. (2022) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022.  XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale.  In _Proc. Interspeech 2022_ , pages 2278–2282.
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Advances in Neural Information Processing Systems_ , volume 33.
  * Bain et al. (2023) Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023.  Whisperx: Time-accurate speech transcription of long-form audio.  _INTERSPEECH 2023_.
  * Becker et al. (2018) Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. 2018.  Interpreting and explaining deep neural networks for classification of audio signals.  _arXiv preprint arXiv:1807.03418_.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008.  Iemocap: interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_ , 42:335–359.
  * Colussi and Ntalampiras (2021) Marco Colussi and Stavros Ntalampiras. 2021.  Interpreting deep urban sound classification using layer-wise relevance propagation.  _arXiv preprint arXiv:2111.10235_.
  * Covert et al. (2021) Ian C Covert, Scott Lundberg, and Su-In Lee. 2021.  Explaining by removing: A unified framework for model explanation.  _The Journal of Machine Learning Research_ , 22(1):9477–9566.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A benchmark to evaluate rationalized NLP models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online. Association for Computational Linguistics.
  * Frommholz et al. (2023) Annika Frommholz, Fabian Seipel, Sebastian Lapuschkin, Wojciech Samek, and Johanna Vielhaben. 2023.  Xai-based comparison of input representations for audio event classification.  _arXiv preprint arXiv:2304.14019_.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online. Association for Computational Linguistics.
  * Jacovi et al. (2021) Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. 2021.  Contrastive explanations for model interpretability.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1597–1611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Koudounas et al. (2023) Alkis Koudounas, Moreno La Quatra, Lorenzo Vaiani, Luca Colomba, Giuseppe Attanasio, Eliana Pastor, Luca Cagliero, and Elena Baralis. 2023.  Italic: An italian intent classification dataset.  In _Interspeech 2023, 24rd Annual Conference of the International Speech Communication Association, Dublin, Ireland, 20-24 August 2023_. ISCA.
  * Lugosch et al. (2019) Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019.  Speech model pre-training for end-to-end spoken language understanding.  In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association_ , pages 814–818.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc.
  * Mishra et al. (2017) Saumitra Mishra, Bob L Sturm, and Simon Dixon. 2017.  Local interpretable model-agnostic explanations for music content analysis.  In _ISMIR_ , volume 53, pages 537–543.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Sanyal and Ren (2021) Soumya Sanyal and Xiang Ren. 2021.  Discretized integrated gradients for explaining language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10285–10299, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Selvaraju et al. (2022) RR Selvaraju, M Cogswell, A Das, R Vedantam, D Parikh, and D Batra. 2022.  Grad-cam: Visual explanations from deep networks via gradient-based localization. arxiv 2016.  _arXiv preprint arXiv:1610.02391_.
  * Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR.
  * Wang et al. (2023) Changhong Wang, Vincent Lostanlen, and Mathieu Lagrange. 2023.  Explainable audio classification of playing techniques with layer-wise relevance propagation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Wu et al. (2023a) Xiaoliang Wu, Peter Bell, and Ajitha Rajan. 2023a.  Can we trust explainable ai methods on asr? an evaluation on phoneme recognition.  _arXiv preprint arXiv:2305.18011_.
  * Wu et al. (2023b) Xiaoliang Wu, Peter Bell, and Ajitha Rajan. 2023b.  Explanations for automatic speech recognition.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Yang et al. (2021) Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. 2021.  Superb: Speech processing universal performance benchmark.  _arXiv preprint arXiv:2105.01051_.
  * Zeiler and Fergus (2013) MD Zeiler and R Fergus. 2013.  Visualizing and understanding convolutional networks. arxiv.  _arXiv preprint arXiv:1311.2901_.