# ProtoEEGNet: An Interpretable Approach for Detecting Interictal Epileptiform
Discharges

Dennis Tang 1, Frank Willard†1, Ronan Tegerdine1, Luke Triplett1, Jon
Donnelly1, Luke Moffett1,
Lesia Semenova1, Alina Jade Barnett1, Jin Jing2, Cynthia Rudin 1, Brandon
Westover‡23
1Department of Computer Science, Duke University
2Department of Neurology, Massachusetts General Hospital
3Harvard Medical School, Beth Israel Deaconess Medical Center, McCance Center
for Brain Health Corresponding author<EMAIL_ADDRESS>authorsCo-
senior authors

###### Abstract

In electroencephalogram (EEG) recordings, the presence of interictal
epileptiform discharges (IEDs) serves as a critical biomarker for seizures or
seizure-like events. Detecting IEDs can be difficult; even highly trained
experts disagree on the same sample. As a result, specialists have turned to
machine-learning models for assistance. However, many existing models are
black boxes and do not provide any human-interpretable reasoning for their
decisions. In high-stakes medical applications, it is critical to have
interpretable models so that experts can validate the reasoning of the model
before making important diagnoses. We introduce ProtoEEGNet, a model that
achieves state-of-the-art accuracy for IED detection while additionally
providing an interpretable justification for its classifications.
Specifically, it can reason that one EEG looks similar to another
“prototypical” EEG that is known to contain an IED. ProtoEEGNet can therefore
help medical professionals effectively detect IEDs while maintaining a
transparent decision-making process.

##  1 Introduction

Accurately identifying the presence of interictal epileptiform discharges
(IEDs) is important when analyzing EEG signals, as they can be a critical
biomarker for epilepsy [1, 2, 3]. However, detecting IEDs is a challenging
task even for trained experts, who frequently disagree on whether an EEG
contains an IED as shown in [4] and Figure 3 in the Appendix. This difficulty
has led clinicians and researchers alike to turn towards machine learning
models to assist them [5, 6, 7, 8, 9]. However, many of these approaches are
black boxes that assert a prediction and offer little to no explanation for
their decisions. Moreover, while black box models may perform well on
experimental benchmarks, they may struggle to generalize to real-world
conditions [10, 11]. These shortcomings are especially problematic in high-
stakes medical settings, as black box models can confuse or frustrate
clinicians with incorrect predictions from faulty reasoning. While post-hoc
analysis methods have been applied to EEG models [12], they were criticized
for being not faithful to the original model and insufficient to ascertain the
model’s underlying behavior [13, 14]. Therefore, it is critical to adopt
inherently interpretable models in high-stakes decision-making [13, 15] as
they enable users to review the reasoning of the model for logical consistency
and plausibility. However, existing interpretable methods for IED detection
such as K-nearest neighbors and template matching are slower [16] or less
accurate [17] when compared to black boxes, making them impractical for
clinical use.

To address these challenges, we introduce ProtoEEGNet, an interpretable neural
network that detects the occurrence of IEDs in EEGs. We demonstrate that
ProtoEEGNet rivals the state-of-the-art IED detection model SpikeNet while
additionally providing a human interpretable justification for its decisions.
Specifically, ProtoEEGNet learns “prototypical” IEDs and interprets EEGs based
on their resemblance to the learned prototypes by reasoning that "This EEG
looks like that EEG" (as demonstrated in Figure 1). This aids medical experts
in better understanding why our model classified a given EEG as containing an
IED.

Figure 1: Example Of ProtoEEGNet’s Reasoning. ProtoPEEGNet makes its
predictions by comparing the current EEG to learned prototypical EEGs for both
spike and non-spike cases. The similarity score is calculated based on cosine
similarity in the latent space representation of the EEGs. The class
connection indicates the weight between the prototype and the predicted class
in the final layer. The two scores are multiplied into a final “points
contributed” score used for classification.

##  2 Methods

Data. We obtained 16,646 1-second EEG samples that spanned a range of ages and
settings from patients at Massachusetts General Hospital (MGH). The EEGs were
filtered (60-Hz notch, 0.5-Hz high-pass) and re-sampled to 128 Hz for
consistency. Each EEG was labeled by 8 physician subspecialists who annotated
whether or not they believed a sample contained an IED, following the
annotation procedure of Jing et al. [5]. Given the task difficulty and high
annotator disagreement, we treated each annotator’s label as a “vote”. The
proportion of expert votes indicating an EEG (0/8 to 8/8) was used as training
labels to create 9 classes, each representing a possible number of votes. A
visualization of class differences is shown in Appendix A.5. Data was
partitioned into 73% training, 12% validation, and 15% testing splits.

ProtoEEGNet. The architecture of ProtoEEGNet (shown in Figure 2a) builds upon
that of ProtoPNet, an interpretable classifier composed of a pre-trained
image-recognition backbone and subsequent “prototype” layers [18, 19]. As
compared to ProtoPNet, ProtoEEGNet uses (1) a modified SpikeNet backbone and
(2) multi-channel 1D EEG signals rather than 2D images. By learning to
identify 9 classes based on the proportion of expert votes instead of binary
labels, ProtoEEGNet is able to learn prototypes associated with multiple
levels of ambiguity (e.g., 8/8 vote versus 4/8 vote prototypes). At inference
time, the latent representation of a new sample is compared with the latent
representation of all learned prototypes to compute a vector of cosine
similarity scores, as detailed in Appendix A.1 and A.6. These scores are then
passed through a linear layer that computes class logits.

For the backbone, we adopted SpikeNet [5], an uninterpretable CNN for IED
detection. SpikeNet was modified by: (1) replacing BatchNorm with LayerNorm
layers due to their empirical improvement and (2) increasing the kernel size
of the final convolution layer from 8 to 10. This increase produced a 1x1
latent output, ensuring that ProtoEEGNet prototypes represented an entire EEG
(as in Figure 2) rather than a specific sub-feature. EEGs have nuanced, inter-
channel dependencies such that it is necessary to evaluate EEGs against full
montage prototypes [20]. We also trained the model end-to-end, thereby
updating the weights of the SpikeNet backbone to fit the new architecture. A
detailed description of the model can be found in Appendix A.1.

At inference time, the latent representation of a new sample is compared with
the latent representation of all learned prototypes to compute a vector of
cosine similarity scores. These scores are then passed through a linear layer
that computes class logits.

Figure 2:  (a) ProtoEEGNet Architecture. An input EEG is propagated through
the backbone which outputs a latent representation of the EEG. The prototype
layers then compute the cosine similarities between the latent representation
and the prototypes which are passed to a fully connected layer that computes
classification logits. Optionally, predicted probabilities can be binarized
into positive and negative class predictions when yes-no spike detection is
sufficient. (b) ROC curves for SpikeNet and ProtoEEGNet. “FL” indicates that
the test data excluded samples with 3/8, 4/8 and 5/8 votes. The parentheses in
the legend indicate AUC values (confidence intervals reported in A.4).

Training ProtoEEGNet is trained end-to-end for 130 epochs and uses the Adam
optimizer. The model was trained to find 12 prototypes per class for a total
of 108 prototypes. We provide a more formal description of ProtoEEGNet and
more details on the training procedure in Appendix A.1.

##  3 Results

Since SpikeNet achieved state-of-the-art IED detection accuracy on an MGH
dataset of 1051 EEGs [5], we used it as our baseline. Since our data is a
superset of the SpikeNet dataset, we ensured that no samples used to train
SpikeNet were partitioned into our test set. To compare ProtoEEGNet against
SpikeNet, we binarized our model’s 9-class predictions by averaging the
positive class (4 votes or more) and negative class (3 votes or less)
prediction probabilities and taking a softmax over the two-class
probabilities. Then, given the binary prediction, we calculated the AUROC of
both models on the held-out test set. Originally in Jing et al. [5], the ROC
was calculated for SpikeNet by filtering out samples with 3/8, 4/8, or 5/8
votes. We follow this schema to create a filtered dataset and report results
for both filtered and unfiltered test data in Figure 2b.

We find that on both the filtered and the unfiltered data ProtoEEGNet
(filtered AUC = 0.928, unfiltered AUC = 0.861) slightly outperforms SpikeNet
(filtered AUC = 0.914, unfiltered AUC = 0.855). Further, ProtoEEGNet follows
interpretable reasoning as demonstrated in Figure 1. This is a major practical
advantage over SpikeNet, as practitioners may “fact check” model reasoning.

##  4 Conclusion

We present ProtoEEGNet, an interpretable method for detecting IEDs that
provides its reasoning on a case-by-case basis. One limitation of ProtoEEGNet
is ensuring each sample is well-represented by at least one prototype. We
leave this challenge of increasing prototype diversity for future work. In
this work, we demonstrated that the ProtoEEGNet outperforms SpikeNet (current
state-of-the-art) at IED detection while allowing users to visually verify its
predictions.

##  5 Potential Negative Societal Impacts

Similar to any other classification model, ProtoEEGNet is not guaranteed to
have perfect accuracy on new samples. Therefore, over-reliance on the model
may result in confusion or even incorrect diagnosis in certain instances.
Additionally, the model may be prone to adversarial privacy attacks which may
allow attackers to recreate training samples or generate inferences about the
patient training data. An exploration of the model’s resistance to privacy
attacks is beyond the scope of this work.

##  6 Acknowledgement

We acknowledge funding from the National Science Foundation under grant
HRD-2222336.

## References

  * Seidel et al. [2016] Stefan Seidel, Eleonore Pablik, Susanne Aull-Watschinger, Birgit Seidl, and Ekaterina Pataraia.  Incidental epileptiform discharges in patients of a tertiary centre.  _Clinical Neurophysiology_ , 127(1):102–107, 2016\.  ISSN 1388-2457.  doi: https://doi.org/10.1016/j.clinph.2015.02.056.  URL https://www.sciencedirect.com/science/article/pii/S1388245715001613.
  * Fountain and Freeman [2006] Nathan B. Fountain and John M. Freeman.  Eeg is an essential clinical tool: Pro and con.  _Epilepsia_ , 47(s1):23–25, 2006.  doi: https://doi.org/10.1111/j.1528-1167.2006.00655.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1528-1167.2006.00655.x.
  * van Donselaar et al. [1992] Cees A. van Donselaar, Robert-Jan Schimsheimer, Ada T. Geerts, and August C. Declerck.  Value of the Electroencephalogram in Adult Patients With Untreated Idiopathic First Seizures.  _Archives of Neurology_ , 49(3):231–237, 03 1992\.  ISSN 0003-9942.  doi: 10.1001/archneur.1992.00530270045017.  URL https://doi.org/10.1001/archneur.1992.00530270045017.
  * Jing et al. [2022] Jin Jing, Wendong Ge, Aaron F. Struck, Marta Fernandes, linda Qiao, Sungtae An, Safoora Fatima, Aline Herlopian, Ioannis Karakis, Jonathan J. Halford, Marcus C. Ng, Emily L. Johnson, Brian L. Appavu, Rani A. Sarkis, Gamaleldin M. Osman, Peter W. Kaplan, Monica B Dhakar, Lakshman Arcot Jayagopal, Zubeda B. Sheikh, Olga Taraschenko, Sarah E. Schmitt, Hiba A. Haider, Jennifer A. Kim, Christa B. Swisher, Nicolas Gaspard, Mackenzie Carpenter Cervenka, Andres A Rodriguez Ruiz, Jong Woo Lee, Mohammad Tabaeizadeh, Emily J. Gilmore, Kristen J. Nordstrom, Ji Yeoun Yoo, Manisha G Holmes, Susan T. Herman, Jennifer A. Williams, Jay S. Pathmanathan, Fábio A. Nascimento, Ziwei Fan, Samaneh Nasiri, Mouhsin M. Shafi, Sydney S. Cash, Daniel B. Hoch, Andrew J. Cole, Eric S. Rosenthal, Sahar F. Zafar, Jimeng Sun, and Michael Brandon Westover.  Interrater reliability of expert electroencephalographers identifying seizures and rhythmic and periodic patterns in eegs.  _Neurology_ , 100:e1737 – e1749, 2022.  URL https://api.semanticscholar.org/CorpusID:254180050.
  * Jing et al. [2020] Jin Jing, Haoqi Sun, Jennifer A. Kim, Aline Herlopian, Ioannis Karakis, Marcus C. Ng, Jonathan J. Halford, Douglas Maus, Fonda Chan, Marjan Dolatshahi, Carlos F. Muniz, Catherine J. Chu, Valeria Sacca, Jay S. Pathmanathan, Wendong Ge, Justin Dauwels, Alice D. Lam, Andrew J. Cole, Sydney S. Cash, and Michael Brandon Westover.  Development of expert-level automated detection of epileptiform discharges during electroencephalogram interpretation.  _JAMA neurology_ , 2020.  URL https://api.semanticscholar.org/CorpusID:204812691.
  * Johansen et al. [2016] Alexander Rosenberg Johansen, Jing Jin, Tomasz Maszczyk, Justin Dauwels, Sydney S. Cash, and M. Brandon Westover.  Epileptiform spike detection via convolutional neural networks.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 754–758, 2016.  doi: 10.1109/ICASSP.2016.7471776.
  * Lourenço et al. [2020] Catarina Lourenço, Marleen C. Tjepkema-Cloostermans, Luís F. Teixeira, and Michel J. A. M. van Putten.  Deep learning for interictal epileptiform discharge detection from scalp eeg recordings.  In Jorge Henriques, Nuno Neves, and Paulo de Carvalho, editors, _XV Mediterranean Conference on Medical and Biological Engineering and Computing – MEDICON 2019_ , pages 1984–1997, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-31635-8.
  * Raghu et al. [2020] Shivarudhrappa Raghu, Natarajan Sriraam, Yasin Temel, Shyam Vasudeva Rao, and Pieter Leonard Kubben.  Eeg based multi-class seizure type classification using convolutional neural network and transfer learning.  _Neural networks : the official journal of the International Neural Network Society_ , 124:202–212, 2020.  URL https://api.semanticscholar.org/CorpusID:211035040.
  * Tveit et al. [2023] Jesper Tveit, Harald Aurlien, S. Plis, Vince D. Calhoun, William O. Tatum, Donald L. Schomer, Vibeke Arntsen, Fieke M.E. Cox, Firas Fahoum, William B. Gallentine, Elena Gardella, Cecil D. Hahn, Aatif M. Husain, Sudha Kilaru Kessler, Mustafa Aykut Kural, Fábio A. Nascimento, Hatice Tankisi, Line Bédos Ulvin, Richard A. Wennberg, and Sándor Beniczky.  Automated interpretation of clinical electroencephalograms using artificial intelligence.  _JAMA Neurology_ , 80:805 – 812, 2023.  URL https://api.semanticscholar.org/CorpusID:259202197.
  * Beede et al. [2020] Emma Beede, Elizabeth Elliott Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis.  A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy.  _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020.  URL https://api.semanticscholar.org/CorpusID:213644599.
  * Zech et al. [2018] John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony Beardsworth Costa, Joseph J. Titano, and Eric Karl Oermann.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  _PLoS Medicine_ , 15, 2018.  URL https://api.semanticscholar.org/CorpusID:49558635.
  * Lopes et al. [2023] Marilia Karla Soares Lopes, Raymundo Cassani, and Tiago H. Falk.  Using cnn saliency maps and eeg modulation spectra for improved and more interpretable machine learning-based alzheimer’s disease diagnosis.  _Computational Intelligence and Neuroscience_ , 2023, 2023.  URL https://api.semanticscholar.org/CorpusID:256753238.
  * Rudin [2018] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1:206–215, 2018.  URL https://api.semanticscholar.org/CorpusID:182656421.
  * Adebayo et al. [2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  _Advances in neural information processing systems_ , 31, 2018.
  * Arun et al. [2021] Nishanth Thumbavanam Arun, N. Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Viktoria Hoebel, Sharut Gupta, Jay B. Patel, Mishka Gidwani, Julius Adebayo, Matthew D Li, and Jayashree Kalpathy-Cramer.  Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging.  _Radiology. Artificial intelligence_ , 3 6:e200267, 2021\.  URL https://api.semanticscholar.org/CorpusID:261967989.
  * Rudin et al. [2021] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  _ArXiv_ , abs/2103.11251, 2021.  URL https://api.semanticscholar.org/CorpusID:232307437.
  * Nascimento et al. [2022] Fábio Augusto Nascimento, Jaden D. Barfuss, Alex Jaffe, Michael Brandon Westover, and Jin Jing.  A quantitative approach to evaluating interictal epileptiform discharges based on interpretable quantitative criteria.  _Clinical Neurophysiology_ , 146:10–17, 2022.  URL https://api.semanticscholar.org/CorpusID:253571863.
  * Chen et al. [2018] Chaofan Chen, Oscar Li, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin.  This looks like that: deep learning for interpretable image recognition.  In _Neural Information Processing Systems_ , 2018.  URL https://api.semanticscholar.org/CorpusID:49482223.
  * Donnelly et al. [2021] Jon Donnelly, Alina Jade Barnett, and Chaofan Chen.  Deformable protopnet: An interpretable image classifier using deformable prototypes.  _CoRR_ , abs/2111.15000, 2021.  URL https://arxiv.org/abs/2111.15000.
  * Kane et al. [2017] Nick M. Kane, Jayant N. Acharya, Sándor Beniczky, Luis Otavio S. F. Caboclo, Simon Finnigan, Peter W. Kaplan, Hiroshi Shibasaki, Ronit M Pressler, and Michel J.A.M. van Putten.  A revised glossary of terms most commonly used by clinical electroencephalographers and updated proposal for the report format of the eeg findings. revision 2017.  _Clinical Neurophysiology Practice_ , 2:170 – 185, 2017\.  URL https://api.semanticscholar.org/CorpusID:52271646.

##  Appendix A Appendix

###  A.1 ProtoEEGNet Definition

####  A.1.1 Backbone

An input image is first fed into a pre-trained CNN, SpikeNet. This CNN has its
final predictive layer removed, so it simply acts as a function $f$ that maps
from images to a latent space, creating a tensor of useful features from an
input image. In the case of ProtoEEGNet, a 1x128x37 input EEG is convolved
into a single 128x1x1 tensor. The original image consists of 128 data points
across 37 different channels, creating a sort of 128x37 “image.” The end
result is 128 different and informative features extracted by the CNN for the
entire EEG.

####  A.1.2 Prototype Layer

Each class is assigned a set number of prototypes, and each prototype is
randomly initialized on the d-dimensional unit hypersphere to start. Every
prototype is of dimension 128x1x1 and represents the entire image, so we
calculate a similarity score per prototype for every image. Prototypes are
expressed as $p^{(c,l)}$, where c is the class the prototype belongs to and l
is what number it is within that class.

####  A.1.3 Similarity Score

The similarity score g for a given prototype and some input vector in the
feature space is calculated using cosine similarity between the two vectors,
where both vectors are always ensured to be on the unit hypersphere.

####  A.1.4 Fully-Connected Layer

After the similarity score is calculated for a given prototype, it is sent to
the final layer, which is fully connected and has no activation function. This
final layer does have normalized outputs using the softmax function:
$\frac{e^{q_{k}}}{\sum_{k\in K}e^{q_{k}}}$ and thus acts as a multi-class
logistic regression. This final output layer is described as the function $h$
and is used to create final “probabilities” for each class so that a
prediction can be made based on whichever logit is highest.

###  A.2 Prototype Training

Training was completed in 4 hours on a P100 GPU.

####  A.2.1 Stochastic Gradient Descent

There is a preliminary “warming” stage that lasts for 5 epochs, using
Stochastic Gradient Descent (SGD). In this stage, only the prototypes are
trained, and the weights in both the CNN backbone and the final output layer
are frozen. This essentially just allows us to get a head-start on training
the prototypes. The CNN is pre-trained, so most of its weights are fixed for
now, except the final output weights, which are chosen in a specific way to
enable us to train the prototypes effectively; for an output logit of class
$k$, any prototype also of class $k$ will have a weight = 1. For any prototype
not of class $k$, the assigned weight will be -0.5. This is a simple and
consistent configuration that makes training the prototypes while freezing the
final layer plausible, as it rewards activation within the correct class and
penalizes activation outside the correct class.

After 5 epochs, the CNN weights are unfrozen and are allowed to move with the
prototypes. This is done to allow the latent space of the CNN to change and
learn as needed to fit the prototypes optimally. This first step may be
repeated multiple times, i.e., multiple epochs may be allowed to pass of just
doing SGD. The number of epochs is a hyperparameter that can be set.

The total loss function used during SGD, including all regularization terms,
can be expressed as follows:

$l=\textrm{CE}+\lambda_{1}l_{\textrm{sep}}+\lambda_{2}l_{\textrm{clst}}+\lambda_{3}l_{\textrm{ortho}}$

Where CE is simply the usual cross-entropy. The loss terms are explained in
their own section further below.

####  A.2.2 Prototype Projection

In this stage, prototypes are “pushed” or projected onto the nearest latent
patch of the same class discovered in training as judged by the cosine
similarity; i.e., a prototype is pushed onto whatever example is most similar
to it. This is useful for two reasons: Firstly, it enables us to explicitly
connect the prototypes to an actual example found in the training data.
Secondly, it ensures that the prototypes are not in some area of the latent
space completely disconnected from the training data. By forcing it to
correspond to a training example, a prototype is able to stay in the correct
“neighborhood.” This step is only applied a few times during training. For
example, in the Original ProtoPNet paper [18], this projection step was only
performed at epochs 10, 20, and 30, meaning that between those times, the
prototypes were allowed to move freely in space without being explicitly tied
to any particular training patch. The epochs at which this step is performed
are also hyperparameters.

####  A.2.3 Convex Optimization

In this stage, we freeze the prototypes and the CNN weights and only optimize
the final fully connected layer. Because there is no activation function and
all other weights are frozen, we are able to use convex optimization rather
than gradient descent to optimize this final layer. This allows us to optimize
the preset weights used in the previous examples. We also try to force the
majority of weights where the class of the prototype does not match the output
logit to be 0 (it was previously fixed at -0.5), making the weights sparse.
This is because we want the model to have an affirmative reasoning process as
opposed to a negative one, where the model chooses a class due to the presence
of distinguishing features, rather than a process of elimination where the
model picks a class because it is not some other class. Note that this step is
always performed directly after the prototype projection stage.

The problem solved is described below:

$\min_{w_{h}}\frac{1}{n}\sum_{i=1}^{n}CrsEnt(f\circ g_{p}\circ
h(x_{i}),y_{i})+\lambda\sum_{k=1}^{K}\sum_{j:p_{j}\notin
P_{k}}|w_{h}^{(k,j)}|$

where $h$ is the CNN backbone, $g_{p}$ represents the prototype layers, $f$
represents the final connected layer, and $w_{h}^{(k,j)}$ is the weight
associated in the final connected layer where $k$ is the class in question and
$j$ is the class of the prototype being examined. Note that there is an
$L_{1}$ regularization term also being computed here to keep the weights
sparse.

####  A.2.4 ProtoEEGNet Loss Terms

On top of the loss function defined above, we have additional regularization
terms for clustering, separation, and orthogonality, which are
$l_{\textrm{clst}}$, $l_{\textrm{sep}}$, and $l_{\textrm{ortho}}$
respectively.

The clustering loss is expressed as
$l_{\textrm{clst}}=-\frac{1}{N}\sum_{i=1}^{N}\max_{\hat{p}^{(c,l)}:c=y^{(i)}}g(\hat{z}^{(i)})^{(c,l)}$.
This loss serves to ensure that the latent representation appears similar to
at least one prototype, ensuring that prototypes are able to "cluster" around
examples that look similar to them.

The separation term is expressed as
$l_{\textrm{sep}}=\frac{1}{N}\sum_{i=1}^{N}\max_{\hat{p}^{(c,l)}:c\neq
y^{(i)}}g_{p}(\hat{z}^{(i)})^{(c,l)}$. This loss serves to reinforce that the
similarity between an image and a prototype not of the same class as that
image is as small as possible, ensuring that prototypes of different classes
are kept separate from one other and thus each class gets its own area of the
latent space.

Finally, the orthogonality loss is expressed as
$l_{\textrm{ortho}}=\sum_{c}\|P^{(c)}{P^{(c)}}^{T}-I^{(L)}\|_{F}^{2}$, where c
is a class and $L$ is the number of prototypes within that class, $I$ is the
identity matrix in $\mathbb{R}^{L\times L}$, and we use the Frobenius norm.
This ensures that prototypes of the same class are as orthogonal to each other
as possible, allowing them to own their own portion of the latent space and
thus preventing prototypes from being too similar and not representing the
full complexity of the class.

###  A.3 Data

####  A.3.1 Data Source

Our dataset includes samples from SpikeNet’s MGH dataset [5] in addition to
more samples collected from MGH EEG archives.

####  A.3.2 Voter Disagreement

Figure 3: Each line indicates the voting tendencies of a different voter.
Given all other experts’ votes (x-axis) for an EEG, each point represents the
probability an individual expert will vote that the EEG contains an IED
(y-axis).

###  A.4 AUROC Confidence Intervals

The 95% confidence interval on unfiltered data for SpikeNet was (0.840, 0.868)
and for ProtoEEGNet – (0.847, 0.875); on filtered data, for SpikeNet –
(0.902,0.928) and for ProtoEEGNet – (0.916, 0.940). Confidence intervals were
computed via 10,000 rounds of bootstrapping.

###  A.5 EEG Class Examples

We visualized the different classes of votes below in Figure 4.

Figure 4: 1-second, EEGs arranged in a standard banana bipolar montage.
Channels are labeled on the left. Each sample is displayed with the proportion
of experts out of 8 that believe the sample contains an IED spike.

###  A.6 Experiment Hyperparameters

To promote the reproducibility of our technique, we release the
hyperparameters used.

Hyperparameter |  Value |  Description
---|---|---
warm_optimizer_lrs["prototype_vectors"] |  0.003 |  Learning rate for the prototype vectors during the warming stage.
prototype_shape |  [128, 1, 1] |  Shape of prototypes tensor: total number x number of features x width x height.
num_warm_epochs |  10 |  Number of warming epochs to be done before joint training.
joint_optimizer_lrs["joint_last_layer_lr"] |  1.0e-05 |  Learning rate for the last layer during joint training.
joint_optimizer_lrs["prototype_vectors"] |  0.05 |  Learning rate for the prototype vectors during joint training.
joint_optimizer_lrs["conv_offset"] |  0.003 |  Learning rate for the convolutional offset layers during joint training.
joint_optimizer_lrs["features"] |  0.001 |  Learning rate for the features during joint training.
joint_optimizer_lrs["add_on_layers"] |  0.001 |  Learning rate for the add-on layers during joining training.
prototype_activation_function |  log |  The activation function associated with prototype layers.
push_start |  70 |  The first epoch at which you push the prototype vectors. Should be > num_warm_epochs.
warm_pre_offset_optimizer_lrs["prototype_vectors"] |  0.003 |  The learning rate for the prototype vectors during the second warming stage.
warm_pre_offset_optimizer_lrs["add_on_layers"] |  0.003 |  The learning rate for the add-on layers during the second warming stage.
warm_pre_offset_optimizer_lrs["features"] |  0.001 |  The learning rate for the features during the second warming stage.
push_epochs |  [110, 120, 130] |  The list of epochs in which you should push prototypes.
joint_lr_step_size |  30 |  The step size for the joint layer optimizer.
num_classes: |  9 |  The number of classes you are classifying between.
num_train_epochs |  130 |  The number of total train epochs. Note that you should always push on the last epoch, and it is 0-indexed.
train_push_batch_size |  75 |  The batch size for the train push step.
num_secondary_warm_epochs |  10 |  The number of epochs dedicated to the second warm stage.
coefs["clst"] |  -0.1 |  The coefficient for the clustering loss.
coefs["offset_weight_l2:"] |  0.1 |  The coefficient for the offset distance loss.
coefs["sep"] |  0 |  The coefficient for the separation loss.
coefs["orthogonality_loss"] |  0.5 |  The coefficient for the orthogonality loss.
coefs["offset_bias_l2"] |  1 |  The coefficient for the offset distance loss.
coefs["l1"] |  0.01 |  The coefficient for L1 regularization loss on the last layer.
coefs["crs_ent"] |  1.25 |  The coefficient for the cross-entropy loss.


11institutetext: The University of Technology Sydney, 15 Broadway, Ultimo,
Australia 22institutetext: Data Science Institute, 15 Broadway, Ultimo,
Australia
22email: {david.hasonrudd@student, huan.huo<EMAIL_ADDRESS>
$*$corresponding author

# Leveraged Mel spectrograms using Harmonic and Percussive Components in
Speech Emotion Recognition

David Hason Rudd 11** Huan Huo 11 Guandong Xu 1122

###### Abstract

Speech Emotion Recognition (SER) affective technology enables the intelligent
embedded devices to interact with sensitivity. Similarly, call centre
employees recognise customers’ emotions from their pitch, energy, and tone of
voice so as to modify their speech for a high-quality interaction with
customers. This work explores, for the first time, the effects of the harmonic
and percussive components of Mel spectrograms in SER. We attempt to leverage
the Mel spectrogram by decomposing distinguishable acoustic features for
exploitation in our proposed architecture, which includes a novel feature map
generator algorithm, a CNN-based network feature extractor and a multi-layer
perceptron (MLP) classifier. This study specifically focuses on effective data
augmentation techniques for building an enriched hybrid-based feature map.
This process results in a function that outputs a 2D image so that it can be
used as input data for a pre-trained CNN-VGG16 feature extractor. Furthermore,
we also investigate other acoustic features such as MFCCs, chromagram,
spectral contrast, and the tonnetz to assess our proposed framework. A test
accuracy of 92.79% on the Berlin EMO-DB database is achieved. Our result is
higher than previous works using CNN-VGG16.

###### Keywords:

Speech Emotion Recognition (SER) Mel spectrogram Convolutional Neural Network
(CNN) voice signal processing acoustic features

##  1 Introduction

The general motivation of SER systems is to recognize specific features of a
speaker’s voice in different emotional situations to provide a more personal
and often superior user experience [6]. For example, a Customer Relationship
Management (CRM) team can use SER to determine a customer’s satisfaction by
their voice during a call. Emotions are universal, although their
understandings, interpretations and reflections are particular and partially
associated with culture [1]. Unlike speech recognition, there is no standard
or integrated approach for recognising emotions and analysing them through
human voices [30].

The fundamental challenge of SER is the extraction of discriminative and
robust features from speech signals. Features used for SER are generally
categorized as prosodic, acoustic, and linguistic features. The prosodic
features include pitch, energy, and zero-crossings of the speech signal [16,
19, 28]. The acoustic features describe speech wave properties including
linear predictor coefficients (LPC), mel-scaled power spectrograms (Mel),
linear predictor cepstral coefficients (LPCC), power spectral analysis (FFT),
power spectrogram chroma (Chroma), and mel-frequency cepstral coefficients
(MFCC) [5]. In SER, the Mel spectrogram, MFCC, and chromagram are the most
effective in decoding emotion from a signal [22].

Among the most common speech feature extraction techniques, this paper
addresses a principal question in Emotion Recognition (ER): How can we
maximise the advantage of the Mel spectrogram feature to improve SER? This
study presents a novel implementation of emotion detection from speech signals
by processing harmonic and percussive components of Mel spectrograms and
combining the result with the log Mel spectrogram feature. Our primary
contribution is the introduction of an effective hybrid acoustic feature map
technique that improves SER. First, we employ CNN-VGG16 as a feature extractor
of emotion identifier, then utilise the MLP networks for classification task.
Furthermore, we tune the MLP network parameters using the random search model
hyperparameter technique to obtain the best model. Based on empirical
experiments, we assert that a data augmentation strategy using an efficient
prosodic and acoustic feature combination analysis is the key to obtaining
state-of-the-art results since input data represents more diversity with
enriched features; these characteristics lead to better model generalisation.

##  2 Related works

Early traditional SER models relied on modification and optimisation of
Support Vector Machine (SVM) classifiers to predict emotions such as anger,
happiness, and sadness, among others [25, 23, 15]. Wu et al. [31] implemented
a traditional machine learning method based on EMO-DB [3] database. The
authors proposed novel sound features named Modulation Spectral Features
(MSFs) that combined prosodic features, and they ultimately obtained 85.8%
validation accuracy for speaker-independent classification using a multi-class
Linear Discriminant Analysis (LDA) classifier. Similarly, Milton et al. [21]
proposed another classical machine learning method for SER by using a
combination of three SVMs to classify emotions in the Berlin EMO-DB.
Furthermore, Huang et al. [13] introduced a hybrid model called a semi-CNN,
which used a deep CNN to learn feature maps and a classic machine learning SVM
to classify seven emotions from EMO-DB. The authors utilised spectrograms as
the input for their proposed model and achieved 88.3% and 85.2% test accuracy
for speaker-dependent and speaker-independent classification, respectively.

The idea of exploiting pre-trained CNN image classifiers [7] for other tasks
involves leveraging transfer learning methods in SER. Surprisingly, using
speech-based spectrograms as the input images for pre-trained image
classifiers produced competitive results when compared with other well-known
traditional methods. Badshah et al. [2] extracted spectrogram speech features,
which were then visualised in 2D images and passed to a CNN; this approach
achieved a 52% test accuracy on EMO-DB. Demircan and Kahramanli [8] developed
several different classifiers and obtained test accuracies 92.86%, 92.86%, and
90%, respectively on SVM, KNN and ANN. Additionally, Wang et al. [29] worked
on MFCCs feature and proposed an acoustic feature called the Fourier Parameter
(FP) , which obtained 73.3% average accuracy with an SVM classifier.
Furthermore, many similar studies were conducted on different databases.
Popova et al. [24] used a fine-tuned DNN and CNN-VGG16 classifier to extract
the Mel spectrogram features in the RAVDESS dataset [17] and obtained an
accuracy of 71% [24]. Satt et al. [27] presented another multi-modal LSTM-CNN
and proposed a novel feature extraction method based on the paralingual data
from spectrograms. The authors obtained 68% accuracy on the IMOCAP [4]
database.

In recent years, some works proposed the use of hybrid feature map techniques
as input data for CNN-based networks. Meng et al. [20] proposed a feature
extraction strategy for Log-Mel spectrograms that extracted a 3D voice feature
representation map by combining log Mel spectrograms with the first and second
derivatives of the log MelSpec of the raw speech signal. The authors proposed
a CNN with a multimodal dilated architecture that used a residual block and
BiLSTM (ADRNN) to improve the classifier accuracy. In addition, the ADRNN
further enhanced the extraction of speech features using the proposed
attention mechanism approach. The model achieved a remarkable performance of
74.96% and the 90.78% accuracy of the IEMOCAP and EMO-DB databases. On the
other hand, Hajarolasvadi et al. introduced a 3D feature frame technique for
use as input data to the network by extracting an 88-dimensional vector of
voice features including MFCCs, intensity, and pitch. The model can reduce
speech signal feature frames by applying k-means clustering on the extracted
features and selecting the k most discriminant frames as keyframes. Then, the
feature data placed in the keyframe sequence were encapsulated in a 3D tensor,
which produced a final extracted feature map for use as input data for a
3D-CNN-based classifier that used the 10-fold cross-validation method. The
authors achieved a weighted accuracy of 72.21% on EMO-DB. Zhao et al. [33]
proposed a multi-modal 2D CNN-LSTM network and extracted the log of the Mel-
spectrograms from the speech signals for use as input data. The outcome of
their work is state-of-the-art with the accuracy of 95.89% for speaker-
independent classification on the Berlin EMO-DB.

##  3 Methodology

This section explains the work procedures used to build the hybrid feature map
representation in our model. We compute the average of the signal’s harmonic
and percussive components and combine the result with the log Mel spectrogram
feature. The proposed hybrid feature map method can be generalised with other
supervised classifiers to obtain better prediction accuracy.

###  3.1 Proposed hybrid features: Harmonic and Percussive components of Mel
spectrogram

Essential features in speech signal processing are the spectrograms on the Mel
scale, chromograms [12], spectral contrast, the tonnetz [12] and MFCCs [32].
Since the average length of the recorded voice samples are four seconds, we
digitise each original utterance signal at an 88KHz sample rate using the
Hanning window function [11] shown in (1) to provide sufficient frequency
resolution and spectral leakage protection. Next, we apply Mel filter banks to
the spectrogram by shifting 0.4ms in a window time of 23ms so that the output
is a group of FFTs located next to one another. The Hanning window is
described in (1),

$H_{m}(n)=0.5[1-cos(\frac{2\pi.n}{M-1})]=sin^{2}(\frac{\pi.n}{M-1})\ \ \\\ \ \
\ \ \ 0\leq\ n\ <\ M-1$ (1)

where M represents the number of points in the output window, which is set to
128 and n denote the number of specific sample point from the signal. Finally,
we construct the Mel spectrogram by multiplying the obtained energy matrix of
the Mel scaled static with the STFT results formulated in (2),

$LMS(m)=\displaystyle\sum_{k=f(m-1)}^{f(m+1)}log(H_{m}(k)\ \ .\ \ |X(k)|^{2})$
(2)

where $|X(k)|^{2}$ represents the energy spectrum in the $kth$ energy block,
$H(k)$ is a Mel-spaced filter bank function, $m$ represents the number of
filter banks, and $k$ points to the number of FFT coefficients. $LMS$
represents the log Mel spectrogram. To perform Mel spectrogram feature
extraction, we use Librosa tools [18] to set the size of Mel filterbanks as
128, the window size as 2048 and hop length as 512. Fig. 1 shows the Mel
spectrogram of sample voices exhibiting five emotions from the EMO-DB dataset.
It is clear that the amplitude and frequency of each emotion image have a high
distinction from other samples.

Figure 1: The above sample Mel spectrograms clearly illustrate the distinction
between amplitude and frequency in each emotion. The red colours represent
frequencies that contribute more than orange and white colours.

The first feature map is built by applying a decomposition process to the Mel
spectrum using the popular method in [9]. The decomposition method can be
formulated such that the harmonic $s_{h}$ and percussive $s_{p}$ components
are separated from the input signal $s$ by applying a STFT on the frames to
obtain spectrogram $S$ of signal $s$ as shown in definitions (3) and (4),

$s\ \ =\ \ S_{h}\ \ +\ \ S_{p}$ (3)

$S(n,\ k)\ \ :=\ \ \displaystyle\sum_{r=0}^{N-1}s(r+nH)\ \ .\ \ \omega(r)\ \
.\ \ e^{(\frac{-j2\pi.k.n}{N}})$ (4)

where $S$ denotes a spectrum of signal $s$ in $k^{th}$ Fourier coefficient on
the $m^{th}$ time frame, $\omega:[0:N-1]$ := $\\{0,1,...,N-1\\}$ is a sine
windowing function that represents the window length $N$, $H$ represents the
hop size value, $n$ indicates current frame number and $N$ is the length of
the discrete Fourier transform. We can obtain the harmonic and percussive
components of the spectrum by applying a median filter in the horizontal
(time-domain) and vertical (frequency-domain) direction on spectrum $S$.
Finally, we extract the first feature map by obtaining the mean of both
components as shown in the following summarised formulas in (5), (6) and (7),

$\widehat{H}=\text{\large$\widehat{S}$}\ \ \bigotimes\ \ \text{\large$M$}_{H}$
(5)

and

$\widehat{P}=\text{\large$\widehat{S}$}\ \ \bigotimes\ \ \text{\large$M$}_{P}$
(6)

obtained by

$\large\mathcal{F}_{2(\text{LMS})}=\frac{\widehat{H}+\widehat{P}}{2}$ (7)

where $\bigotimes$ denotes the multiplication element of the median filter in
$\text{\large$M$}_{H}$, which is the horizontal direction filtering used to
obtain the $\widehat{H}$ harmonic components of the original spectrogram
$\widehat{S}$. Subsequently, $\text{\large$M$}_{P}$ represents the vertical
median filtering results $\text{\large$M$}_{P}$, which is the percussive
component of the original spectrogram, $\widehat{S}$ shown in (4). Fig. 2
shows the harmonic and percussive components as two distinctive spectrograms
in the 128 Mel filterbank.

Figure 2: The harmonic and percussive components of the Mel spectrograms for a
sample neutral emotion

The second feature map is extracted by applying the log of the Mel spectrogram
obtained in (2) to measure the sensitivity of the Mel spectrogram output value
fluctuation concerning changes in the voice signal amplitude. A sample 2D
hybrid feature representation in our work is visualised in Fig. 3, which
clearly shows that each sample feature map is combined in a two-dimensional
image. This specific feature combination improves the prediction accuracy in a
simple full contact neural network classifier based on our empirical
experiments.

Figure 3: Visualising an achieved 2D hybrid feature maps from the Berlin EMO-
DB

###  3.2 Model architecture and training

We use the CNN-VGG16 [26] as a feature extractor to learn from high
dimensional feature maps since the network can learn from small variations
that occur in the extracted features maps. However, the high-capacity memory
storage requirements for a simple classification task can be considered a
partial limitation of VGG16 applications.

The details of the proposed architecture are shown in Fig. 4; the architecture
consists of an VGG16 and MLP network, which serve as an feature extractor and
emotion classifier, respectively. First, the subsamples are extracted from a
fixed window size and then feature maps are built using the proposed feature
map function. Therefore, the input to the VGG16 feature extractor is a 2-D
feature map in the dimension of (128 x 128 x 2). The input to the MLP
classifier is a 2048 one-dimensional vector generated by VGG16. The MLP
classifier includes four fully connected layers with the ReLU activation
function and softmax in the output layer. Dense 1 and 2 have a 1024 input with
a 0.5 dropout value, and dense 3 and 4 are set to 512 input with 0.3 dropouts.
The ADAM optimiser with a learning rate of 0.0001 is selected for our
architecture design.

Figure 4: Model architecture, which includes a 2D hybrid feature map built
using the harmonic and percussive components, as well as the log of the Mel
spectrogram in the feature map generator function. The features are extracted
using a CNN-VGG16 network. Finally, the MLP network classifies seven emotions.

##  4 Experimental analysis

This section analyses the experimental configuration and the result of the
feature extractor and MLP classifier on EMO-DB [3]. The sample voices are
randomly partitioned and 80% are used for the training set and 10% for the
validation and test set for the speaker-independent classification task. We
apply an oversampling strategy to compensate the minority classes and increase
the voice samples before feeding them to the feature extractor network during
the pre-processing phase. The classifier is trained on 128 epochs with a batch
size of 128 and used an Nvidia GPU. The window size is set to 2048 with (128 x
128) bands and frames to obtain each subsample length = 2.9 sec. Then, the
subsamples are created in each defined data frame. Finally, 167426 signal
subsamples and 9717 feature maps are obtained from a sample rate of 88KHz.
Based on the time-frequency trade-off, large frame size is chosen to obtain
high-frequency resolution rather than time resolution since analysing the
frequency of speech signal enables us to decode emotion. Several time-
consuming experiments are conducted to assess the effectiveness of the
proposed hybrid feature, which aims to find the best data augmentation through
feature combination.

###  4.1 Results Analysis

To assess our enriched feature representation method in the MLP classifier,
the result of evaluation metrics such as the confusion matrix and test
accuracy are observed on different sample rates and feature map dimensions
(bands and frames). We also evaluate our model output based on the setting of
various parameters in the feature map function. For example, the prediction
accuracy results based on some different parameters setting are shown in Table
1. These results indicate that the superior result is achieved on feature map
dimensions of 128 x 128 with a sample rate of 88200, Since the highest
subsample length of 2.9seconds is achieved and more sample points can
contribute in each subsample.

Table 1: The emotion classifier accuracy based on different feature map representation dimensions and signal sampling ratios Band | Frame | Sample rate | Accuracy
---|---|---|---
32 | 32 | 88200 | 71.92%
128 | 128 | 44100 | 87.04%
128 | 128 | 22050 | 88.54%
32 | 32 | 22050 | 89.54%
64 | 64 | 44100 | 92.02%
128 | 128 | 88200 | 92.79%

We examine the effect of the different number of subsamples from the signal by
increasing the window size and sample rate on ten different feature map
representations, including 1D, 2D, and 3D maps, and we then compare their
results with our hybrid feature extraction method. With respect to the primary
research question, it is found that we can take maximum advantage of the
powerful Mel spectrogram feature through harmonic and percussive components in
emotion recognition.

As shown in Table 2, the proposed hybrid feature map representation achieves
better results than other well-known feature combinations techniques.
Furthermore, the results in Table 2. indicate that the accuracy increases in
the high range of the sample rate and window size in most represented methods
since the feature map generator function handles more data points via a higher
overlapping between frames. Consequently, for most feature extraction methods,
the VGG16 network can learn from better-enriched features when the sample rate
is higher. In contrast, an increased number of data points in the subsamples
requires a memory capacity in the gigabyte range to store the base, train,
validation, test feature map files in the pkl format. For instance, in our
model, a signal sampling rate of 88KHz and a window size of 2048 occupy an
approximately 3-gigabyte memory space to store the pkl files for analysing the
whole voice files in the EMO-DB; this requirement can limit its application.

Table 2:  Evaluation of the prediction accuracy based on the different feature extraction methods, sample rate and window size Window Size | 512 | 1024 | 2048
---|---|---|---
Sample Rate | 22050 | 44100 | 88200
Feature extraction methods | Acc % | Acc % | Acc %
1D-MFCCs | 65.81 | 68.39 | 69.03
1D-Mel spectrogram | 75.48 | 75.48 | 82.71
1D-Chromagram | 80.01 | 80.13 | 81.29
1D-Tonnets | 56.77 | 63.08 | 56.81
1D-Spectral | 54.84 | 50.93 | 47.10
2D-MFCCs+Chromagram | 83.87 | 83.23 | 91.59
2D-Mel spectrogram+MFCCs | 88.39 | 85.16 | 85.81
2D-Mel spectrogram+Spectral | 82.01 | 85.13 | 80.65
3D-Mel spectrogram+MFCCs+Chromagram | 83.87 | 88.39 | 81.94
2D-log-MSS+Avg.HP(proposed) | 92.02 | 89.54 | 92.79

The fluctuation in the prediction accuracy per emotion class is illustrated
for various feature representation methods in Fig. 5. The boxplot graph shows
that the model output is more reliable and stable when predicting seven
emotions using our proposed hybrid feature extraction ”2D-log-MSS+Avg.HP” and
two more feature representations built by combining the delta of the Mel
spectrogram (MSS) and log Mel spectrogram or MFCCs features.

Figure 5:  Variation in the prediction accuracy per emotion class for
different feature representation methods

The model’s confusion matrix in Table 3. shows that the network performs
better when recognising specific emotions (anger, sadness, happiness, and
fear) while its performance is comparatively poor when predicting emotions
such as neutral and boredom. Many experiments are conducted and the highest
test accuracy of %92.79 is achieved. The Python Keras based network
implementation for the proposed model and more experimental results and
visualisations are available in our GitHub
repositories111https://github.com/DavidHason/ser.

Table 3: Confusion matrix (%) of the model with an average accuracy of 92.71% on the EMO-DB dataset Emotion: | Anger | Boredom | Disgust | Fear | Happiness | Neutral | Sadness
---|---|---|---|---|---|---|---
Anger |  94.92 |  0 |  0 |  0 |  5.12 |  0 |  0
Boredom |  0 |  78.77 |  0 |  0 |  0 |  9.9 |  11.54
Disgust |  0 |  0 |  89.47 |  0 |  9.8 |  0 |  0
Fear |  0 |  0 |  0 |  96 |  0 |  0 |  3.85
Happiness |  0 |  0 |  0 |  0 |  100 |  0 |  0
Neutral |  0 |  12.81 |  0 |  0 |  0 |  88.87 |  0
Sadness |  0 |  0 |  0 |  0 |  0 |  0 |  100

###  4.2 Model comparison with previous works on EMO-DB

As shown in Table 4, our method achieves superior results compared with most
previous studies except for two works in terms of accuracy that are not
significantly higher than our results. However, their work frame is more
sophisticated than our proposed model. Zhao et al. [33] combined two 1-D and
2-D LSTM CNN networks in the feature learning process. Demirican et al. [8]
used a model with three classifiers KNN, SVM and ANN to improve the prediction
accuracy. Nevertheless, the major advantage of our architecture comes from its
simplicity and generality, which can be employed for other acoustic features,
as shown in Table 2. Another advantage of the architecture is the capability
of storing the feature maps into cloud storage in pkl format that enables us
to share them for simultaneous analysis with other networks.

Table 4: Model comparison with previous works on EMO-DB Previous works | Learner | Feature extraction method | Accuracy
---|---|---|---
Badshah et al. [2] | CNN | log Mel spectrogram | 52%
Popova et al. [24] | VGG16 | Mel spectrograms | 71%
Hajarol. et al. [10] | CNN | Mel spectrograms+MFCCs | 72.21%
Wang et al. [29] | SVM | Fourier Parameter+MFCCs | 73.3%
Huang et al. [13] | CNN | Spectrogram | 85.2%
Issa et al. [14] | VGG16 | MFCCs+Chroma.+Mel spec.+Contrast+Tonnetz | 86.10%
Meng et al. [20] | CNN-LSTM | log Mel spec.+1st & 2nd delta(log Mel spec.) | 90.78%
Wu et al. [31] | SVM | Modulation Spectral Features (MSFs) | 91.60%
Our model | VGG16-MLP | Harmonic-Percussive (HP)+log Mel spec. | 92.79%
Demircan et al. [8] | SVM | LPC+MFCCs | 92.86%
Zhao et al. [33] | CNN-LSTM | log Mel spectrogram | 95.89%

##  5 Conclusion

The key research question in this study focuses on leveraging Mel spectrogram
components in a hybrid-based feature engineering technique as well as
proposing a novel acoustic feature extraction method to improve emotion
recognition. The proposed feature map generator function extracts the harmonic
and percussive components by applying a median filter on the horizontal (time-
domain) and vertical (frequency-domain) directions of the spectrum, and is
implemented with a four-layer MLP classifier to predict emotions in the human
voice. The performance of the proposed hybrid feature technique is tested on
the Berlin EMO-DB and compared with other 1D, 2D, and 3D feature extraction
methods. To the best of our knowledge, this is the first study on speech
emotion recognition that combines this specific component of the spectrogram.
The results show that our work significantly outperforms most previous works
due to its achievement of a 92.79% test accuracy which is also a superior
result in VGG16 feature learning methods. In future investigations, facial
expression analysis and linguistic features can be embedded into the framework
to improve the emotion recognition as an acoustic-only method is not constant
across different languages and cultures.

## Acknowledgement:

This work is partially supported by Australian Research Council under grant
number: DP22010371, LE220100078, DP200101374 and LP170100891

## References

  * [1] Alu, D., Zoltan, E., Stoica, I.C.: Voice based emotion recognition with convolutional neural networks for companion robots. Science and Technology 20, 222–240
  * [2] Badshah, A.M., Ahmad, J., Rahim, N., Baik, S.W.: Speech emotion recognition from spectrograms with deep convolutional neural network. In: 2017 international conference on platform technology and service (PlatCon). pp. 1–5 (2017)
  * [3] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., Weiss, B., et al.: A database of german emotional speech. In: Interspeech. vol. 5, pp. 1517–1520 (2005)
  * [4] Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation 42(4), 335–359 (2008)
  * [5] Chu, S., Narayanan, S., Kuo, C.C.J.: Environmental sound recognition with time–frequency audio features. IEEE Transactions on Audio, Speech, and Language Processing 17(6), 1142–1158 (2009)
  * [6] Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G.: Emotion recognition in human-computer interaction. IEEE Signal processing magazine 18(1), 32–80 (2001)
  * [7] Cummins, N., Amiriparian, S., Hagerer, G., Batliner, A., Steidl, S., Schuller, B.W.: An image-based deep spectrum feature representation for the recognition of emotional speech. In: Proceedings of the 25th ACM international conference on Multimedia. pp. 478–484 (2017)
  * [8] Demircan, S., Kahramanli, H.: Application of fuzzy c-means clustering algorithm to spectral features for emotion classification from speech. Neural Computing and Applications 29(8), 59–66 (2018)
  * [9] Fitzgerald, D.: Harmonic/percussive separation using median filtering. In: Proceedings of the International Conference on Digital Audio Effects (DAFx). vol. 13, pp. 1–4 (2010)
  * [10] Hajarolasvadi, N., Demirel, H.: 3d cnn-based speech emotion recognition using k-means clustering and spectrograms. Entropy 21(5), 479–495 (2019)
  * [11] Harris, F.J.: On the use of windows for harmonic analysis with the discrete fourier transform. Proceedings of the IEEE 66(1), 51–83 (1978)
  * [12] Harte, C., Sandler, M., Gasser, M.: Detecting harmonic change in musical audio. In: Proceedings of the 1st ACM workshop on Audio and music computing multimedia. pp. 21–26 (2006)
  * [13] Huang, Z., Dong, M., Mao, Q., Zhan, Y.: Speech emotion recognition using cnn. In: Proceedings of the 22nd ACM international conference media. pp. 801–804 (2014)
  * [14] Issa, D., Demirci, M.F., Yazici, A.: Speech emotion recognition with deep convolutional neural networks. Biomedical Signal Processing and Control 59, 101894–101904 (2020)
  * [15] Jin, Q., Li, C., Chen, S., Wu, H.: Speech emotion recognition with acoustic and lexical features. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 4749–4753 (2015)
  * [16] Li, M., Han, K.J., Narayanan, S.: Automatic speaker age and gender recognition using acoustic and prosodic level information fusion. Computer Speech & Language 27(1), 151–167 (2013)
  * [17] Livingstone, S.R., Russo, F.A.: The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13(5), 1–35 (2018)
  * [18] McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., Nieto, O.: librosa: Audio and music signal analysis in python. In: Proceedings of the 14th python in science conference. vol. 8, pp. 18–25 (2015)
  * [19] Meinedo, H., Trancoso, I.: Age and gender classification using fusion of acoustic and prosodic features. In: Eleventh annual conference of the international speech communication association. pp. 1–4 (2010)
  * [20] Meng, H., Yan, T., Yuan, F., Wei, H.: Speech emotion recognition from 3d log-mel spectrograms with deep learning network. IEEE access 7, 125868–125881 (2019)
  * [21] Milton, A., Roy, S.S., Selvi, S.T.: Svm scheme for speech emotion recognition using mfcc feature. International Journal of Computer Applications 69(9) (2013)
  * [22] Motlıcek, P.: Feature extraction in speech coding and recognition. Tech. rep., Technical Report of PhD research internship in ASP Group (pp 1-50,2002)
  * [23] Pérez-Rosas, V., Mihalcea, R., Morency, L.P.: Utterance-level multimodal sentiment analysis. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 973–982 (2013)
  * [24] Popova, A.S., Rassadin, A.G., Ponomarenko, A.A.: Emotion recognition in sound. In: International Conference on Neuroinformatics. pp. 117–124 (2017)
  * [25] Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Prasad, R.: Ensemble of svm trees for multimodal emotion recognition. In: Proceedings of the 2012 Asia Pacific signal and information processing association annual summit and conference. pp. 1–4 (2012)
  * [26] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211–252 (2015)
  * [27] Satt, A., Rozenberg, S., Hoory, R.: Efficient emotion recognition from speech using deep learning on spectrograms. In: Interspeech. pp. 1089–1093 (2017)
  * [28] Shriberg, E., Ferrer, L., Kajarekar, S., Venkataraman, A., Stolcke, A.: Modeling prosodic feature sequences for speaker recognition. Speech communication 46(3-4), 455–472 (2005)
  * [29] Wang, K., An, N., Li, B.N., Zhang, Y., Li, L.: Speech emotion recognition using fourier parameters. IEEE Transactions on affective computing 6(1), 69–75 (2015)
  * [30] Weninger, F., Wöllmer, M., Schuller, B.: Emotion recognition in naturalistic speech and language—a survey. In book: Emotion Recognition: A Pattern Analysis Approach pp. 237–267 (2015)
  * [31] Wu, S., Falk, T.H., Chan, W.Y.: Automatic speech emotion recognition using modulation spectral features. Speech communication 53(5), 768–785 (2011)
  * [32] Xu, M., Duan, L.Y., Cai, J., Chia, L.T., Xu, C., Tian, Q.: Hmm-based audio keyword generation. In: Pacific-Rim Conference on Multimedia. pp. 566–574 (2004)
  * [33] Zhao, J., Mao, X., Chen, L.: Speech emotion recognition using deep 1d & 2d cnn lstm networks. Biomedical Signal Processing and Control 47, 312–323 (2019)