# Decoding Emotions: A comprehensive Multilingual Study
of Speech Models for Speech Emotion Recognition

###### Abstract

Recent advancements in transformer-based speech representation models have
greatly transformed speech processing. However, there has been limited
research conducted on evaluating these models for speech emotion recognition
(SER) across multiple languages and examining their internal representations.
This article addresses these gaps by presenting a comprehensive benchmark for
SER with eight speech representation models and six different languages. We
conducted probing experiments to gain insights into inner workings of these
models for SER. We find that using features from a single optimal layer of a
speech model reduces the error rate by 32% on average across seven datasets
when compared to systems where features from all layers of speech models are
used. We also achieve state-of-the-art results for German and Persian
languages. Our probing results indicate that the middle layers of speech
models capture the most important emotional information for speech emotion
recognition. GitHub111https://github.com/95anantsingh/Decoding-Emotions

Index Terms—  Speech Emotion Recognition, HuBERT, wav2vec2, Edge Probing,
Feature Extraction.

##  1 Introduction

In recent years, several transformer-based speech representation models,
trained on massive amounts of speech data, have been introduced [1] [2] [3]
[4]. These models have demonstrated remarkable improvements in performance
across various downstream tasks. Although several benchmarks exist for
evaluating their performance [5] [6], insufficient attention has been given to
evaluating these models across multiple languages. Previous large-scale works
on speech emotion recognition either do not evaluate the performance of the
latest speech representation models [7] or they only evaluate a subset of
models or languages [8] [9] [10]. Consequently, there is a gap in the
literature when it comes to comprehensive evaluations of these models for
speech emotion recognition across various languages.

Another critical aspect that has received limited attention is analyzing the
inner workings of these models using probing techniques, which can provide
valuable insights into how these models process and encode various linguistic
and acoustic features. By examining the responses of these models to specific
linguistic or emotional cues, we can develop a deeper understanding of the
strengths and limitations of these models. Recognizing emotion in speech
requires an understanding of both phonetic and the prosodic content in the
spoken utterance [11]. While previous work has explored probing techniques to
understand the phonetic content of speech representation models [12] [13]
[14], we haven’t seen studies doing the same for tasks that have a higher
dependence on prosodic content.

This paper has a dual focus. Firstly, we present a comprehensive benchmark of
multiple speech representation models for speech emotion recognition across a
range of languages. This benchmark ensures that the models are just as
applicable and relevant in diverse cultural and linguistic contexts. One of
the problems in SER is the lack of standardized training and testing splits
which allows different papers to report different performances for the task.
To counter this, we adopt a standardized train-dev-test split of Scheidwasser
et al. (2022) [7] to facilitate consistent comparisons. By providing a
standardized evaluation framework, we enable effective comparisons and
assessments of the performance of different speech emotion recognition models.

Secondly, we conduct probing experiments to gain insights into the underlying
mechanisms of speech emotion recognition across multiple languages. Unlike
[11], we find that the most important layer for speech emotion recognition are
the center layers, as can be seen in Figure 1. This observation then inspired
us to just utilize one single layer instead of the final layer or aggregate
all layers [8] for SER. We surprisingly find that when features are extracted
from the optimal layer, which are usually the center layers of the model, we
achieve best performance. This is contrary to previous work where
conventionally only final layer features or aggregating feature from different
layers [8] have lead to best results. As a result, we report state-of-the-art
results for German and Persian where only speech input is used for SER. Our
findings challenge the prevailing notion and underscore the importance of
selecting the appropriate layer to maximize the efficacy of speech
representation models in SER tasks.

Dataset | Language | Classes | Utterances | Speakers | Average Duration (s) | Total Duration (h)
---|---|---|---|---|---|---
AESDD | Greek | 5 | 604 | 6 | 4.2 | 0.7
CaFE | French | 7 | 864 | 12 | 4.5 | 1.1
EmoDB | German | 7 | 535 | 10 | 2.8 | 0.4
EMOVO | Italian | 7 | 588 | 6 | 3.1 | 0.5
IEMOCAP | English | 4 | 5,531 | 10 | 3.4 | 7.0
RAVDESS | English | 8 | 1,440 | 24 | 3.7 | 1.5
ShEMO | Persian | 6 | 3,000 | 87 | 4.0 | 3.3
Table 1: Key Attributes of Datasets Used: Average Duration, Language, Number
of Classes, Utterances, Speakers, and Total Duration of All Samples

##  2 Background

Previous research on probing speech representation models have primarily
focused on studying how speech models understand phonetic content [12] [13]
[14]. In order to probe each layer, these studies adopt a technique of
obtaining the time-average of feature vectors from the target layer and then
passing them through a linear layer for downstream tasks. The idea is to use
the least complex model for classification so that the descriptive power of
the model can be studied. Notably, these investigations reveal that wav2vec2
exhibits an auto-encoder-like behavior, with the initial and final layers
resembling the input, while the intermediate layers generate higher-level
representations that encapsulate maximum contextual information. In a study
conducted by Lin et al. (2023) regarding the utilization of speech
representation models for prosodic tasks, it was discovered that only the
initial layers encoded prosodic information. We find that speech emotion
recognition is a task done best by the center layers of the model, producing
state-of-the-art results with a very simple linear model when applied on the
right layer.

A large number of research articles exists in literature for making systems
for speech emotion recognition. In this paper, we focus on doing speech
emotion recognition with transformer-based speech representation models. We
specifically focus on using three pre-trained speech models - wav2vec2 [1],
XLSR [2] and HUBERT [3]. We do not fine-tune the weights of the speech
representation models [15] [9][16][17] and only use them as feature extractors
as we believe such systems are more practical in the real world setting -
where a single feature extractor is used to extract speech features and
multiple systems perform different tasks using the same set of features.
Unlike pre-trained language models in natural language processing where the
final layer features are used to do most tasks, Pepino et al [8] showed that
the best way to use speech models for downstream task is not to use the final
layer representations. They took a weighted averaged the features of across
all layers of wav2vec2 and then used an LSTM model to achieve state-of-the-art
results for the IEMOCAP (English) dataset. In our paper, we show that better
results can be achieved using information from the optimal layer.

Fig. 1: Dataset-wise accuracy for Probing using a Linear classifier with
various feature extractors. The presented data is an average of five runs
conducted on the test set.

In this paper, we also study speech emotion recognition across multiple
languages. Most studies on speech emotion are centred around English [8] [18]
[19] or use one or two additional languages [20] [10] [21]. Our work also
presents a benchmark for speech emotion recognition across multiple languages
using pre-trained speech representation models. Previous multlingual SER works
either don’t evaluate these representation models [7] or only use one such
model for multiple languages [9]. While such benchmarks exist in low-resourced
languages [22] [23] [24] for other tasks [24] [6], we do not yet have a
comprehensive benchmark for speech emotion recognition.

##  3 Datasets

This research explores seven speech emotion classification tasks conducted on
six distinct language datasets. The study focuses on English, French, German,
Greek, Italian, and Persian. Specifically, the English datasets used in this
study consist of two widely recognized collections, namely IEMOCAP (IEM4) [25]
and RAVDESS [26]. Additionally, the other languages are represented by CaFE
(French) [27], EmoDB (German) [28], AESDD (Greek) [29], EMOVO (Italian) [30],
and ShEMO (Persian) [31] dataset.

The datasets exhibit variations in several aspects, including size (number of
utterances), number of speakers, class distribution, and number of classes as
described in Table 1. While emotions such as anger, happiness, and sadness are
present across all datasets, additional emotions such as disgust, fear,
neutral emotion, surprise, calm, and boredom appear in at least one of the
datasets. Each dataset consists of speech samples characterized by three
crucial attributes: audio data represented as raw single channel waveforms,
speaker identification, and emotion labels encompassing various emotions such
as anger, happiness, and sadness. It is worth noting that all the datasets
exhibit similar average utterance durations, which range from 2.5 to 4.5
seconds.

Model | Layers | Training Data
---|---|---
wav2vec2 Base | 12 | 960 (1)
wav2vec2 Large | 24 | 960 (1)
wav2vec2 XLSR 53 | 24 | 56,000 (53)
wav2vec2 XLSR 300M | 24 | 436,000 (128)
wav2vec2 ASR Large | 24 | 960 + 960 (1)
HuBERT Base | 12 | 960 (1)
HuBERT Large | 24 | 60,000 (1)
HuBERT ASR Large | 24 | 60,000 + 960 (1)
Table 2: Overview of Feature Extractor Models with Number of Layers and
Training Data. Dataset size is in Hours and number of languages are mentioned
inside parenthesis and ’+’ indicates additional finetuning training data
hours.

The selection of benchmark datasets for this study was primarily based on two
key factors: dataset popularity and language diversity. The chosen benchmark
datasets, including EmoDB [28], IEMOCAP [25], and RAVDESS [26], are widely
used in the field of speech emotion recognition. To address class imbalance, a
subset of the IEMOCAP [25] dataset, specifically the four emotion classes
(IEM4), was used. For the remaining tasks, all samples and classes from the
original datasets were retained. To represent Italic languages, CaFE [27] and
EMOVO [30] were chosen, while AESDD [29] and ShEMO [31] represented the
Hellenic and Indo-Iranian branches of the Indo-European family, respectively.
The majority of the benchmark datasets primarily comprise scripted and acted
speech, with IEM4, RAVDESS [26], and ShEMO [31] also incorporating spontaneous
utterances.

To train, optimize, and evaluate language-specific speech emotion classifiers,
each dataset, following Scheidwasser et al. [7], was divided into training,
validation, and testing sets. The standard split employed for most datasets
involved allocating 60% of the data for training, 20% for validation, and 20%
for testing purposes. Speaker independence was carefully maintained in each
partition, ensuring that the sets of speakers in each partition were mutually
exclusive. This fixed data split design facilitated the assessment of the
performance of experimental setups using different amounts of data,
considering the variations in dataset sizes within the benchmark.

##  4 Models

In this paper, we work with three pre-trained speech models - wav2vec2 [1],
XLSR [2] and HUBERT [3]. We use these models as feature extractors. A summary
of these models can be found in Table 2, presenting a concise overview of
their key characteristics. For classification, we use to different heads on
top of the features extracted from these speech representation models, as
described in section 4.2.

###  4.1 Feature Extractors

We selected feature extraction models that vary in terms of pre-training data
and the number of languages involved. By incorporating models with distinct
pre-training data and linguistic coverage, we aimed to enhance the
comprehensiveness and robustness of our research findings.

####  4.1.1 wav2vec2

A total of 3 versions of wav2vec2 were studied. Among them, two versions are
pre-trained models, namely Wav2vec2 Base and Wav2vec2 Large. Additionally,
there is one version of wav2vec2 finetuned for ASR called wav2vec2 ASR Large.
These models differ in several aspects, including the number of layers in the
transformer encoder and the training data hours, as well as the number of
languages they were pre-trained on. These statistics are shown in Table 2.

The pre-training of wav2vec2 base, wav2vec2 large, and wav2vec2 ASR large was
performed using the LibriSpeech dataset, which consists of English speech data
[32].

####  4.1.2 XLS-R

We studied two versions of the XLS-R model: Wav2vec2 XLSR 53 and Wav2vec2 XLSR
300M. These models share the same architecture, with 24 encoder layers and
300M parameters.

Wav2vec2 XLSR 53 was pre-trained on a diverse dataset consisting of 53
languages. On the other hand, Wav2vec2 XLSR 300M was pre-trained on an even
more extensive dataset, comprising 128 languages. The inclusion of these two
models allowed us to explore the impact of different language coverage on the
performance and capabilities of the XLS-R model in our research.

Fig. 2: Maximum accuracies for all the classifiers models and datasets.

####  4.1.3 HuBERT

In our research experiments, we explored HuBERT by utilizing two pretrained
models and one finetuned for ASR version. The pretrained versions of HuBERT
include HuBERT Base, which has 12 encoder layers, and HuBERT Large, which has
24 encoder layers. HuBERT Base was pretrained on 960 hours of the LibriSpeech
English dataset [32]. On the other hand, HuBERT Large was pretrained using a
significantly larger dataset, specifically 60,000 hours of the Libri-Light
English dataset [33]. We also employed a finetuned version of HuBERT for ASR,
called HuBERT ASR Large. This model consists of 24 layers in its encoder and
it was pretrained on 60,000 hours of the Libri-Light dataset and then further
finetuned on 960 hours of the LibriSpeech dataset.

###  4.2 Classification Heads

Classification head is a model which takes in the learned features or
representations of the input speech data from the preceding feature extractor
and make predictions about the target class or emotion which a given input
belongs. We used two classification heads which are described below.

####  4.2.1 Linear

The Linear classification head consists of two linear layers. First layer
takes the averaged input features over the time sequence, and the rectified
linear unit (ReLU) activation function is applied to the output with hidden
dimension as 128. The resulting tensor is then passed through the second
layer, which produces the final logits for each label. This classification
head is based on the probing models used in [12] [13] [14]. The idea here is
to use the simplest feed forward neural network to be able to study the speech
representation power of pre-trained models.

####  4.2.2 Dense

Dense classification model uses piece-wise linear layers at every time step of
a layer or a CNN layer with kernel size 1. Two such layers are used with the
hidden dimensions of 256 hidden units. These features are then averaged across
time and then passed through a final classification layer. This model is used
with both single layer features and multilayer features. When using features
from multiple layers, we aggregate the features following [8] before using the
dense layer.

Fig. 3: Error reduction percentage from Aggregate Dense classifier to Probing
with Linear and Dense classifier.

##  5 Experiments

We conducted two fundamental experiments by integrating all the feature
extractors and classification heads. The first experiment involved a
straightforward aggregation model, while the second experiment focused on edge
probing to gain insights into the internal encoding schema of emotions in
speech representation models. As part of our research, we treated the CNN
layer that precedes the transformer layer as a separate layer, which we have
denoted as the ”zeroth” layer [8]. To enhance the statistical validity and
ensure the consistency of our findings, we conducted each individual trial
five times.

Models | Datasets
---|---
| AESDD | CaFE | EmoDB | EMOVO | IEMOCAP | RAVDESS | ShEMO
wav2vec 2 Base | 69.6 [5] | 70.9 [2] | 91.7 [3] | 44.4 [5] | 66.8 [6] | 65.3 [8] | 92.2 [3]
wav2vec 2 Large | 79.7 [15] | 79.9 [8] | 98.8 [9] | 50.5 [16] | 67.7 [7] | 73.7 [12] | 91.9 [5]
wav2vec 2 ASR LARGE | 79.2 [8] | 75.2 [12] | 97.6 [9] | 56.1 [9] | 69.0 [13] | 70.3 [11] | 92.2 [4]
wav2vec 2 XLSR 53 | 85.0 [14] | 84.2 [7] | 100 [13] | 61.7 [9] | 71.7 [10] | 77.7 [9] | 95.2 [6]
wav2vec 2 XLRS 300 | 83.1 [16] | 83.3 [13] | 100 [7] | 69.9 [14] | 73.9 [13] | 79.0 [18] | 94.6 [17]
HuBERT Base | 72.9 [7] | 73.1 [7] | 96.4 [4] | 48.5 [6] | 69.2 [4] | 65.7 [7] | 91.7 [1]
HuBERT Large | 80.2 [14] | 83.8 [13] | 100 [15] | 62.8 [12] | 72.5 [22] | 76.3 [11] | 93.5 [5]
HuBERT ASR LARGE | 82.6 [13] | 82.5 [10] | 100 [10] | 67.3 [12] | 71.4 [10] | 75.3 [12] | 93.8 [8]
Table 3: Maximum accuracies for Probing using the Dense classifier with
various feature extraction models. The corresponding layer of the encoder in
the feature extractor is denoted within square brackets.

###  5.1 Edge Probing - Linear

Edge probing involves conducting targeted analyses and experiments to
investigate how the model’s internal representations and attention mechanisms
respond when an input is sent through the model. For a given model with layers
L, we attached a separate classification head to each layer and trained them
independently to predict a target label. The classifier solely relies on input
from a single layer. We conducted this experiment using both a Linear
classification head and a Dense classification head. Consequently, the
classifier heavily relies on the encoder to provide meaningful information
regarding the relationship between spans and their respective roles within a
sentence. This approach empowered us to evaluate the encoder’s proficiency in
capturing and conveying emotional features from the high density speech input
data.

The results of the probing experiments for five models are shown in Figure 1.
We find that the initial and final layers perform worst for the task of speech
emotion recognition. The initial layers are unable to create a rich enough
representation of speech to classify emotions and classifying emotions
requires both the understanding of the phonetic as well as prosodic content.
The final layers of these models, as shown in [12] [13] [14] are focused on
reconstructing the input and lose the rich contextual representations for
emotion recognition to be able to provide enough phonetic content for speech
reconstruction. This can also be seen in Figure 1 where the performance of the
final ASR layers are always worse than the non-ASR models, showing that models
trained to convert speech to text lose out on a lot more prosodic information
than their non-ASR counterparts. The center layers contain the richest
contextual features that contain enough phonetic and prosodic content to do
speech emotion recognition. This observation is true across six different
languages.

###  5.2 Aggregation

The aggregation experiments follow the work in [8], where feature
representations from all the layers are used to train an SER model. To combine
features from different layers, we use a weighted average for each layer as
done in [8], where these weights are learnable parameter. We then use the
dense layer to work with these weighted multi-layer features. Upon performance
evaluation of this model, we found that the aggregation based models were
performing worse than the linear edge-probing models. This can also be seen in
Figure 2, where Agg. Dense performance if always worse than the Probing
withLinear classifier performance except for the IEMOCAP (English) dataset.
This motivated us to push the performance using a single layer of the pre-
trained models using the same classification head.

###  5.3 Edge Probing - Dense

To extract even more out of the edge probing models, we use the dense
classification head on the features extracted from a single layer. We find
that the the dense probing models outperform both linear probing and dense
aggregate models as shown in Figure 2. This means that features extracted from
a single layer are enough to achieve best performance on SER over using
features extracted from all layers, even though the features are combined
using learnable weights.

The improvement in performance over aggregation experiments is highlighted
further in Figure 3. The plots show the error reduction percentage over the
aggregation models when using the linear and dense probing models. We observed
that for 5 out of 7 datasets, the linear probing model performs better than
the aggregation models, and the dense probing model is better than the
aggregation model for all datasets. We also noticed that the dense probing
model closes the error of the aggregation model by 5-100%, with an average
error reduction about 32% across 7 datasets and 6 languages.

Figure 3 also shows that the maximum improvements with dense probing models
happen for the smallest datasets. This means that for low resourced scenarios,
if the training dataset is small, the middle layers become even more crucial
for accurately classifying emotions and aggregation models as proposed in [8]
require a larger amount of data to achieve optimal performance. The exact
values of classification accuracies for the dense probing head with the
corresponding best layer for each model are shown in Table 3. Unsurprisingly,
we find the XLSR models trained on large multilingual data performs best for
speech emotion recognition, including doing SER for English.

##  6 Conclusion

We conducted a comprehensive evaluation of transformer-based speech
representation models for speech emotion recognition (SER) across multiple
languages. Our findings challenge prevailing notions and provide valuable
insights for optimizing SER models in multilingual and low-resourced
scenarios. Our probing experiments show that the middle layers of the speech
representation models capture the most important features for SER. This is
contrary to previous work [11], which shows that the early layers of speech
models capture prosodic information, highlighting the importance of both
phonetic and prosodic information for recognizing emotion in speech.

We also discovered that single-layer probing models consistently outperformed
aggregation models in terms of accuracy and variability. The findings of this
study challenge the previous work by Pepino et al. [8], as they demonstrate
that utilizing features from a single layer of a speech representation model
outperforms the approach of aggregating features from all layers of the model.
The optimal layers were usually the center layers of the model, thus further
highlighting the importance of the center layers for the task of speech
emotion recognition. The optimal single layer although seems to differ from
task to task and dataset to dataset, and finding the optimal layer is a part
of our future investigations.

Additionally, dataset size played a crucial role, with aggregation models
requiring more data to perform well. Furthermore, we observed that models
trained on a larger number of languages exhibited better encoding of emotions,
emphasizing the importance of linguistic diversity in pre-training.

## References

  * [1] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [2] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli,  “Unsupervised cross-lingual representation learning for speech recognition,”  arXiv preprint arXiv:2006.13979, 2020.
  * [3] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [4] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever,  “Robust speech recognition via large-scale weak supervision,”  in International Conference on Machine Learning. PMLR, 2023, pp. 28492–28518.
  * [5] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “Superb: Speech processing universal performance benchmark,”  arXiv preprint arXiv:2105.01051, 2021.
  * [6] Jiatong Shi, Dan Berrebbi, William Chen, Ho-Lam Chung, En-Pei Hu, Wei Ping Huang, Xuankai Chang, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, et al.,  “Ml-superb: Multilingual speech universal performance benchmark,”  arXiv preprint arXiv:2305.10615, 2023.
  * [7] Neil Scheidwasser-Clow, Mikolaj Kegler, Pierre Beckmann, and Milos Cernak,  “Serab: A multi-lingual benchmark for speech emotion recognition,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7697–7701.
  * [8] Leonardo Pepino, Pablo Riera, and Luciana Ferrer,  “Emotion recognition from speech using wav2vec 2.0 embeddings,”  arXiv preprint arXiv:2104.03502, 2021.
  * [9] Mayank Sharma,  “Multi-lingual multi-task speech emotion recognition using wav2vec 2.0,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6907–6911.
  * [10] Zihan Wang, Qi Meng, HaiFeng Lan, XinRui Zhang, KeHao Guo, and Akshat Gupta,  “Multilingual speech emotion recognition with multi-gating mechanism and neural architecture search,”  in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 806–813.
  * [11] Guan-Ting Lin, Chi-Luen Feng, Wei-Ping Huang, Yuan Tseng, Tzu-Han Lin, Chen-An Li, Hung-yi Lee, and Nigel G Ward,  “On the utility of self-supervised models for prosody-related tasks,”  in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1104–1111.
  * [12] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah,  “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,”  arXiv preprint arXiv:2101.00387, 2021.
  * [13] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [14] Ankita Pasad, Bowen Shi, and Karen Livescu,  “Comparative layer-wise analysis of self-supervised speech models,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [15] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba,  “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,”  arXiv preprint arXiv:2111.02735, 2021.
  * [16] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Zhuo Chen, Peidong Wang, Gang Liu, Jinyu Li, Jian Wu, Xiangzhan Yu, et al.,  “Why does self-supervised learning for speech recognition benefit speaker recognition?,”  arXiv preprint arXiv:2204.12765, 2022.
  * [17] Li-Wei Chen and Alexander Rudnicky,  “Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [18] Xingyu Cai, Jiahong Yuan, Renjie Zheng, Liang Huang, and Kenneth Church,  “Speech emotion recognition with multi-task learning.,”  in Interspeech, 2021, vol. 2021, pp. 4508–4512.
  * [19] Yuanchao Li, Yumnah Mohamied, Peter Bell, and Catherine Lai,  “Exploration of a self-supervised speech model: A study on emotional corpora,”  in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 868–875.
  * [20] Ali Yazdani, Hossein Simchi, and Yasser Shekofteh,  “Emotion recognition in persian speech using deep neural networks,”  in 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE). IEEE, 2021, pp. 374–378.
  * [21] Samir Sadok, Simon Leglaive, and Renaud Séguier,  “A vector quantized masked autoencoder for speech emotion recognition,”  arXiv preprint arXiv:2304.11117, 2023.
  * [22] Akshat Gupta,  “On building spoken language understanding systems for low resourced languages,”  arXiv preprint arXiv:2205.12818, 2022.
  * [23] Akshat Gupta, Xinjian Li, Sai Krishna Rallabandi, and Alan W Black,  “Acoustics based intent recognition using discovered phonetic units for low resource languages,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7453–7457.
  * [24] Akshat Gupta, Olivia Deng, Akruti Kushwaha, Saloni Mittal, William Zeng, Sai Krishna Rallabandi, and Alan W Black,  “Intent recognition and unsupervised slot identification for low-resourced spoken dialog systems,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 853–860.
  * [25] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan,  “Iemocap: interactive emotional dyadic motion capture database,”  Language Resources and Evaluation, vol. 42, pp. 335–359, 2008.
  * [26] Steven R. Livingstone and Frank A. Russo,  “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,”  PLOS ONE, vol. 13, pp. 1–35, 05 2018.
  * [27] Philippe Gournay, Olivier Lahaie, and Roch Lefebvre,  “A canadian french emotional speech dataset,” Apr. 2018.
  * [28] Felix Burkhardt, Astrid Paeschke, M. Rolfes, Walter F. Sendlmeier, and Benjamin Weiss,  “A database of german emotional speech,”  in Interspeech, 2005.
  * [29] Nikolaos Vrysas, Rigas Kotsakis, A. Liatsou, Charalampos A. Dimoulas, and George M. Kalliris,  “Speech emotion recognition for performance interaction,”  Journal of the Audio Engineering Society, 2018.
  * [30] Giovanni Costantini, Iacopo Iaderola, Andrea Paoloni, and Massimiliano Todisco,  “EMOVO corpus: an Italian emotional speech database,”  in Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, May 2014, pp. 3501–3504, European Language Resources Association (ELRA).
  * [31] Omid Mohamad Nezami, Paria Jamshid Lou, and Mansoureh Karami,  “Shemo – a large-scale validated database for persian speech emotion detection,” 2019.
  * [32] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [33] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux,  “Libri-light: A benchmark for asr with limited or no supervision,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7669–7673.