# Double Multi-Head Attention Multimodal System for Odyssey 2024 Speech
Emotion Recognition Challenge

###### Abstract

As computer-based applications are becoming more integrated into our daily
lives, the importance of Speech Emotion Recognition (SER) has increased
significantly. Promoting research with innovative approaches in SER, the
Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the
Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we
describe the Double Multi-Head Attention Multimodal System developed for this
challenge. Pre-trained self-supervised models were used to extract informative
acoustic and text features. An early fusion strategy was adopted, where a
Multi-Head Attention layer transforms these mixed features into complementary
contextualized representations. A second attention mechanism is then applied
to pool these representations into an utterance-level vector. Our proposed
system achieved the third position in the categorical task ranking with a
34.41% Macro-F1 score, where 31 teams participated in total.

##  1 Introduction

Natural Human-Computer Interaction (HCI) is becoming more and more important
as machines and computer-based applications are gaining significance in our
daily lives [1]. A key step in HCI is to recognize and address people’s
emotional states. In addition, automatic emotion detection and classification
are essential for automated tutoring systems, call centers, gaming, personal
assistants and more. For these reasons, progress in these technologies can
enable transformative applications in diverse areas such as healthcare,
security and defense, education and entertainment. Humans use various
modalities to express their emotions, such as speech, text, facial expressions
or gestures. Because of the subtle expressive behaviors that arise during
human interactions, automatic emotion recognition from speech in realistic
domains is still a challenging task [2].

Speech is essential for expressing a person’s emotional state through prosody
and/or paralinguistic context. In the domain of Speech Emotion Recognition
(SER), Machine Learning (ML) models have been constructed relying on the
utilization of hand-crafted features, such as Mel-frequency Cepstral
Coefficients (MFCC), pitch, energy, entropy or zero-crossing rate [3]. The
correlation between different emotions and specific speech features is still
uncertain, and ongoing research is being conducted to investigate specific
speech representations that can effectively capture human emotions’ patterns.
However, deep learning-based models do not require the extraction of a large
set of hand-crafted features because they can learn the features that are
relevant to the task from spectrograms or even directly from raw waveforms.
Nevertheless, a large amount of training data is needed to construct these
complex systems because a lack of it could result in poor generalization
performance on unseen data conditions.

Emotions can also be captured using text information. Several techniques have
been developed to represent and model this type of data. In many Natural
Language Processing (NLP) applications, word and sentence embeddings (such as
Word2Vec [4] or BERT-based [5]) have proven to be the most efficient
representations, yielding significant improvements in deep learning models
performance. Despite offering an enhanced representation of textual
information, these techniques have certain limitations when applying them
effectively to different tasks. One of the principal difficulties is that
training these models requires huge amounts of data and computational power.

The Odyssey2024 Speech Emotion Recognition Challenge [6] was organized as part
of the Odyssey 2024 Speaker and Language Recognition Workshop to compare
different emotion recognition systems submitted by teams all around the world.
The challenge consists of two independent tasks allowing to compare different
emotion recognition systems submitted by teams all around the world. The first
task objective is to classify speech segments across the eight categorical
emotional classes provided: anger, happiness, sadness, fear, surprise,
contempt, disgust and a neutral state. The second task consists of predicting
emotional attributes for arousal (calm to active), valence (negative to
positive), and dominance (weak to strong). The challenge uses recordings from
the MSP-Podcast corpus, which contains speech segments obtained from audio-
sharing websites [7].

In [8], the authors proposed a sub-vector-based multi-head attention pooling
to efficiently obtain discriminative speaker embeddings given non-fixed-length
speech utterances. In this system, a Convolutional Neural Network (CNN)
encodes short-term speaker features from the spectrogram, which are split into
homogeneous sub-vectors. Each self-attention pooling head is applied over a
split of sub-vectors. This mechanism allows the model to attend to the most
informative patterns from different positions of the encoded sequence. In a
later work [9], the authors present the Double Multi-Head Attention (DMHA),
where a second self-attention pooling is applied. This additional pooling
layer allows the model to select which head context vectors are the most
relevant to produce an enhanced utterance-level vector. Since this network is
trained as a speaker classifier, it can be easily adapted to solve Speaker
Classification tasks such as SER [10].

In this paper we describe our Double Multi-Head Attention Multimodal System.
We used this architecture for the Odyssey 2024 Speech Emotion Recognition
Challenge categorical recognition task. Since self-supervised pre-trained
models learn universal representations from massive unlabeled speech data and
adapt effectively across various downstream tasks, several pre-trained
versions of these architectures were explored to extract acoustic features.
Additionally, speech utterances were automatically transcribed with Whisper
[11] and then text features were extracted using a pre-trained BERT [5] model.
A Multi-Head Attention layer is used to transform these speech and text
features into contextualized representations. Using this attention technique
allows the model to learn complex relationships between acoustic and
linguistic information. These contextualized representations are then pooled
into an utterance-level vector using a second attention mechanism. Finally, we
used a set of fully connected layers to classify the corresponding emotions.
Models were trained by applying an on-line data augmentation process. Three
different models were used in a hard-voting ensemble to achieve the third
position in the Task 1 Challenge ranking, where 31 teams participated in
total.

The rest of this paper is structured as follows. In Section 2 we describe
several studies that are related to our work. In Section 3 Double Multi-Head
Attention is explained. Section 4 details our proposed system. In Section 5
experimental setups and results are included. Concluding remarks are given in
Section 6.

##  2 Related Work

Deep Neural Networks have demonstrated remarkable success in extracting
emotions from speech signals in the last years [12, 13, 14, 15, 16]. Commonly,
a SER system includes a feature extraction component and a pooling layer that
aggregates the extracted features into a single utterance-level vector.
Finally, a classification layer takes the pooled vector as input and outputs
each emotion probability. Nevertheless, the automatic recognition of emotions
from speech in realistic domains remains a significant challenge, primarily
due to the intricate nature of expressive behaviors that manifest during human
interactions.

One major obstacle in SER is still the lack of substantial amounts of labelled
data needed to construct complex enough deep learning models [17]. To address
this problem, several data augmentation methods were developed to generate
synthetic data [18, 19]. Additionally, unsupervised or self-supervised pre-
training of neural networks emerged as an effective technique for settings
where labelled data is scarce. The key idea is to learn general
representations in a setup where substantial amounts of data are available.
These learned representations are then used to improve performance on a
downstream task for which the amount of data is limited. This is particularly
interesting for tasks like SER, where a substantial effort to obtain labelled
data is required.

SER systems take audio signals as input. In order to improve the performance
of these systems, a feature extraction process is usually applied. In this
process, features are extracted from the audio signal by changing the speech
waveform to a form of parametric representation. Several options are available
to parametrically represent the speech signal for the recognition process. The
Mel-frequency Cepstral Coefficients (MFCC) feature extraction strategy is
recognized as one of the most effective and universally adopted techniques in
the speech recognition domain [20]. Nevertheless, there are no guarantees that
these hand-crafted features are optimal for all speech-related tasks and they
might limit the potential of the powerful representation of DNN systems.

To mitigate this drawback, some works aim to extract learnable acoustic
features from the raw audio waveforms. In [21], the wav2vec2.0 architecture
quantizes continuous speech data into discrete units automatically and then
guesses the correct discrete units at randomly masked locations using
Transformers [22]. XLS-R [23] is a large-scale model for cross-lingual speech
representation learning based on wav2vec2.0. Since it covers a wide range of
languages, data regimes and domains, it shows a remarkable generalization
capacity. Hidden-Unit BERT (HuBERT) [24] is another self-supervised speech
representation learning approach, which utilizes an offline clustering step to
provide aligned target labels for a BERT-like prediction loss. WavLM [25]
jointly learns masked speech prediction and denoising in pre-training. Some
inputs are simulated noisy/overlapped speech with masks and the target is to
predict the pseudo-label of the original speech on the masked region like
HuBERT. By doing this, wavLM learns universal speech representations from
massive unlabeled speech data and adapts effectively across various speech
processing tasks.

Regarding text feature extraction methods, word embeddings have become the
standard representation in many natural language processing tasks, being
Word2Vec [4] and GloVe [26] some of the most widely adopted. Both unsupervised
models have shown great success in a range of NLP tasks such as sentiment
analysis, document indexing, and topic model analysis. Nevertheless, a
significant drawback of these models is that they ignore word order, which
results in a loss of syntactic and semantic information of words in sentences.
By extracting representations from text data to capture the context, BERT [5]
models are used to solve this limitation. BERT stands for Bidirectional
Encoder Representations from Transformers and is a language model trained on
large amounts of unlabelled text data that has achieved state-of-the-art
results on many NLP tasks. In order to enhance the performance of emotion
recognition, several studies have effectively combined speech-based
representations with BERT-based embeddings [27, 28, 12].

Pooling strategies can be classified into two main classes: statistical and
learning-based pooling. Within the statistical pooling strategies, the most
commonly used are average pooling and statistical pooling [12, 29]. These
strategies assume that all elements of the pooling component input vectors
contribute equally to the utterance-level vector creation. Since every frame
may not provide equal speech discriminative information, many works applied
self-attention mechanisms for weighted learning-based pooling layers. Single-
head attention pooling with different self-attentive scoring functions was
explored in [30, 16, 31]. Also, multi-head attention pooling approaches were
applied successfully in several works [32, 33, 9]. Because pooling layers
behave differently with different datasets, network structures or loss
functions, it is difficult to conclude which one is the best, if there is one.
Pooling functions with learnable parameters achieved better results than the
simple pooling layers such as the average pooling and statistical pooling in
most cases, with a weakness of higher computational complexity than the
latter.

Multimodal emotion recognition (MER) aims to identify human emotional states
by combining different data modalities such as text, speech, images or videos.
With the advance of deep learning technologies and the increasing availability
of multimodal datasets many MER systems have been explored [34, 35]. The
expressiveness of unimodal information is naturally restricted, so it
frequently fails to fully convey the wide range of human emotions. Since
spoken data consists of audio and text information, studies have explored the
combination of acoustic features and linguistic content for emotion
recognition. The fusion of diverse types of signals to obtain complementary
information is key to improving model performance in MER. These approaches
implement this combination by mixing either each modality embeddings (known as
early fusion) or decision scores (late fusion) [36, 27, 28, 12].

##  3 Double Multi-Head Attention

Attention mechanisms have become a standard component of deep learning
networks, contributing to remarkable results improvements [37]. In [22], a
Multi-Head Attention (MHA) mechanism was proposed. Instead of performing a
single attention function, first the queries, keys and values are projected
$H$ times with different learned linear projections. Then, an attention
function is applied over each of these projected versions of queries, keys and
values.

In [8], the authors used a sub-vector-based MHA pooling as an efficient
mechanism to obtain discriminative speaker embeddings from speech utterances.
Frame-level features are split into $H$ sets of equally-sized sub-vectors and
an attention pooling is applied over each set of sub-vectors. The attention
performed over each set of sub-vectors is a dot-product attention, where the
keys and the values projections are the sub-vectors and there is only one
trainable query. In contrast with standard MHA [22], this sub-vector MHA
outputs $H$ pooled vectors and significantly reduces the number of learnable
parameters.

In the architecture designed in [8], the authors extracted speech features
from the spectrogram using a CNN component. In this paper we extract features
directly from the raw waveform with pre-trained self-supervised models,
allowing to remove the CNN component. Combining this removal with a Data
Augmentation process enables to significantly increase the number of
parameters of the model without losing its generalization ability beyond the
training data. Motivated by this, we experiment replacing the efficient sub-
vector MHA [8] layer with a standard MHA [22] layer, allowing the model to
capture more complex relationships and patterns.

In a later work [9], the authors presented the Double MHA (DMHA) approach.
After a first sub-vector-based MHA layer, a second attention layer is applied.
This additional layer uses an attention pooling mechanism over the first layer
output vectors by performing a dot-product attention, where the keys and the
values projections are the vectors and there is only one trainable query. This
second layer allows the model to select which vectors are the most relevant to
produce the final pooled vector.

Standard and sub-vector variants for the first attention layer of the DMHA are
formulated in subsections 3.1 and 3.2. The second attention layer of the DMHA
architecture is explained in subsection 3.3.

###  3.1 Standard Multi-Head Attention

Given a sequence of hidden-states $\\{h_{t}\in\mathbb{R}^{D}\mbox{ }|\mbox{
}t=1,...,T\\}$, the output of a $H$-headed standard MHA [22] layer are $T$
vectors. These are computed using learnable matrices $W_{j}^{Q}$, $W_{j}^{K}$,
$W_{j}^{V}\in\mathbb{R}^{D\times D}$ for each head $j$, and
$W^{O}\in\mathbb{R}^{HD\times D}$:

$\mbox{MHA}(X)=\mbox{concat}\left(\mbox{head}_{1},\dots,\mbox{head}_{H}\right)W^{O},$
(1)

where $X\in\mathbb{R}^{T\times D}$ is the result of packing together the input
sequence $h_{1},...,h_{T}$ and

$\mbox{head}_{j}=\mbox{softmax}\left(\frac{XW_{j}^{Q}(XW_{j}^{K})^{T}}{\sqrt{D}}\right)XW_{j}^{V}.$
(2)

This mechanism, which we refer to as standard MHA (sometimes also referred to
as Global MHA), allows the model to jointly attend to information from
different representation subspaces at different positions.

###  3.2 Sub-vector Multi-Head Attention

Given a sequence of hidden-states $\\{h_{t}\in\mathbb{R}^{D}\mbox{ }|\mbox{
}t=1,...,T\\}$, the output of a $H$-headed sub-vector MHA [8] layer are $H$
vectors. Each hidden state $h_{t}$ is split into $H$ new hidden states so that
$h_{t}=[h_{t1},...,h_{tH}]$, where $h_{tj}\in\mathbb{R}^{D/H}$. Now, for each
head $j$, self-attention is applied over $[h_{1j},...,h_{Tj}]$. Each head
weights are defined as:

$w_{tj}=\frac{\text{exp}\left(\frac{h_{tj}^{T}u_{j}}{\sqrt{d_{h}}}\right)}{\sum_{l=1}^{T}\text{exp}\left(\frac{h_{lj}^{T}u_{j}}{\sqrt{d_{h}}}\right)},$
(3)

where $w_{tj}$ corresponds to the attention weight of the head $j$ on the step
$t$ of the sequence, $u_{j}\in\mathbb{R}^{D/H}$ is a trainable parameter and
$d_{h}$ corresponds to the hidden state dimension $D/H$. We then compute a
pooled representation $c_{j}$ for each head:

$c_{j}=\sum_{t=1}^{T}w_{tj}^{{}^{\prime}}h_{tj}$ (4)

Each self-attention pooling for the head $j$ is a dot-product attention over
$[h_{1j},...,h_{Tj}]$, where the keys and the values are both
$[h_{1j},...,h_{Tj}]$ and there is only one trainable query $u_{j}$. With this
design, this mechanism allows the model to efficiently attend to the most
informative patterns from different positions of the hidden states sequence,
significantly reducing the number of learnable parameters.

###  3.3 Attention Pooling

Let $\\{c_{l}\in\mathbb{R}^{C}\mbox{ }|\mbox{ }l=1,...,L\\}$ be the output
vectors of the first attention layer. If standard MHA is used in the first
layer, $L$ is equal to $T$; if sub-vector MHA is used, $L$ is equal to $H$.
Self-attention is now used to pool these vectors to obtain an overall
utterance-level vector $c$:

$w_{i}^{{}^{\prime}}=\frac{\text{exp}\left(\frac{c_{i}^{T}u^{{}^{\prime}}}{\sqrt{C}}\right)}{\sum_{l=1}^{L}\text{exp}\left(\frac{c_{l}^{T}u^{{}^{\prime}}}{\sqrt{C}}\right)}$
(5) $c=\sum_{l=1}^{L}w_{l}^{{}^{\prime}}c_{l}$ (6)

where $w_{i}^{{}^{\prime}}$ corresponds to the aligned weight of $c_{i}$ and
$u^{{}^{\prime}}\in\mathbb{R}^{C}$ is a trainable parameter.

With this method, each vector $c$ is computed as a weighted average of
vectors, allowing the system to learn the relevance of each of these vectors
for each utterance.

##  4 System Description

Figure 1: Double Multi-Head Attention Multimodal System architecture.

Figure 1 illustrates the architecture of our proposed multimodal emotion-
recognition framework. The following subsections describe the components used
in the Double Multi-Head Attention Multimodal System architecture.

###  4.1 Speech Features

Acoustic features were extracted directly from the waveform using pre-trained
self-supervised models. Some of the most widely used and best-performing
models to that end are wav2vec2.0 [21], XLS-R [23], HuBERT [24] and wavLM
[25]. These pre-trained models get speech features representations using
several Transformer layers. Every model has different versions with variations
in the number of Transformer layers, the dimension of the speech
representations and the training data. We experimented with the following:

  * •

wav2vec2.0: wav2vec2.0 model (”large-lv60k” architecture), pre-trained on
60,000 hours of unlabeled audio, not fine-tuned.

  * •

XLS-R: XLS-R model with 300 million parameters, pre-trained on 436,000 hours
of unlabeled audio from multiple datasets in 128 languages, not fine-tuned.

  * •

HuBERT: HuBERT model (”large” architecture), pre-trained on 60,000 hours of
unlabeled audio, not fine-tuned.

  * •

wavLM: wavLM Large model (”large” architecture), pre-trained on more than
80,000 hours, not fine-tuned.

Every one of these versions uses 24 Transformer layers to get 1024-dimensional
speech representations. Since every Transformer layer may produce a different
level of complexity, a multi-level aggregation is often suggested [38]. To
explore this approach, we extract the acoustic features of each model
performing a weighted sum of every Transformer layer, where the weights were
learned by the model at the training stage.

###  4.2 Text Features

To extract text features from the waveforms we first used Whisper [11] to get
audio transcriptions. Whisper is an Automatic Speech Recognition Transformer-
based system trained with up to 680,000 hours of multilingual and multitask
data. When compared to humans, this model approaches their accuracy and
robustness. A pre-trained BERT model was used to extract text features from
the transcriptions. The BERT large uncased pre-trained version was used, which
was trained on lower-cased English text. This version has a total of 340M
parameters, using 24 Transformer layers to output 1024-dimensional text
representations. In this case, only the last Transformer layer output was
used.

###  4.3 Double Multi-Head Attention Multimodal Fusion

Given a speech segment, the input of the DMHA Multimodal Fusion component is
the acoustic and text features. Several studies have explored the combination
of acoustic features and linguistic content for emotion recognition,
concluding that the fusion of diverse types of signals is key to improving
model performance [34]. Motivated by this, these acoustic and text features
were mixed into a first MHA layer to let the model learn complementary
information using self-attention scores. Then, a second attention layer is
applied to generate an utterance-level pooled vector.

If there are $T_{1}$ acoustic features and $T_{2}$ text features, the
$T=T_{1}+T_{2}$ features are input to the first MHA layer. This first
attention layer can consist of a sub-vector MHA or a standard MHA as described
in section 3. If a $H$-headed sub-vector MHA is used, this first layer will
output $H$ vectors. In the case of a $H$-headed standard MHA, this first layer
will output $T$ vectors. We experimented with 4-headed sub-vector and standard
MHA layers. This first MHA layer outputs multimodal contextualized
representations which are then aggregated into a single utterance-level vector
using a second attention layer with an attention pooling strategy as described
in section 3.3.

###  4.4 Classification Layer

The aggregated utterance-level vector $c$ obtained from the pooling component
is fed into a classification layer. This component consists of a set of fully
connected layers: an input layer, several hidden layers and an output layer.
The input layer has the same width as $c$’s dimension. The output layer has
the same width as the number of classes of the task and a SoftMax layer is
applied to get each class probability.

##  5 Experiments

###  5.1 Database

We evaluate the effectiveness of the proposed multimodal emotion recognition
system on the MSP-Podcast dataset [7], which contains speech segments obtained
from audio-sharing websites. The speaking turns have been perceptually
annotated by at least five raters with categorical and attribute-based
emotional labels. Categorical eight emotional classes provided in the dataset
are happiness, sadness, fear, surprise, contempt, disgust and a neutral state.
While the distribution for attribute-based emotional labels has balanced
content, the distribution for emotional categories is less balanced. The
training set has 68,119 speaking turns. The development set has 19,815
speaking segments from 454 speakers. The test set comprises 2,347 unique
segments from 187 speakers, for which the labels have not been made publicly
available for the challenge.

It has been proven that using Data Augmentation techniques to increase the
volume of the training data significantly improves the performance of speech
recognition systems [19]. An online Data Augmentation process [19] was used
applying speaker augmentation with speed perturbation [39], room impulse
responses (RIRs) from [40] and background noises from the MUSAN database [41]
directly to the waveform.

###  5.2 Experimental Setup

Raw waveforms were normalized using their overall mean and standard deviation
from the training dataset. We trained our models using batches of size 32.
Every speech signal was randomly cropped with a 5.5-second window, given that
the median of the training utterances length is 5.2 seconds. When needed, a
repetition padding was used. Each waveform was augmented with a $0.5$
probability. In that case, one augmentation technique was randomly chosen from
speed perturbation, RIRs or background noises. Full-length waveforms without
augmentation were used at the evaluation stage.

Acoustic and text features were extracted using frozen self-supervised pre-
trained models. Each feature has a dimension equal to 1024. Pre-computed
Whisper transcriptions were used before extracting text features. A 4-headed
Double Multi-Head Multimodal Fusion was used as the pooling component. We
experimented using both standard and sub-vector-based MHA layers. Finally, the
8-emotion classification was done by passing the pooled vector through the
classification component. The input layer of the classification component has
the same width as the aggregated utterance-level vector dimension obtained
from the pooling component. The hidden layers of the classification component
are a set of 4 512-dimensional dense layers. Each of the input and hidden
layers is followed by a Layer Normalization [42], Gaussian Error Linear Units
(GELU) activations [43] and a $0.1$ probability of dropout.

Since the F1-score (macro) was the metric defined to evaluate the categorical
emotions prediction task (task 1 from the Challenge), training and validation
phases were monitored and evaluated with this score. Each team had a maximum
of three submissions per task and the submitted systems were evaluated in the
test set, from which labels were not publicly available. Every model was
trained for 20 epochs, using early stopping to avoid overfitting. All our
models were trained and evaluated using 2 GPUs (NVIDIA GeForce GTX TITAN X,
NVIDIA TITAN Xp or NVIDIA GeForce RTX 2080 Ti). Every 20 epochs training took
around 24 hours to finish. AdamW [44] was selected as the optimizer. Different
learning rates were tried, using a 50% decay every 5 epochs without validation
of F1-score improvement.

Regarding loss functions, we experimented with Weighted Cross-Entropy (WCE)
Loss and Focal Loss [45] since they both act as effective alternatives to deal
with class imbalance. The training inverse frequency of each class was used as
weights for the WCE Loss. The Focal Loss function is a dynamically scaled
cross-entropy loss, where the scaling factor decays to zero as confidence in
the correct class increases. Intuitively, this scaling factor can
automatically down-weight the contribution of easy examples during training
and rapidly focus the model on hard examples. This scaling factor is governed
by a parameter gamma, which was set to $2$.

Once each model was trained, a per-class threshold adjustment was performed.
The idea is to adjust each class decision threshold to improve their precision
and recall. For each class, we convert the task to a binary classification by
treating the rest as a whole one negative class. Then, different thresholds
were explored to maximize the training Macro F1-score. Once every threshold is
set, the way of predicting the final class is the following: if the class with
the highest probability output surpass its corresponding threshold, then that
class is predicted (no change is done). If the class with the highest
probability output do not surpass its corresponding threshold, then the class
with the next highest probability is predicted.

After the training phase and threshold adjustments were applied, we
experimented with mixing different models to get a 3-model ensemble. The
motivation for using ensemble models is to reduce the generalization error of
the final prediction. As long as the base models are diverse enough, the
prediction error of the ensembled model can decrease. A hard voting strategy
was adopted for the ensemble. In case of a tie, we used the prediction of a
predefined model (the one with the best validation Macro F1-score).

Configuration Train Macro F1-score Validation Macro F1-score $\downarrow$
Ensemble of models $34.88\%$ $33.80\%$ WCE Loss XLS-R $34.42\%$ $33.43\%$
Focal Loss XLS-R $39.15\%$ $33.37\%$ WCE Loss wav2vec2.0 $34.83\%$ $32.69\%$
Focal Loss HuBERT $37.84\%$ $32.40\%$ WCE Loss HuBERT $36.73\%$ $32.18\%$ WCE
Loss wavLM $32.48\%$ $31.44\%$ Focal Loss wav2vec2.0 $35.75\%$ $31.27\%$ Focal
Loss wavLM $33.24\%$ $30.77\%$ Challenge Official Baseline - $30.70\%$

Table 1:  Experimental results using the standard MHA mechanism in the DMHA
component. Threshold adjustment was applied to every model, except for the
Challenge Official Baseline. Ensemble of models combine the following three
models: WCE Loss and XLS-R; WCE Loss and wav2vec2.0; WCE Loss and wavLM.

###  5.3 Results

We trained several models combining different loss functions (WCE loss and
Focal loss), pre-trained self-supervised speech features extractors
(wav2vec2.0, XLS-R, HuBERT and wavLM) and MHA mechanisms (standard and sub-
vector MHA). A model ensemble using the following three configurations was
also evaluated: WCE loss and XLS-R speech features extractor; WCE loss and
wav2vec2.0 speech features extractor; WCE loss and wavLM speech features
extractor. The choice of these three models was motivated by the idea that
extracting different patterns from the same speech utterance may contribute to
the final system diversity. After applying threshold adjustment, we evaluate
them using Macro F1-score both in the training and validation datasets of the
competition. Applying threshold adjustment increased validation Macro F1-score
approximately in a 1% absolute both for the Standard and the sub-vector-based
models, being a useful strategy for this competition.

We can see from Table 1 results that our best model is the ensemble of models.
This configuration was used to obtain our best submission ranking at the
Challenge. This model outperforms the challenge official baseline with a 3.1%
absolute increase in the validation Macro F1-score. In terms of relative
improvement, our best model gained 10.1% validation Macro F1-score compared to
the baseline. When using standard MHA, models with XLS-R [23] as speech
feature extractor were the ones with the best performance.

Configuration Train Macro F1-score Validation Macro F1-score $\downarrow$ WCE
Loss HuBERT $37.88\%$ $30.84\%$ Focal Loss XLS-R $37.41\%$ $30.73\%$ Challenge
Official Baseline - $30.70\%$ WCE Loss XLS-R $36.05\%$ $30.53\%$ Focal Loss
wav2vec2.0 $31.20\%$ $29.31\%$ WCE Loss wav2vec2.0 $30.29\%$ $28.93\%$ Focal
Loss HuBERT $31.94\%$ $28.89\%$ Focal Loss wavLM $32.28\%$ $27.67\%$ WCE Loss
wavLM $31.69\%$ $26.13\%$

Table 2:  Experimental results using the sub-vector MHA mechanism in the DMHA
component. Threshold adjustment was applied to every model, except for the
Challenge Official Baseline.

Figure 2 shows the validation confusion matrix for the ensemble model. We can
see that the system achieves the best results detecting happiness and anger.
There are some emotions that may be more similar and the model can confuse:
when the real emotion is contempt, it predicts anger 30% of the time; a
similar situation arises for surprise and happiness; disgust is predicted as
anger or contempt around 45% of the cases. Finally, fear is the worst
performance class.

Figure 2: Validation confusion matrix of the ensemble model.

We used the trained model corresponding to the WCE loss and XLS-R speech
feature extractor configuration with standard MHA to analyze the features
fusion in the MHA layer. To that end, each head attention weights of a
particular audio sample were calculated. For each head, every contextual
representation uses learned attention weights (matrix rows) to attend to each
of the speech and text features (matrix columns), as visualized in figures 3
and 4. Figure 3 shows that, for head 0, attention is focused both on some
speech and text features (specially in rows 20 to 30) to generate contextual
representations. On the other hand, in figure 4 we can see that, for head 1,
some contextual representations attend mostly to speech features and others
mostly to text features. This shows how our Multi-Head Attention Multimodal
Fusion creates complex contextual representations combining speech and text
information in different ways.

Figure 3: Head 0 attention weights visualization. The white vertical dashed
line divides speech features from text features. Figure 4: Head 1 attention
weights visualization. The white vertical dashed line divides speech features
from text features.

The results of the experiments using the sub-vector MHA approach are shown in
Table 2. Given that both the configurations with WCE Loss and HuBERT speech
feature extractor and with Focal Loss and XLS-R speech feature extractor
surpass the baseline validation score, we conclude that combining a powerful
self-supervised feature extractor with a sub-vector-based Double Multi-Head
Attention Multimodal Fusion component can obtain good results for this task.
Since the sub-vector MHA has significantly less parameters than the standard
MHA, this efficient architecture could be useful in settings where large
models are not allowed and/or training data is scarce, which is a common
scenario in SER.

##  6 Conclusion

In this paper we have described our Double Multi-Head Attention Multimodal
System for the Odyssey 2024 Speech Emotion Recognition Challenge. Acoustic and
text features were extracted using pre-trained self-supervised models. These
multimodal features are mixed adopting an early fusion strategy. First, an MHA
layer generates complementary contextualized representations. A second
attention layer is then applied to pool these representations into an
utterance-level vector. For the first attention layer, we experimented with
different mechanisms: standard MHA and sub-vector MHA. Since the sub-vector
variant has significantly less parameters than the standard one, our obtained
results show that this efficient architecture could be useful in settings
where large models are not allowed and/or training data is scarce. On the
other hand, applying standard MHA allows the model to capture complex
relationships jointly attending to information from different representation
subspaces. In terms of relative improvement, our best model gained 10.1%
validation Macro F1-score compared to the baseline. This model outperforms the
challenge official baseline with a 3.1% absolute increase in the validation
Macro F1-score, achieving the third position in the categorical task ranking,
where 31 teams participated in total.

##  7 Acknowledgments

This work has been promoted and financed by the Generalitat de Catalunya
through the Aina project and by the Spanish Ministerio de Ciencia e Innovación
through the project AdaVoice PID2019-107579RB-I00. The first author is
supported by a FI grant from the Catalan government.

## References

  * [1] Björn W Schuller,  “Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,”  Communications of the ACM, vol. 61, no. 5, pp. 90–99, 2018.
  * [2] Carlos Busso, Murtaza Bulut, Shrikanth Narayanan, J Gratch, and S Marsella,  “Toward effective automatic recognition systems of emotion in speech,”  Social emotions in nature and artifact: emotions in human and human-computer interaction, vol. 7, no. 17, pp. 110–127, 2013.
  * [3] Dimitrios Ververidis and Constantine Kotropoulos,  “Emotional speech recognition: Resources, features, and methods,”  Speech communication, vol. 48, no. 9, pp. 1162–1181, 2006.
  * [4] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean,  “Efficient estimation of word representations in vector space,”  arXiv preprint arXiv:1301.3781, 2013.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [6] L. Goncalves, A. N. Salman, A. Reddy Naini, L. Moro-Velazquez, T. Thebaud, L.P. Garcia, N. Dehak, B. Sisman, and C. Busso,  “Odyssey2024 - speech emotion recognition challenge: Dataset, baseline framework, and results,”  in Odyssey 2024: The Speaker and Language Recognition Workshop), Quebec, Canada, June 2024, vol. To appear.
  * [7] Reza Lotfian and Carlos Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2017.
  * [8] Miquel India, Pooyan Safari, and Javier Hernando,  “Self multi-head attention for speaker recognition,”  arXiv preprint arXiv:1906.09890, 2019.
  * [9] Miquel India, Pooyan Safari, and Javier Hernando,  “Double multi-head attention for speaker verification,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6144–6148.
  * [10] Federico Costa, Miquel Àngel India Massana, and Francisco Javier Hernando Pericás,  “Speaker characterization by means of attention pooling,”  in IberSPEECH 2022: Granada, Spain, 14-16 November 2022. International Speech Communication Association (ISCA), 2022, pp. 166–170.
  * [11] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever,  “Robust speech recognition via large-scale weak supervision,”  in International Conference on Machine Learning. PMLR, 2023, pp. 28492–28518.
  * [12] Sarala Padi, Seyed Omid Sadjadi, Dinesh Manocha, and Ram D Sriram,  “Multimodal emotion recognition using transfer learning from speaker recognition and bert-based models,”  arXiv preprint arXiv:2202.08974, 2022.
  * [13] Omar Mohamed and Salah A Aly,  “Arabic speech emotion recognition employing wav2vec2. 0 and hubert based on baved dataset,”  arXiv preprint arXiv:2110.04425, 2021.
  * [14] Soonil Kwon et al.,  “Att-net: Enhanced emotion recognition system using lightweight self-attention module,”  Applied Soft Computing, vol. 102, pp. 107101, 2021.
  * [15] Sanghyun Lee, David K Han, and Hanseok Ko,  “Fusion-convbert: parallel convolution and bert fusion for speech emotion recognition,”  Sensors, vol. 20, no. 22, pp. 6688, 2020.
  * [16] Yuanchao Li, Tianyu Zhao, Tatsuya Kawahara, et al.,  “Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning.,”  in Interspeech, 2019, pp. 2803–2807.
  * [17] Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman,  “Emotion recognition in speech using cross-modal transfer in the wild,”  in Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 292–301.
  * [18] Aggelina Chatziagapi, Georgios Paraskevopoulos, Dimitris Sgouropoulos, Georgios Pantazopoulos, Malvina Nikandrou, Theodoros Giannakopoulos, Athanasios Katsamanis, Alexandros Potamianos, and Shrikanth Narayanan,  “Data augmentation using gans for speech emotion recognition.,”  in Interspeech, 2019, pp. 171–175.
  * [19] Weicheng Cai, Jinkun Chen, Jun Zhang, and Ming Li,  “On-the-fly data loader and utterance-level aggregation for speaker and language recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1038–1051, 2020.
  * [20] Muhammad Mohsin Kabir, M. F. Mridha, Jungpil Shin, Israt Jahan, and Abu Quwsar Ohi,  “A survey of speaker recognition: Fundamental theories, recognition methods and opportunities,”  IEEE Access, vol. 9, pp. 79236–79263, 2021.
  * [21] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.
  * [23] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv preprint arXiv:2111.09296, 2021.
  * [24] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [25] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [26] Jeffrey Pennington, Richard Socher, and Christopher D Manning,  “Glove: Global vectors for word representation,”  in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.
  * [27] Wen Wu, Chao Zhang, and Philip C Woodland,  “Emotion recognition by fusing time synchronous and time asynchronous representations,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6269–6273.
  * [28] Leonardo Pepino, Pablo Riera, Luciana Ferrer, and Agustín Gravano,  “Fusion approaches for emotion recognition from speech using acoustic and text-based features,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6484–6488.
  * [29] Sarthak Yadav and Atul Rai,  “Learning discriminative features for speaker identification and verification.,”  in Interspeech, 2018, pp. 2237–2241.
  * [30] Hao Meng, Tianhao Yan, Fei Yuan, and Hongwei Wei,  “Speech emotion recognition from 3d log-mel spectrograms with deep learning network,”  IEEE access, vol. 7, pp. 125868–125881, 2019.
  * [31] Pooyan Safari, Miquel India, and Javier Hernando,  “Self attention networks in speaker recognition,”  Applied Sciences, vol. 13, no. 11, pp. 6410, 2023.
  * [32] Zhiming Wang, Kaisheng Yao, Xiaolong Li, and Shuo Fang,  “Multi-resolution multi-head attention in deep speaker embedding,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6464–6468.
  * [33] Yingke Zhu, Tom Ko, David Snyder, Brian Mak, and Daniel Povey,  “Self-attentive speaker embeddings for text-independent speaker verification.,”  in Interspeech, 2018, vol. 2018, pp. 3573–3577.
  * [34] Hailun Lian, Cheng Lu, Sunan Li, Yan Zhao, Chuangao Tang, and Yuan Zong,  “A survey of deep learning-based multimodal emotion recognition: Speech, text, and face,”  Entropy, vol. 25, no. 10, pp. 1440, 2023.
  * [35] Sharmeen M Saleem Abdullah Abdullah, Siddeeq Y Ameen Ameen, Mohammed AM Sadeeq, and Subhi Zeebaree,  “Multimodal emotion recognition using deep learning,”  Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 73–79, 2021.
  * [36] Shuai Liu, Peng Gao, Yating Li, Weina Fu, and Weiping Ding,  “Multi-modal fusion network with complementarity and importance for emotion recognition,”  Information Sciences, vol. 619, pp. 679–694, 2023.
  * [37] Zhaoyang Niu, Guoqiang Zhong, and Hui Yu,  “A review on the attention mechanism of deep learning,”  Neurocomputing, vol. 452, pp. 48–62, 2021.
  * [38] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “Superb: Speech processing universal performance benchmark,”  arXiv preprint arXiv:2105.01051, 2021.
  * [39] Hitoshi Yamamoto, Kong Aik Lee, Koji Okabe, and Takafumi Koshinaka,  “Speaker augmentation and bandwidth extension for deep speaker embedding.,”  in Interspeech, 2019, pp. 406–410.
  * [40] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5220–5224.
  * [41] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [42] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton,  “Layer normalization,”  arXiv preprint arXiv:1607.06450, 2016.
  * [43] Dan Hendrycks and Kevin Gimpel,  “Gaussian error linear units (gelus),”  arXiv preprint arXiv:1606.08415, 2016.
  * [44] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  arXiv preprint arXiv:1711.05101, 2017.
  * [45] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár,  “Focal loss for dense object detection,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988.