# Speech Emotion Recognition via an Attentive Time-Frequency Neural Network

Cheng Lu, Wenming Zheng†, Hailun Lian, Yuan Zong, Chuangao Tang, Sunan Li, and
Yan Zhao † Corresponding author.Cheng Lu, Hailun Lian, Sunan Li and Yan Zhao
are with the Key Laboratory of Child Development and Learning Science
(Ministry of Education), School of Information Science and Engineering,
Southeast University, Nanjing 210096, China. (e-mail:
cheng.lu@seu.edu.cn).Wenming Zheng, Yuan Zong and Chuangao Tang are with the
Key Laboratory of Child Development and Learning Science (Ministry of
Education), School of Biological Science and Medical Engineering, Southeast
University, Nanjing 210096, China. (e-mail: wenming_zheng,
xhzongyuan@seu.edu.cn).

###### Abstract

Spectrogram is commonly used as the input feature of deep neural networks to
learn the high(er)-level time-frequency pattern of speech signal for speech
emotion recognition (SER). Generally, different emotions correspond to
specific energy activations both within frequency bands and time frames on
spectrogram, which indicates the frequency and time domains are both essential
to represent the emotion for SER. However, recent spectrogram-based works
mainly focus on modeling the long-term dependency in time domain, leading to
these methods encountering the following two issues: (1) neglecting to model
the emotion-related correlations within frequency domain during the time-
frequency joint learning; (2) ignoring to capture the specific frequency bands
associated with emotions. To cope with the issues, we propose an attentive
time-frequency neural network (ATFNN) for SER, including a time-frequency
neural network (TFNN) and time-frequency attention. Specifically, aiming at
the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder)
based on the Transformer encoder and a time-domain encoder (T-Encoder) based
on the Bidirectional Long Short-Term Memory (Bi-LSTM). The F-Encoder and
T-Encoder model the correlations within frequency bands and time frames,
respectively, and they are embedded into a time-frequency joint learning
strategy to obtain the time-frequency patterns for speech emotions. Moreover,
to handle the second issue, we also adopt time-frequency attention with a
frequency-attention network (F-Attention) and a time-attention network
(T-Attention) to focus on the emotion-related frequency band ranges and time
frame ranges, which can enhance the discriminability of speech emotion
features. Extensive experimental results on three public emotional databases,
i. e., IEMOCAP, ABC, and CASIA, show that the proposed ATFNN outperforms the
state-of-the-art methods.

###### Index Terms:

speech emotion recognition, time-frequency neural network, attention
mechanism, spectrogram.

##  I Introduction

Speech emotion recognition (SER) task is to make the machine automatically
recognize the emotional state from the human speech. Recently, it has been a
research hotspot in affective computing and pattern recognition [1], [2], [3],
[4], [5]. Generally speaking, the key to deal with the SER problem is to
extract the discriminative and generalized features of emotional speech [6],
[7], [8], [9], [10]. These features mainly contain two categories at present:
the hand-crafted features and the deep features.

The hand-crafted features are primarily adopted in the earlier SER works,
known as the low-level descriptors (LLDs) [11], [12], e. g., Frame
Energy/Loudness, Fundamental Frequency, Zero/Mean-Crossing Rate, Mel-
Frequency-Cepstral Coefficients (MFCC), etc. A mount of related works explored
various LLDs or these combinations to improve the performance of the SER
model. For instance, Schuller et al. [11] integrated 6 552 hand-crafted
features into an open-source toolkit, i. e., openEAR, for four affective
recognition tasks and then extended the openEAR to a multi-functional and
convenient toolkit for diverse speech-related tasks, i .e., openSMILE [12]. In
the earlier works, these LLDs and their combinations were widely utilized as
the benchmark features for SER [6], [7].

Recently, with the emergence of deep learning, handcrafted features are
gradually replaced by deep features extracted by Deep Neural Networks (DNNs),
e. g. Deep Convolutional Neural Networks (DCNNs) [8], [9], Recurrent Neural
Networks (RNNs) [13], [14], [15]. Instead of LLDs, these deep features are
high(er)-level features. In the extraction of deep features, generally,
spectrogram features (e. g., MFCC, magnitude spectrogram, and Mel-spectrogram)
with rich time-frequency information, are often used as input features of DNNs
to learn the high(er)-level time-frequency patterns of speech signal for SER
[7], [8], [9], [13]. Mao et al. [8] investigated to utilize the
dimensionality-reduced magnitude spectrogram to learn the salient features by
CNNs. Zhang et al. [9] extended the spectrogram to a 3-D Mel-Spectrogram to
learn utterance-level speech emotion features from segment features by a
discriminant temporal pyramid matching strategy. Wang et al. [13] combined the
MFCC features and Mel-spectrogram to input a special Long Short-Term Memory
(LSTM), called Dual-Sequence LSTM, to extract the utterance-level emotion
features.

Furthermore, a time-frequency joint learning strategy is increasingly
integrated into DNNs to extract more effective time-frequency representations
of speech [16], [17], [18], [19]. Satt et al. [18] proposed a combined
convolution-LSTM network to extract the utterance-level feature with segment-
level features. Chen et al. [19] also adopt the 3-D attention-based
convolutional recurrent neural network (3-D ACRNN) to represent the utterance
level affective-salient features of emotional speech.

Nevertheless, the related works of the time-frequency feature extraction using
spectrograms commonly encounter two issues at present. The first issue is that
the existing works mainly focus on the correlations between speech frames to
characterize the long-term dependency in time domain [8], [13], [17]. There
are also some works, however, which have revealed that the energy of the
frequency band is changing in different emotions [20], [21], [22]. In [21],
Cowie et al. reported the variations of acoustic parameters associated with
emotions, and the results of the acoustic spectral parameters indicated that
each emotion usually corresponds to some energy activations of specific
frequency band ranges on the spectrogram. For instance, the investigation in
[21] reveals that happiness has increased in high-frequency energy, while
sadness has decreased in high-frequency energy. These situations indicate that
capturing the energy variations in different frequency ranges is also
essential for the representation of speech emotions. Therefore, this paper
will jointly model the correlations both within the frequency and time domains
to obtain robust time-frequency representations of emotions.

Figure 1: Overview of the Attentive Tim-Frequency Neural Network (ATFNN)
framework for SER, including a time-frequency neural network (TFNN) and time-
frequency attention. The TFNN contains a frequency domain encoder (F-Encoder)
and time domain encoder (T-Encoder). The time-frequency attention includes
frequency attention (F-Attention) and time attention (F-Attention). In ATFNN,
the log-Mel-spectrogram $\boldsymbol{x}$ is firstly encoded as the enhanced
frequency feature $\boldsymbol{x}^{\prime}$ and the enhanced frequency
encoding $\hat{\boldsymbol{x}}^{\prime}$ by F-Attention and F-Encoder (i. e.,
the encoder of Transformer), respectively. Then
$\hat{\boldsymbol{x}}^{\prime}$ is transformed to the enhanced time feature
$\hat{\boldsymbol{x}}^{\prime\prime}$ by T-Attention. Finally,
$\hat{\boldsymbol{x}}^{\prime\prime}$ is as the input of T-Encoder (i. e., Bi-
LSTM) to generate the enhanced time encoding $\boldsymbol{h}^{\prime}$ (i. e.,
time-frequency encdoing) for the emotion classification.

Another issue faced by the time-frequency representation of speech emotion is
that the distribution of emotion information in an utterance is sparse in both
frequency domain and time domain [19], [23]. Specifically, it is intuitive
that there are some non-speech frames without contents in a long utterance.
Thus, not all frames in a sentence contain emotional information. In other
words, the frames of an utterance have different contribution degrees in the
representation of emotional speech, i. e., there are key frame regions highly
associated with emotions. For the purpose of capturing these emotion-related
regions, Chen et al. [19] investigated adding an attention layer after CNN and
LSTM to capture specific temporal frames for the speech emotion
representation. Also, Zhang et al. [23] adopted an attention mechanism to
empower the sub-layer to focus on emotion salient regions of the spectrogram.

However, these studies mainly embed attention into high-level time-frequency
features, leading to introduce the redundant information in the feature
extraction and roughly locate the emotion-related regions in the time-
frequency domain. In addition, they mainly focus on the selection on the time
frames while ignoring the selection on the frequency bands. Actually, some
studies have proven that each emotion does not have an energy activation on
all frequency bands in the frequency domain [21], [22], [20]. Therefore, we
need to capture the key time frames in the time domain, and the key frequency
bands in the frequency domain, which are all highly related to emotions.

To deal with the issues mentioned above, in this paper, we propose an
Attentive Time-Frequency Neural Network (ATFNN) to learn the discriminative
speech emotion feature for SER, as shown in Figure.1. In detail, we propose a
time-frequency neural network (TFNN), including a frequency-domain encoder
(F-Encoder) to model frequency features and a time-domain encoder (T-Encoder)
to model time features, to capture the correlations within the frequency
domain and within the time domain. Moreover, an attention mechanism with a
frequency attention network (F-Attention) and a time attention network
(T-Attention) is also embedded into TFNN to focus on the specific frequency
bands and time frames highly corresponding to emotions.

In summary, the contributions of the paper mainly include the following three
points:

  1. 1.

We propose a TFNN to jointly model the correlations both in the time and
frequency domains, capturing more emotion information in the time-frequency
representation of the speech signal.

  2. 2.

We propose a time-frequency attention mechanism, measuring the contribution
degree of frequency bands and time frames associated with different emotions,
to capture the critical frequency bands and time frames.

  3. 3.

By visualizing the attentions in the frequency and time domains, we
demonstrate the correlations both within frequency bands and within time
frames under different emotions on three public speech emotion databases, i.
e., IEMOCAP, ABC, and CASIA.

The rest of the paper is organized as follows. The proposed ATFNN method is
described in detail in Section II. Section III describes the experimental
settings and analyzes the experimental results. In the end, Section IV
concludes the paper and discusses some future works.

##  II ATFNN for SER

Basically, the ATFNN, shown in Figure.1, is to embed the time-frequency
attention strategy (i. e., F-Attention and T-Attention) into the TFNN. Thus,
in this section, we firstly introduce the TFNN, then describe the ATFNN model
in detail.

###  II-A TFNN

The TFNN aims to learn the high(er)-level time-frequency representation from
the input spectrogram feature for SER, which includes two modules, i. e.,
F-Encoder and T-Encoder.

####  II-A1 F-Encoder

The role of F-Encoder is to characterize the input spectrogram from the
frequency domain, including the encoding of frequency information and the
correlation modeling within frequency bands. Since different emotions
correspond to the activations of specific frequency bands [21], the modeling
in frequency domain can ignore the order relations between frequency bands. In
the Transformer [24], the self-attention is adopted to calculate the
correlations between tokens in Natural Language Processing (NLP). Inspired by
this point, we utilize self-attention to aggregate the correlations across
frequency bands into the frequency domain encoding. Thus, we propose an
F-Encoder based on the revised encoder of the Transformer to learn the
frequency domain representation of emotional speech, shown in Figure. 2, in
which the position encoding module is abandoned to ignore the order
relationship modeling between frequency bands. The proposed F-Encoder is
introduced in detail below.

Given the input log-Mel-spectrogram feature of emotional speech as
$\boldsymbol{x}\in\mathbb{R}^{f\times t\times 1}$, where $t$ corresponds the
frame number and $f$ is the number of frequency bands onto the Mel-scale, the
F-Encoder $F_{e}(\cdot)$ transforms the input $\boldsymbol{x}$ to the
frequency domain encoding $\boldsymbol{\hat{x}}\in\mathbb{R}^{f\times t\times
c}$, represented as $\boldsymbol{\hat{x}}=F_{e}(\boldsymbol{x})$.
Specifically, the input $\boldsymbol{x}$ is firstly extended the channel
information to $\bar{\boldsymbol{x}}\in\mathbb{R}^{f\times t\times c}$ by a
linear layer. Then, the multi-head self-attention is utilized to calculate the
correlations within frequency bands into frequency features
$\bar{\boldsymbol{x}}$. After a series of convolution (with the kernel size is
$1\times 5$ and the channel number is 8) and add-norm operations (i. e.,
residual connection operations followed by layer normalization [24]), the
frequency domain encoding $\boldsymbol{\hat{x}}$ is obtained by integrating
the learning of correlations within frequency bands into the time-frequency
representation. In addition, it is noting that, $f$, $t$, and $c$ in the paper
are set to $80$, $128$, and $8$, respectively.

####  II-A2 T-Encoder

The T-Encoder needs to capture the long-term dependence in time frames of
speech for the representation of emotional information. RNNs, especially LSTM,
have demonstrated their powerful capabilities in modeling the long-range
dependence of the time series signal, and have been widely used in speech
signal processing [13], [14], [15]. Therefore, we adopt a T-Encoder based on a
revised Bidirectional LSTM (Bi-LSTM) [13], [15].

The proposed T-Encoder $T_{e}(\cdot)$ takes the $i^{th}$ frame
$\boldsymbol{\hat{x}}_{i}\in\mathbb{R}^{f\times c}$ of the frequency encoding
$\boldsymbol{\hat{x}}$ as the input of each time step in Bi-LSTM to learn the
time-frequency representation $\boldsymbol{h}$ of emotional speech. In detail,
the time domain encoding process can be defined as
$\boldsymbol{h}_{i}=T_{e}(\boldsymbol{\hat{x}}_{i})$, where $i\in[1,2,...,t]$
represents the frame index,
$\boldsymbol{\hat{x}}=\\{\boldsymbol{\hat{x}}_{i}\\}_{i=1}^{t}$, and
$\boldsymbol{h}_{i}$ is the $i^{th}$ time-step output of the T-Encoder.
Notably, since the T-Encoder is based on three-layer Bi-LSTM with 128 hidden
nodes in the paper, the final time-frequency feature
$\boldsymbol{h}\in\mathbb{R}^{1\times 256}$ is obtained by concatenating the
last hidden states in the forward direction
$\overrightarrow{\boldsymbol{h}_{t}}\in\mathbb{R}^{1\times 128}$ and the last
hidden states in reversed direction
$\overleftarrow{\boldsymbol{h}_{t}}\in\mathbb{R}^{1\times 128}$, which can be
represented as
$\boldsymbol{h}=[\overrightarrow{\boldsymbol{h}_{t}},\overleftarrow{\boldsymbol{h}_{t}}]$.

####  II-A3 TFNN

The TFNN integrates F-Encoder and T-Encoder into time-frequency joint learning
strategy to ensure that the emotion patterns of speech are captured in both
frequency domain and time domain. Specifically, the input log-Mel-spectrogram
feature $\boldsymbol{x}$ is firstly encoded in frequency bands by F-Encoder to
obtain frequency domain encoding $\boldsymbol{\hat{x}}$, represented as
$\boldsymbol{\hat{x}}=F_{e}(\boldsymbol{x})$. Then, $\boldsymbol{\hat{x}}$ is
used as the input of T-Encoder to capture the long-term dependence between
time frames to obtain the time domain encoding $\boldsymbol{h}$ through a
time-frequency joint learning strategy, which can be formalized as
$\boldsymbol{h}=T_{e}(F_{e}(\boldsymbol{x}))$.

Figure 2: Architecture of the F-Encoder, where the input $\boldsymbol{x}$ is
firstly transformed to $\boldsymbol{x}^{\prime}$ by a linear layer. Then,
after the multi-head self-attention and two convolution layers (with the
kernel size is $1\times 5$ and the channel number are 32 and 8) and two add-
norm layers, the correlations within frequency bands are integrated into the
frequency domain encoding $\boldsymbol{\hat{x}}$.

###  II-B ATFNN

In the ATFNN, two attention networks, i. e., F-Attention and T-Attention, are
employed in the TFNN to focus on the emotion-related regions in the frequency
and time domains.

####  II-B1 F-Attention

F-Attention aims to capture salient frequency bands contributing to different
emotions, which is implemented by a convolution based block with the kernel
size $(1\times 5\times c)$, shown in Figure. 3. To calculate the frequency
attention, we firstly represent the input log-Mel-spectrogram
$\boldsymbol{x}=\\{\boldsymbol{x}_{i}\\}_{i=1}^{t}$ as a set of
$\boldsymbol{x}_{i}\in\mathbb{R}^{f\times c}$, where $i\in[1,2,...,t]$ is the
frame index and $\boldsymbol{x}_{i}$ represents the frequency domain feature
of $i^{th}$ frame on $\boldsymbol{x}$. Since the frequency bands related to
emotions are within a certain range, e. g., high frequency, intermediate
frequency, and low frequency [18], [21], we employ 5 frequency bands of
$\boldsymbol{x}_{i}$ as a band group for the convolution to preserve the
correlations within frequency bands. Thus, after the convolution operation,
the frequency attention weights
$\boldsymbol{a}^{f}_{i}\in\mathbb{R}^{1\times(f/5)\times c}$ of different
frequency band groups can be obtained, expressed as
$\boldsymbol{a}^{f}_{i}=F_{a}(\boldsymbol{x}_{i})$. Moreover, as shown in
Figure. 3, we utilize the frequency attention $\boldsymbol{a}^{f}_{i}$ to
perform a weighted average operation on $\boldsymbol{x}_{i}$ with respect to
the channel $c$ to produce the attention-based frequency features, then
integrate these frequency features into the origin input features
$\boldsymbol{x}_{i}$ by the element-wise addition operation to produce the
enhanced frequency feature $\boldsymbol{x}_{i}^{\prime}$, which can be
represented as

$\displaystyle\boldsymbol{x}_{i}^{\prime}$
$\displaystyle=\boldsymbol{x}_{i}\oplus(F_{a}(\boldsymbol{x}_{i})\otimes\boldsymbol{x}_{i})$
(1)
$\displaystyle=\boldsymbol{x}_{i}\oplus\left(\frac{1}{c}\sum_{m=1}^{c}{\boldsymbol{a}^{f}_{i,m}\otimes\boldsymbol{x}_{i}}\right),$

where $\boldsymbol{a}^{f}_{i,m}\in\mathbb{R}^{1\times(f/5)}$ represents the
attention weight of the $m^{th}$ channel on the frequency attention weight
$\boldsymbol{a}^{f}_{i}$, and $m\in[1,2,...,c]$. Moreover, $\otimes$ and
$\oplus$ represent the broadcasting element-wise multiplication and element-
wise addition, respectively. It is noting that, in $\otimes$ of F-Attention,
each 5 frequency bands of $\boldsymbol{x}_{i}$ share the same attention
weights for duplication.

After that, we embeds the F-Attention into the F-Encoder, called as the
attentive frequency neural network (AFNN), to produce the enhanced frequency
encoding $\boldsymbol{\hat{x}}^{\prime}\in\mathbb{R}^{f\times t\times c}$. The
AFNN can be defined as
$\boldsymbol{\hat{x}}_{i}^{\prime}=F_{e}(\boldsymbol{x}_{i}^{\prime})$, where
$\boldsymbol{\hat{x}}^{\prime}_{i}\in\mathbb{R}^{f\times c}$ and the enhanced
frequency encoding $\boldsymbol{\hat{x}}^{\prime}\in\mathbb{R}^{f\times
t\times c}$ is a set of $\boldsymbol{\hat{x}}^{\prime}_{i}$, written as
$\boldsymbol{\hat{x}}^{\prime}=\\{\boldsymbol{\hat{x}}^{\prime}_{i}\\}_{i=1}^{t}$.

Figure 3: Pipeline of the F-Attention in ATFNN, in which the log-Mel-
spectrogram $\boldsymbol{x}$ is adopted to calculate the frequency attention
weight $\boldsymbol{a}^{f}_{i}$, then integrate the attention-based frequency
feature into $\boldsymbol{x}$ to generate the enhanced frequency feature
$\boldsymbol{x}^{\prime}$. $\otimes$ and $\oplus$ are the same operations in
Figure. 1. Figure 4: Pipeline of the T-Attention in ATFNN, where we also use
the log-Mel-spectrogram $\boldsymbol{x}$ to calculate the time attention
weight $\boldsymbol{a}^{t}_{j}$, then integrate the attention-based time
feature into the enhanced frequency encoding $\boldsymbol{\hat{x}}^{\prime}$
to generate the enhanced time encoding $\boldsymbol{\hat{x}}^{\prime\prime}$
for the final speech emotion classification. $\otimes$ and $\oplus$ are the
same operations in Figure. 1.

####  II-B2 T-Attention

Similar to F-Attention, the T-Attention is also performed by a convolution
based block with the kernel size $(1\times 8\times c)$ to capture salient
frames that contribute to emotions, shown in Figure. 4. To calculate the time
domain attention, we firstly divide the input lo-Mel-spectrogram
$\boldsymbol{x}$ into multiple $\boldsymbol{x}_{j}$ in frequency domain, where
$j\in[1,2,...,f]$ represents the $j^{th}$ frequency band of the input
$\boldsymbol{x}$ and $\boldsymbol{x}_{j}\in\mathbb{R}^{t\times c}$ is the time
domain feature of the $j^{th}$ frequency band on $\boldsymbol{x}$, i. e,
$\boldsymbol{x}=\\{\boldsymbol{x}_{j}\\}_{j=1}^{f}$. In T-Attention, 8 time
frames of $\boldsymbol{x}_{j}$ are employed as a frame group for the
convolution to obtain the time attention weight
$\boldsymbol{a}_{j}^{t}\in\mathbb{R}^{1\times(t/8)\times c}$, expressed as
$\boldsymbol{a}_{j}^{t}=T_{a}(\boldsymbol{x}_{j})$. Then, as shown in Figure.
4, we also employ the time attention $\boldsymbol{a}^{t}_{j}$ to perform a
weighted average operation on $\boldsymbol{x}_{j}$ with respect to the channel
$c$ to generate the attention-based time features, and integrate them into the
enhance frequency encoding $\boldsymbol{\hat{x}}_{j}$ by the element-wise
addition operation to obtain the enhanced time feature
$\boldsymbol{\hat{x}}_{j}^{\prime\prime}\in\mathbb{R}^{t\times c}$, which can
be written as

$\displaystyle\boldsymbol{\hat{x}}^{\prime\prime}_{j}$
$\displaystyle=\boldsymbol{\hat{x}}^{\prime}_{j}\oplus(T_{a}(\boldsymbol{x}_{j})\otimes\boldsymbol{\hat{x}}^{\prime}_{j})$
(2)
$\displaystyle=\boldsymbol{\hat{x}}^{\prime}_{j}\oplus\left(\frac{1}{c}\sum_{n=1}^{c}{\boldsymbol{a}^{t}_{j,n}\otimes\boldsymbol{\hat{x}}^{\prime}_{j}}\right),$

where $\boldsymbol{a}^{t}_{j,n}\in\mathbb{R}^{1\times(t/8)}$ represents the
attention weight of the $n^{th}$ channel on the time attention weight
$\boldsymbol{a}^{t}_{j}$, $n\in[1,2,...,c]$, and the enhanced time feature
$\boldsymbol{\hat{x}}^{\prime\prime}\in\mathbb{R}^{f\times t\times c}$ is a
set of $\boldsymbol{\hat{x}}^{\prime\prime}_{j}$, i .e.,
$\boldsymbol{\hat{x}}^{\prime\prime}=\\{\boldsymbol{\hat{x}}^{\prime\prime}_{j}\\}_{j=1}^{f}$.
In $\otimes$ of T-Attention, each 8 time frames of
$\boldsymbol{\hat{x}}^{\prime}_{j}$ share the same attention weights for
duplication. Since the T-Encoder is encoded according to the time frame, the
ATFNN can also be expressed as
$\boldsymbol{h}^{\prime}_{i}=T_{e}(\boldsymbol{\hat{x}}^{\prime\prime}_{i})$,
where $\boldsymbol{h}^{\prime}=\\{\boldsymbol{h}^{\prime}_{i}\\}_{i=1}^{t}$ is
the enhance time encoding.

####  II-B3 ATFNN

Integrating F-Attention and T-Attention into F-Encoder and T-Encoder
respectively, we extend TFNN to attention-based TFNN, i. e., ATFNN. With
F-Attention, the ATFNN can fucus on the key frequency bands related to
emotions to generate the enhanced frequency domain coding
$\boldsymbol{\hat{x}}^{\prime}$, expressed as

$\boldsymbol{\hat{x}}^{\prime}=F_{e}(\boldsymbol{x}\oplus
F_{a}(\boldsymbol{x})).$ (3)

Then, the T-Attention is adopt to capture key time frames related to emotions
to obtain a more discriminative time-frequency representation
$\boldsymbol{h}^{\prime}_{t}$, which can be formalized as

$\displaystyle\boldsymbol{h}^{\prime}_{t}$
$\displaystyle=T_{e}(\boldsymbol{\hat{x}}^{\prime}\oplus
T_{a}(\boldsymbol{\hat{x}}^{\prime}))$ (4)
$\displaystyle=T_{e}(F_{e}(\boldsymbol{x}\oplus F_{a}(\boldsymbol{x}))\oplus
T_{a}(F_{e}(\boldsymbol{x}\oplus F_{a}(\boldsymbol{x})))).$

Finally, the final time-domain encoding $\boldsymbol{h}^{\prime}$ can be
regarded as an enhanced time-frequency representation to input into the
classifier for SER. Actually, the ATFNN can be regarded as the TFNN that
embeds time-frequency attention into the process of time-frequency joint
learning.

##  III Experiments

In this section, we describe the used speech emotion databases, then analyze
and discuss the experimental results on our proposed ATFNN.

###  III-A Experimental Databases

Three public emotional speech databases, i. e., the Interactive Emotional
Dyadic Motion Capture database (IEMOCAP) [25], the Airplane Behaviour Corpus
(ABC) [26], and China Emotional Database (CASIA) [27], [28], are adopted in
our experiments to prove the effectiveness of the proposed ATFNN method.

IEMOCAP is a multimodal database with video, speech, and text scripts,
collected by the Speech Analysis and Interpretation Laboratory (SAIL) at the
University of Southern California (USC). It is recorded in dyadic sessions
where 10 actors (5 females and 5 males) perform improvised or scripted
scenarios to elicit several emotional expressions, i.e., _angry_ , _happy_ ,
_sad_ , _neutral_ , _frustrated_ , _excited_ , _fearful_ , _surprised_ ,
_disgusted_ , and _others_. In the paper, the improvised sentences with 4
emotions (i.e., _angry_ , _happy_ , _sad_ , and _neutral_) are all used to
perform the experiments according to [18], which has 2280 speech samples.

ABC is an audiovisual emotion database with the German collected for the
particular target application of public transport surveillance. Also, it has
430 speech samples generated by 8 subjects (4 females and 4 males) in gender
balance. They were induced to express one of 6 emotions, i.e., _aggressive_ ,
_cheerful_ , _intoxicated_ , _nervous, neutral_ , and _tired_ , by the given
scripts.

CASIA is a Chinese speech emotion database released by the Institute of
Automation of Chinese Academy of Sciences. It totally consists of 9,600 wave
files with 6 emotional states, i. e., _angry, fear, happy, neutral, sad_ , and
_surprise_. Notably, in the paper, we adopt 1200 utterances of ABC released
publicly for the experiment section. 4 volunteers with 2 males and 2 females
are required to simulate these 6 emotions to produce 300 utterances for each
emotion. To describe the selected databases clearly, we also list the detailed
parameters of three databases in Table. I.

TABLE I: Description of the selected speech emotion databases, i. e., IEMOCAP,
ABC, and CASIA, where ’SR’ represents the sampling rate.

Database | Language | #Emotions | #Speaker | #Utterance | SR(Hz)
---|---|---|---|---|---
IEMOCAP | English |  _angry_(289), _happy_(284), _neutral_(1099), _sad_(608) | 10 (5 males) | 2280 | 16000
ABC | German |  _aggressive_(95), _cheerful_(105), _intoxicate_(33), _nervous_(93), _neutral_(79), _tired_(25) | 8 (4 males) | 430 | 16000
CASIA | Chinese |  _angry_(200), _fear_(200), _happy_(200), _neutral_(200), _sad_(200), _surprise_(200) | 4 (2 males) | 1200 | 16000

  * •

#Emotions denotes the sample distribution of emotions; #Speaker denotes the
number of speakers used in the database; #Utterance denotes the number of
utterance samples used in the database; ’SR’ represents the sampling rate.

###  III-B Experimental Protocol

To evaluate the performance of the proposed method, we adopt Leave-One-
Speaker-Out (LOSO) cross-validation protocol for three selected databases
according to [6], [7]. Specifically, as shown in Table. I, the IEMOCAP, ABC,
and CASIA databases consist of 10 speakers, 8 speakers, and 4 speakers,
respectively. Therefore, in our experiments, the speech samples of one speaker
are utilized as the testing data, while the samples of other speakers are used
as the training data. It is noting that, since the IEMOCAP includes 5 sessions
on 10 speakers, the leave-one-session-out strategy is widely adopt in IEMOCAP
database according to [18], where four sessions (8 speakers) are for training,
and one session (2 speakers) is for testing. Therefore, we also select several
state-of-the-art methods based on this protocol for comparison [18], [29],
[23], [30], shown in Figure. II.

Besides, we also utilize two evaluation metrics [6], [7], i. e., the weighted
average recall (WAR) and the unweighted average recall (UAR), to effectively
measure the performance of the proposed method, which are commonly adopted in
SER task. WAR is known as the ’normal’ recognition actuary, while UAR reflects
the class-wise recognition accuracy defined as the recall per class divided by
the number of classes. Because the selected IEMOCAP and ABC databases are both
class-imbalanced in our experiments, the WAR and UAR can better measure the
performance of comparison methods.

###  III-C Experimental Setting

To perform experiments conveniently, we make the preprocessing for the speech
samples. In detail, following the works in [9], [8], [18], speech utterances
are all divided into small segments with 128 frames (i. e., 20ms), which can
not only preserve the completeness of speech emotion but also augment the
dataset. Then, the log-Mel-spectrogram is extracted by the Short-Time Fourier
Transform (STFT), in which 20 ms Hamming window size with 50% frame
overlapping is adopted and 512-point FFT is used on each frame. Besides, the
number of Mel-filter bands is set as 80.

The proposed ATFNN is implemented by the deep learning framework of Pytorch
with NVIDIA GeForce RTX3090 GPUs, which is trained from scratch with the batch
size of 128 and optimized by the Adam Optimizer with the initialized learning
rate of 0.0005.

###  III-D Results and Analysis

####  III-D1 Results on IEMOCAP

To compare the proposed ATFNN model with other methods, we select several
state-of-the-art works on the IEMOCAP database, i. e., DNN-HMM using an
alignment generated from the subspace based Gaussian Mixture Model based HMMs
(DNN-HMM_SGMM-Ali.) [31], 3 convolution layers and LSTM with 10Hz grid
resolution (CNN$+$LSTM Model) [18], GRU layer upon CNN layers with sequential
Capsules (CNN_GRU-SeqCap) [29], attention based fully convolutional network
(FCN$+$Attention Model) [23], fusion model of acoustic and linguistic features
(Model-3_fusion) [30], and an adaptive domain-aware representation learning
based model (ADARL) [32].

The experimental results on the IEMOCAP database are shown in the Table. II.
From these results, it is obvious that our proposed ATFNN achieves the state-
of-the-art performance. Specifically, our ATFNN obtains the best result on WAR
(73.81$\%$) than all comparison methods, and it also obtains suboptimal
results on UAR (64.48$\%$) than ADARL (65.86$\%$) [32]. It is worth noting
that, since ADARL adds a domain adaptation strategy to eliminate the
discrepancy between speakers of the training data and the testing data, it
obtains better performance than ATFNN on UAR. Nevertheless, ADARL depends on
the hypothesis that the distribution between training and testing data is
within a certain upper error bound, such that its universality is worse than
our ATFNN.

Besides, the confusion matrix of ATFNN reported in Figure. 5(a) also
demonstrates the performance of our method in detail, which reveals that ATFNN
achieved high recognition accuracies on three emotions, i. e., _angry_ ,
_neutral_ , and _sad_ , while obtains poor performance on _happy_. It is
because IEMOCAP is an extremely class imbalance database shown in Table. I,
where _happy_ has the smallest number of samples while _neutral_ has the
largest number of samples. Therefore, this situation may affect the trained
classifier to make it easily recognize _happy_ as _neutral_.

TABLE II: Experimental results with WAR and UAR on the three public speech
emotion databases (i. e., IEMOCAP, ABC, and CASIA), where the best results are
highlight in bold.

Database | Experimental Protocol |  Comparison Method |  Accuracy(%)
---|---|---|---
WAR | UAR
IEMOCAP |  Leave One Session/Speaker Out (LOSO) (5 Sessions or 10 Speakers) | DNN-HMM_SGMM-Ali. [31] | 62.28 | 58.02
CNN$+$LSTM Model [18]∗ | 68.80 | 59.40
CNN_GRU-SeqCap [29]∗ | 72.73 | 59.71
FCN$+$Attention Model [23]∗ | 70.40 | 63.90
Model-3_Fusion [30]∗ | 72.34 | 58.31
ADARL [32] | 73.02 | 65.86
ATFNN (ours) | 73.81 | 64.48
ABC |  Leave One Speaker Out (LOSO) (8 Speakers) | LLDs$+$HMM/GMM [6] | 57.70 | 48.80
RGSR-LPDA [33]∗∗ | N/A | 49.40
LLDs$+$SVM [6] | 61.40 | 55.50
GerDA [7] | 61.50 | 56.10
ATFNN (ours) | 68.84 | 57.57
CASIA |  Leave One Speaker Out (LOSO) (4 Speakers) | LLD$+$DR [34] | 39.50 | 39.50
DNN$+$ELM [35] | 41.17 | 41.17
HuWSF [36] | 43.50 | 43.50
DTPM [9] | 45.42 | 45.42
ATFNN (ours) | 48.75 | 48.75

  * •

∗ denotes that the used experiment protocol is leave-one-session-out, i. e.,
four sessions (8 speakers) are for training and one session (2 speakers) is
for testing, according to [18]; ∗∗ denotes that the samples of 4 speakers are
for training and 2 speakers are for testing, according to [33]; ’N/A’
indicates the results are not reported in the corresponding paper.

(a) Confusion matrix of ATFNN on IEMOCAP

(b) Confusion matrix of ATFNN on ABC

(c) Confusion matrix of ATFNN on CASIA

Figure 5: Confusion matrices of ATFNN on three selected speech emotion
databases, i. e., IEMOCAP, ABC, and CASIA.

####  III-D2 Results on ABC

For the comparison purpose, we also select several public works on the ABC
database, i. e., LLDs with HMM/GMM (LLDs$+$HMM/GMM) [6], random anchor points
generalized spectral regression with locally penalized discriminant analysis
(RGSR-LPDA) [33], LLDs with SVM (LLDs$+$SVM) [6], and generalized discriminant
analysis (GerDA) based on DNNs [7].

The results on the ABC database, as shown in Table. II, also demonstrate that
our proposed ATFNN is superior both on WAR (68.84$\%$) and UAR (57.57$\%$) to
other comparison methods not only on traditional methods (i. e.,
LLDs$+$HMM/GMM, RGSR-LPDA, and LLDs with SVM), but also on deep learning based
methods (i. e., GerDA), which proves that the ATFNN can extract more
discriminative and instance-adaptive representation of speech emotion on the
ABC database. Moreover, we also report the confusion matrix on the ABC
database, shown in Figure. 5(b). It is clear that the ATFNN have high
recognition accuracies on four emotions, i. e., _aggressive_ , _cheerful_ ,
_nervous_ , and _neutral_ , while have poor performance on two emotions, i.
e., _intoxicate_ and _tired_. Specifically, on the ABC, the recognition of
_intoxicate_ is easily confusing with _cheerful_ and _neutral_ , whereas
_tired_ is more recognized as _nervous_ and _neutral_. The cause of these
results on ABC is similar to reasons for IEMOCAP, that is, the numbers of
speech samples on _intoxicate_ and _tired_ are obviously smaller than other
emotions, leading to influence the trained classifier. In addition, another
reason may be that the arousal and valance of _intoxicate_ and _cheerful_ are
all close, resulting in the confusing recognition.

TABLE III: Ablation study of different architectures for ATFNN on the three public speech emotion databases (i. e., IEMOCAP, ABC, and CASIA), where the best results are highlight in bold. Moreover, ’✓’ or ’✗’ represents the network with or without the corresponding architecture. Network | Ablation Architecture |  IEMOCAP(%) |  ABC(%) |  CASIA(%)
---|---|---|---|---
Attention | F-Encoder | T-Encoder |  WAR |  UAR |  WAR |  UAR |  WAR |  UAR
AFNN | ✓ | ✓ | ✗ | 70.71 | 59.10 | 58.14 | 48.22 | 32.50 | 32.50
ATNN | ✓ | ✗ | ✓ | 71.76 | 61.90 | 66.05 | 55.01 | 46.42 | 46.42
TFNN | ✗ | ✓ | ✓ | 72.99 | 63.60 | 67.21 | 55.99 | 46.67 | 46.67
ATFNN | ✓ | ✓ | ✓ | 73.81 | 64.48 | 68.84 | 57.57 | 48.75 | 48.75

####  III-D3 Results on CASIA

For the fair comparison on the CASIA, four sate-of-the-art methods are chosen
in the experiments, including LLDs with a dimension reduction combining PCA
and LDA (LLD$+$DR) [34], DNNs with the extreme learning machine (DNN_ELM)
[35], weighted spectral feature learning based on local Hu moment (HuWSF)
[36], and deep convolutional neural network with discriminant temporal pyramid
matching (DTPM) [9].

The experimental results on the CASIA are reported in the Table. II, which
reveal that our proposed ATFNN achieves the highest recognition accuracies
than other comparison methods in terms of both WAR (48.75$\%$) and UAR
(48.75$\%$). In detail, deep learning based methods, i. e., DNN_ELM and DTPM,
outperform the LLDs based traditional method (LLD$+$DR), while our ATFNN is
significantly superior than the current sate-of-the-art methods. It is noted
that since the number of samples used in each emotion of the CASIA in the
paper is equal, the WAR in the experimental results is equal to UAR.
Furthermore, the confusion matrix of ATFNN on the CASIA is also calculated to
verify the detailed performance of our proposed method, which is given in
Figure. 5(c). From these results, we can observe that our ATFNN achieve the
high performance on four emotions, i. e., _angry_ , _happy_ , _neutral_ , and
_sad_ , whereas has a certain confusion in the recognition of other two
emotions, i. e., _fear_ and _surprise_. Specifically, _fear_ is easier to
confuse with _sad_ , while _surprise_ is always confused with _angry_ and
_happy_. The reason may be that _fear_ and _sad_ are relatively close in
valence and arousal, causing two emotions to be induced by each other
sometimes, while _surprise_ , _angry_ , and _happy_ are the high arousal
emotions, which may affect the recognition of each other slightly.

####  III-D4 Experimental Result Discussion

Throughout all experiments on IEMOCAP, ABC, and CASIA, some results are also
worthy of discussing, from which we can investigate the applicability and
limitation of our proposed ATFNN. Firstly, two selected databases, i.e.,
IEMOCAP, ABC, are class-imbalanced, resulting in the discrepancy between WAR
and UAR. Moreover, the recognition of different emotions has obvious
confusion, as shown in Figure. 5, where $58.27\%$ of speech samples on _happy_
are recognized as _neutral_ in IEMOCAP, and $45.45\%$ of samples on
_intoxicate_ in ABC are recognized as _cheerful_. These issues may be caused
by the following reasons: one reason is that these emotions are relatively
close in valence or arousal, and the other reason is that the long-tailed
distribution of speech samples leads the classifier being biased towards
emotions with more samples. Next, although the speech samples of CASIA are
class-balanced, its recognition performance is much lower than that of IEMOCAP
and ABC. This may be because CASIA is collected by forcing the speaker to
express different emotions in the same sentence, limiting the diversity of
speech samples. Also, CASIA is based on a tonal language (i. e., Chinese)
rather than the non-tonal language (i. e., English). In addition, since the
used LOSO protocol in our experiments is all based on speaker-independent, the
results on three databases demonstrate that our ATFNN without any domain
adaptation strategy not only preserves the discriminability of emotional
speech representation but also eliminates the influence of unrelated
information in the emotional speech, e. g., speakers, speaking styles.

####  III-D5 Ablation Experiment

To verify the effectiveness of our proposed framework, the additional
experiments are implemented for different architectures of ATFNN, and the
results in terms of WAR and UAR are reported in Table. III, where AFNN and
TFNN represent the FNN with F-Attention and the TNN with T-Attention
respectively, described in Section II. From the ablation results, obviously,
we can observe that the attention based time-frequency joint learning method
(i. e., ATFNN) has better performance than the time-frequency joint learning
method (i. e., TFNN), which indicates the proposed attention-based method can
focus on the emotion-related parts in the time-frequency domain. In addition,
we also observe that the time-frequency based method (i. e., ATFNN) also
perform better than the time or frequency based method (i. e., AFNN or ATNN),
which also shows that time-frequency joint learning can indeed obtain the
discriminative representation of speech emotion. It is also interesting to see
that, although the frequency domain representation is meaningful for emotions,
the experimental results also show that ATNN performs better than AFNN.

###  III-E Visualization of F-Attention and T-Attention

(a) Spectrogram of _happy_ on IEMOCAP

(b) F-Attention of _happy_ on IEMOCAP

(c) T-Attention of _happy_ on IEMOCAP

(d) Spectrogram of _sad_ on IEMOCAP

(e) F-Attention of _sad_ on IEMOCAP

(f) T-Attention of _sad_ on IEMOCAP

(g) Spectrogram of _cheerful_ on ABC

(h) F-Attention of _cheerful_ on ABC

(i) T-Attention of _cheerful_ on ABC

(j) Spectrogram of _tired_ on ABC

(k) F-Attention of _tired_ on ABC

(l) T-Attention of _tired_ on ABC

(m) Spectrogram of _happy_ on CASIA

(n) F-Attention of _happy_ on CASIA

(o) T-Attention of _happy_ on CASIA

(p) Spectrogram of _sad_ on CASIA

(q) F-Attention of _sad_ on CASIA

(r) T-Attention of _sad_ on CASIA

Figure 6: Visualization of log-Mel-spectrogram, T-Attention, and F-Attention
under different emotions. (a)-(f) show the attentions under _happy_ and _sad_
on IEMOCAP. (g)-(l) show the attentions under _cheerful_ and _tired_ on ABC.
(m)-(r) show the attentions under _happy_ and _sad_ on CASIA. In each figure,
the upper part of the black dotted line ’\- -’ means the high frequency energy
activation region, while the lower part represents the middle and low
frequency energy activation region.

To further investigate the correlations of the frequency band activations on
frequency domain and key frame regions on time domain, we visualize the
F-Attention and the T-Attention on the log-Mel-spectrogram, as shown in
Figure. 6. For this purpose, high-arousal and positive-valence emotions, e.
g., _happy_ , _cheerful_ , and low-arousal and negative-valence emotions, e.
g., _sad_ , _tired_ , are selected as comparison for attention visualization
[37], [38]. Among them, we choose _happy_ and _sad_ for a comparison on
IEMOCAP and CASIA, whereas on _cheerful_ and _tired_ , which have close
valence and arousal with _happy_ and _sad_ , are chosen as an alternative on
ABC. Figure. 6(a)-6(f) show the attentions under _happy_ and _sad_ on IEMOCAP,
where Figure. 6(a), 6(b), and 6(c) respectively correspond to the
visualizations of log-Mel-spectrogram, F-Attention, and T-Attention under
_happy_ , while Figure. 6(d), 6(e), and 6(f)) are the visualizations of above
three items under _sad_. Similarly, the details of 6(g)-6(r) are given in
Figure. 6. Notably, as shown on Figure. 6, the black dotted line ’\- -’ on
each figure represents the Mel-scaled frequency of 50, corresponding to the
actual frequency band around 2 700Hz. Therefore, the upper part of the black
dotted line means the high frequency energy activation region, while the lower
part represents the middle and low frequency energy activation region. In
addition, the attention visualizations reveal the specific activations under
different emotions, where the colors of visualizations from red to blue
represent corresponding activation intensity from large to small.

From the comparison of Figure. 6(a) and 6(b), it is obvious to observe that
there are frequent energy activations of _happy_ above 50 at the Mel-scaled
frequency (corresponding to the frequency band around 2 700Hz), which
demonstrates that _happy_ has an increased in high-frequency energy. Whereas,
as shown in Figure. 6(d) and 6(e), we can clearly find that energy activations
of _sad_ are infrequent above 50 at the Mel-scaled frequency, which indicates
_sad_ has a decreased in high-frequency energy. Similar results also appear in
ABC and CASIA, as shown in Figure. 6(g) and 6(j), Figure. 6(m) and 6(p). These
visualization results are consistent with the concepts reported in [20], [21],
[22]. Notably, the results in Figure. 6(j) and 6(k) also reveal that the
F-Attention of _tired_ on ABC has frequent energy activations in the high-
frequency bands from $10^{th}$ frame to $40^{th}$ frame and $125^{th}$ frame
to $250^{th}$ frame. This is because the used speech sample has obvious sound
of ’yawning’ in these two frame intervals. On the contrary, the sample from
$50^{th}$ frame to $80^{th}$ frame contains a conventional emotion expression
of _tired_ , thus its F-Attention reveals infrequent energy activations above
50 Mel-scaled frequency, which is consist with the above demonstration.
Moreover, the visualization of F-Attention confirms that our proposed method
can focus on the frequency band activations related emotions, and then combine
the correlations between these frequency bands into the frequency domain
representation.

As for T-Attention, the Figure. 6(c) indicates that _happy_ on IEMOCAP has
high activations in $50^{th}$ frame to $90^{th}$ frame and $110^{th}$ frame to
$270^{th}$ frame. These frames correspond to the regions with rich speech
contents shown in Figure. 6(a), which are all key regions in frames that
highly contribute to speech emotion representation. From the visualization of
T-Attention on other emotions or databases, i. e., Figure. 6(f), 6(i), 6(l),
6(o), and 6(r), it is clear to discover similar results with Figure. 6(c).
Also, our proposed T-Attention has few activations on all silent frames, e.
g., $0^{th}$ frame to $150^{th}$ frame and $360^{th}$ frame to $384^{th}$
frame in Figure. 6(d), $220^{th}$ frame to $256^{th}$ frame in Figure. 6(g),
$80^{th}$ frame to $120^{th}$ frame and $210^{th}$ frame to $480^{th}$ frame
in Figure. 6(j), $70^{th}$ frame to $120^{th}$ frame in Figure. 6(m),
$40^{th}$ frame to $60^{th}$ frame in Figure. 6(p). These salient results
further emphasize that our proposed T-Attention effectively focuses on the key
regions related to emotions in time frames.

##  IV Conclusion

In the paper, we propose an attentive time-frequency neural network (ATFNN) to
extract the discriminative presentation for the SER. We first utilize a time-
frequency neural network (TFNN), integrating a domain encoder and a time
domain encoder into the time-frequency joint learning strategy, to model the
correlations within frequency bands and within time frames. Then, a frequency
attention network and a time attention network are also embedded into the TFNN
to capture the specific frequency bands and time frames related to emotions.
Experimental results on the two public databases, i. g., IEMOCAP and ABC,
prove that our ATFNN outperforms the state-of-the-art methods. Furthermore,
the visualization of attentions reveals that the ATFNN can focus on the
regions related to emotions of the frequency domain and the time domain.
Moreover, precise correspondence between different emotions and frequency
bands should be deeply explored in future research to improve the robustness
and discriminability of speech emotion features.

## References

  * [1] B. Schuller and A. Batliner, _Computational paralinguistics: emotion, affect and personality in speech and language processing_. John Wiley & Sons, 2013.
  * [2] C. Busso, M. Bulut, S. Narayanan, J. Gratch, and S. Marsella, “Toward effective automatic recognition systems of emotion in speech,” _Social emotions in nature and artifact: emotions in human and human-computer interaction, J. Gratch and S. Marsella, Eds_ , pp. 110–127, 2013.
  * [3] I. Gat, H. Aronowitz, W. Zhu, E. Morais, and R. Hoory, “Speaker normalization for self-supervised speech emotion recognition,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7342–7346.
  * [4] L. Guo, L. Wang, J. Dang, E. S. Chng, and S. Nakagawa, “Learning affective representations based on magnitude and dynamic relative phase information for speech emotion recognition,” _Speech Communication_ , vol. 136, pp. 118–127, 2022.
  * [5] M. Li, B. Yang, J. Levy, A. Stolcke, V. Rozgic, S. Matsoukas, C. Papayiannis, D. Bone, and C. Wang, “Contrastive unsupervised learning for speech emotion recognition,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6329–6333.
  * [6] B. Schuller, B. Vlasenko, F. Eyben, G. Rigoll, and A. Wendemuth, “Acoustic emotion recognition: A benchmark comparison of performances,” in _2009 IEEE Workshop on Automatic Speech Recognition & Understanding_. IEEE, 2009, pp. 552–557.
  * [7] A. Stuhlsatz, C. Meyer, F. Eyben, T. Zielke, G. Meier, and B. Schuller, “Deep neural networks for acoustic emotion recognition: Raising the benchmarks,” in _2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2011, pp. 5688–5691.
  * [8] Q. Mao, M. Dong, Z. Huang, and Y. Zhan, “Learning salient features for speech emotion recognition using convolutional neural networks,” _IEEE transactions on multimedia_ , vol. 16, no. 8, pp. 2203–2213, 2014.
  * [9] S. Zhang, S. Zhang, T. Huang, and W. Gao, “Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching,” _IEEE Transactions on Multimedia_ , vol. 20, no. 6, pp. 1576–1590, 2017.
  * [10] C. Lu, Y. Zong, W. Zheng, Y. Li, C. Tang, and B. W. Schuller, “Domain invariant feature learning for speaker-independent speech emotion recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 2217–2230, 2022.
  * [11] F. Eyben, M. Wöllmer, and B. Schuller, “Openear—introducing the munich open-source emotion and affect recognition toolkit,” in _2009 3rd international conference on affective computing and intelligent interaction and workshops_. IEEE, 2009, pp. 1–6.
  * [12] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [13] J. Wang, M. Xue, R. Culhane, E. Diao, J. Ding, and V. Tarokh, “Speech emotion recognition with dual-sequence lstm architecture,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6474–6478.
  * [14] M. B. Akçay and K. Oğuz, “Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers,” _Speech Communication_ , vol. 116, pp. 56–76, 2020.
  * [15] Y. Xie, R. Liang, Z. Liang, C. Huang, C. Zou, and B. Schuller, “Speech emotion classification using attention-based lstm,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 11, pp. 1675–1685, 2019\.
  * [16] T. N. Sainath, O. Vinyals, A. Senior, and H. Sak, “Convolutional, long short-term memory, fully connected deep neural networks,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 4580–4584.
  * [17] J. Li, A. Mohamed, G. Zweig, and Y. Gong, “Lstm time and frequency recurrence for automatic speech recognition,” in _2015 IEEE workshop on automatic speech recognition and understanding (ASRU)_. IEEE, 2015, pp. 187–191.
  * [18] A. Satt, S. Rozenberg, and R. Hoory, “Efficient emotion recognition from speech using deep learning on spectrograms.” in _Interspeech_ , 2017, pp. 1089–1093.
  * [19] M. Chen, X. He, J. Yang, and H. Zhang, “3-d convolutional recurrent neural networks with attention model for speech emotion recognition,” _IEEE Signal Processing Letters_ , vol. 25, no. 10, pp. 1440–1444, 2018.
  * [20] S. Wu, T. H. Falk, and W.-Y. Chan, “Automatic speech emotion recognition using modulation spectral features,” _Speech communication_ , vol. 53, no. 5, pp. 768–785, 2011.
  * [21] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, “Emotion recognition in human-computer interaction,” _IEEE Signal processing magazine_ , vol. 18, no. 1, pp. 32–80, 2001.
  * [22] L. Kaiser, “Communication of affects by single vowels,” _Synthese_ , pp. 300–319, 1962.
  * [23] Y. Zhang, J. Du, Z. Wang, J. Zhang, and Y. Tu, “Attention based fully convolutional network for speech emotion recognition,” in _2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2018, pp. 1771–1775.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [25] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, no. 4, pp. 335–359, 2008.
  * [26] B. Schuller, D. Arsic, G. Rigoll, M. Wimmer, and B. Radig, “Audiovisual behavior modeling by combined feature spaces,” in _2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07_ , vol. 2. IEEE, 2007, pp. II–733.
  * [27] J.-C. Wang, Y.-H. Chin, B.-W. Chen, C.-H. Lin, and C.-H. Wu, “Speech emotion verification using emotion variance modeling and discriminant scale-frequency maps,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 10, pp. 1552–1562, 2015.
  * [28] J. T. F. L. M. Zhang and H. Jia, “Design of speech corpus for mandarin text to speech,” in _The Blizzard Challenge 2008 workshop_ , 2008.
  * [29] X. Wu, S. Liu, Y. Cao, X. Li, J. Yu, D. Dai, X. Ma, S. Hu, Z. Wu, X. Liu _et al._ , “Speech emotion recognition using capsule networks,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6695–6699.
  * [30] S. Bhosale, R. Chakraborty, and S. K. Kopparapu, “Deep encoded linguistic and acoustic cues for attention based end to end speech emotion recognition,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7189–7193.
  * [31] S. Mao, D. Tao, G. Zhang, P. Ching, and T. Lee, “Revisiting hidden markov models for speech emotion recognition,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6715–6719.
  * [32] W. Fan, X. Xu, X. Xing, and D. Huang, “Adaptive domain-aware representation learning for speech emotion recognition.” in _INTERSPEECH_ , 2020, pp. 4089–4093.
  * [33] X. Xu, J. Deng, E. Coutinho, C. Wu, L. Zhao, and B. W. Schuller, “Connecting subspace learning and extreme learning machine in speech emotion recognition,” _IEEE Transactions on Multimedia_ , vol. 21, no. 3, pp. 795–808, 2018.
  * [34] Z.-T. Liu, Q. Xie, M. Wu, W.-H. Cao, Y. Mei, and J.-W. Mao, “Speech emotion recognition based on an improved brain emotion learning model,” _Neurocomputing_ , vol. 309, pp. 145–156, 2018.
  * [35] K. Han, D. Yu, and I. Tashev, “Speech emotion recognition using deep neural network and extreme learning machine,” in _Interspeech 2014_ , 2014.
  * [36] Y. Sun, G. Wen, and J. Wang, “Weighted spectral features based on local hu moments for speech emotion recognition,” _Biomedical signal processing and control_ , vol. 18, pp. 80–90, 2015.
  * [37] P. J. Lang, “The emotion probe: Studies of motivation and attention.” _American psychologist_ , vol. 50, no. 5, p. 372, 1995.
  * [38] J. Kim and E. André, “Emotion recognition based on physiological changes in music listening,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 12, pp. 2067–2083, 2008.