# Graph-based multi-Feature fusion method for speech emotion recognition

Xueyu Liu<EMAIL_ADDRESS>School of Economics and Management,Tongji
University, No.1 Zhangwu Road
Shanghai,200090,China
Jie Lin111Corresponding author<EMAIL_ADDRESS>School of Economics and
Management,Tongji University, No.1 Zhangwu Road
Shanghai,200090,China,
Chao Wang<EMAIL_ADDRESS>School of Economics and Management,Tongji
University, No.1 Zhangwu Road
Shanghai,200090,China

###### Abstract

Exploring proper way to conduct multi-speech feature fusion for cross-corpus
speech emotion recognition is crucial as different speech features could
provide complementary cues reflecting human emotion status. While most
previous approaches only extract a single speech feature for emotion
recognition, existing fusion methods such as concatenation, parallel
connection, and splicing ignore heterogeneous patterns in the interaction
between features and features, resulting in performance of existing systems.
In this paper, we propose a novel graph-based fusion method to explicitly
model the relationships between every pair of speech features. Specifically,
we propose a multi-dimensional edge features learning strategy called Graph-
based multi-Feature fusion method for speech emotion recognition. It
represents each speech feature as a node and learns multi-dimensional edge
features to explicitly describe the relationship between each feature-feature
pair in the context of emotion recognition. This way, the learned multi-
dimensional edge features encode speech feature-level information from both
the vertex and edge dimensions. Our Approach consists of three modules: an
Audio Feature Generation(AFG)module, an Audio-Feature Multi-dimensional Edge
Feature(AMEF) module and a Speech Emotion Recognition (SER) module. The
proposed methodology yielded satisfactory outcomes on the SEWA dataset.
Furthermore, the method demonstrated enhanced performance compared to the
baseline in the AVEC 2019 Workshop and Challenge. We used data from two
cultures as our training and validation sets: two cultures containing German
and Hungarian on the SEWA dataset, the CCC scores for German are improved by
17.28% for arousal and 7.93% for liking, and for Hungarian, the CCC scores are
improved by 11.15% for arousal and 131.11% for valence. The outcomes of our
methodology demonstrate a 13% improvement over alternative fusion techniques,
including those employing one dimensional edge-based feature fusion approach.

###### keywords:

Speech emotion recognition; Speech feature fusion; Multi-dimensional Edge;
Graph representation Learning.

##  1 Introduction

Speech is an easily accessible medium for humans to communicate their
intentions and messages, which can be conveyed via ubiquitous devices such as
microphones and phone. Consequently, many previous studies devoted to
recognizing human emotions from their speeches [1]. Recent development in
speech emotion recognition has greatly transformed consumer shopping,
information gathering, and communication [2]. Practice feedback has been well
applied commercially to the fields of human-computer interaction, healthcare,
and marketing [3, 4].

A key challenge in speech emotion recognition is to learn discriminative
emotional representations from raw speech signals. Early studies [5] used low-
level hand-crafted features to complete speech emotion recognition tasks,
including Low level descriptors(LLDS) and High level statistics functions
(HSFs, e.g. mean and maximum values of LLDs) [6], Mel Frequency Cepstral
Coefficients (MFCCs) [7], Linear Predictive Coding (LPC) [8], and Perceptual
Linear Prediction (PLP) [9, 10]. Based on LLDs, researchers applied audio word
bag (BoAW) to extract features in an unsupervised manner, and obtained more
statistical information by aggregating multiple frame-level descriptive words
from audio segments [11]. However, the above-mentioned hand-crafted features
may fail to learn task-specific (e.g., emotion-related) features from the
given audio signals. As a result, with the emergence of deep learning model,
more and more speech emotion recognition methods have been built on Deep
Neural Network (DNNs), which can be optimized based on the target and thus can
extract task-specific features from audios to make better predictions [12, 13,
14, 15]. Deep learning methods can extract hierarchical and discriminative
feature representations through supervised learning, and the application of
models such as CNN and LSTM has also been shown to further improve accuracy
[6, 16, 17]. Recently, other artificial intelligence (AI) techniques, such as
self-attention mechanisms, have also been proposed for SER tasks [18].

Figure 1: Comparison between different speech feature fusion methods, with
examples shown as follows, Use audio features (AF), recognition results
prediction, and feature vectors V. (I) Perform feature-level fusion using
concat methods; (II) Perform decision-level feature fusion based on weighted
recognition results (I) and (II) without using edge features; and (III) Our
method Showing a unique association pattern encoded for each feature vector in
the node, these features also determine the specific task topology of the
graph, and additionally describe the use of multidimensional features for each
edge (the relationship between a pair of AFs).

In previous studies, researchers frequently predict emotions based on a single
speech feature described above [19, 20, 21]. In the context of SER tasks, low-
level hand-crafted features have been employed in small-sample data sets due
to their transparent computational principles and straightforward structure.
In contrast, deep learning methods are more suited to feature extraction tasks
in massive data sets, as they are capable of automatically learning optimal
feature representations and combining low-level features to form novel
features [22, 23]. For speech emotion recognition, the use of a single type of
feature in some cases leads to low recognition accuracy. First, when dealing
with speech data from speakers with different backgrounds, due to the
influence of contextual factors such as gender information, language, culture,
etc., these problems lead to a single audio feature that cannot cover all the
emotional information [24]; second, when dealing with speech data with a large
amplitude of emotional change, due to the influence of the speaker’s emotion,
the existence of changes in the speed of speech, as well as the difference in
the energy between the various spectra, a single feature can only express
speech’s emotional information from one side to express the emotional
information of speech, so a single type of feature has a performance
bottleneck in speech emotion recognition [25, 26].

To overcome the limitations of speech recognition based on a single speech
feature, combining different speech features has proven to be an effective
solution [27]. Specifically, the most common solution is to directly
concatenate multiple different speech features [28]. For example, Zhou et al.
combine emotion feature maps extracted from different spectral diagrams,
thereby enhancing the accuracy of speech recognition [29]. Alternatively, some
studies conduct decision-level fusion, which first make an emotion prediction
from each feature and then average all predictions [30, 31]. However, both
strategies fail to comprehensively model the inter-relation between different
speech features, which may lead to not only the final combined feature contain
redundant information but also ignore crucial relationship cues for the
emotion recognition. Recently, Graph Neural Networks (GNNs) have attracted
more attentions from researchers for speech emotion analysis [32]. However,
the node features used in this study are all deep learning features from LSTM
outputs, and as mentioned earlier, a single feature type can provide very
limited useful information. Therefore, we propose the hypothesis that using
different types of speech features as nodes, more useful information can be
learned from neighboring nodes using an effective aggregation method [6].

In this paper, we propose a novel graph-based fusion strategy that leverages
the merits of both deep-learned and hand-crafted audio features for speech
emotion recognition. Inspired by GRATIS [33], we propose a novel method that
learns a multi-dimensional edge feature to explicitly describe the
relationship between each pair of to address the challenge of feature
integration in cross-corpus speech emotion recognition. Compared to methods
such as GIN, it has higher coupling with non-graphical heterogeneous data such
as deep learning and hand-crafted audio features and can easily be combined
with various deep learning backbone and GNN to predict different downstream
tasks, facilitating subsequent speech emotion classification. Our approach
consists of three modules: an Audio Feature Generation (AFG) module, an Audio-
Feature Multi-dimensional Edge Feature (AMEF) module and a Speech Emotion
Recognition (SER) module. Initially, it draws upon two conventional features,
eGeMAPs and MFCCs, as well as two bag-of-words models, BoAW-e and BoAW-m, and
the deep learning feature, Deep Spectrum. AFG module extracts five feature
vectors representing these five speech features within an LSTM framework. In a
subsequent phase, the AMEF module was used to construct a graph representation
based on multidimensional edges, which is ultimately integrated into a
combined feature. Finally, the combined feature is fed back into the model as
an input. The SER module completes the emotion recognition of speech vectors
through regression methods, resulting in the outcome. The main contribution of
this work is summarized as follows:

To the best of our knowledge, we introduce the first approach to leverage
multi-dimensional edge features for graph-based cross-corpus sentiment
recognition tasks. By constructing learnable multi-dimensional edge features,
we explicitly consider and model the relationships between complex and diverse
audio features.

We present a method for fusing different types of speech features in a way
that captures the information and relationships of different speech features.
Through cross-attention, we effectively extract novel features and utilize
them to guide the updating of multi-dimensional edge features.

We conducted experiments on SEWA dataset,an official cross-lingual speech
dataset,significant improvements in the accuracy of speech emotion recognition
have been achieved by integrating fusion methods into existing graph-based
frameworks.

##  2 Related Work

###  2.1 Speech Emotion Recognition Methods

The primary task of Speech Emotion Recognition (SER) is to extract and
identify the emotional information contained in speech. Over the past several
decades, the study of identifying human emotional states from speech through
computer technology has been a popular field of research. The SER system’s
recognition process of speech emotions is divided into two steps. Firstly,
speech features need to be extracted. These features carry contextual
information and are important sources of information for computers to
recognize emotions. In early research, the common features used in speech
emotion recognition by academia included prosody features, spectral features,
and timbre features [9]. The hand-crafted features include statistical
functions such as mean values and extrema, as well as features such as Mel
Frequency Cepstral Coefficients (MFCCs), Linear Predictive Coding (LPC), and
Perceptual Linear Prediction (PLP) [9, 10]. On the basis of LLDs, researchers
have used the Audio Word Bag (BoAW) to extract features in an unsupervised
manner [11].

Recently, many studies have begun to use deep learning methods to extract Deep
Spectrum representations from audio segments. Deep Spectrum representations
are mainly due to the advantages of transfer learning, as they are formed by
passing spectral maps through pre-trained image classification CNNs such as
AlexNet or VGG [34]. Deep Spectrum features are also typical Deep Spectrum
features, and have been proven effective for various general audio recognition
tasks [35]. For example, the DeepSpectrumLite transfer learning framework is
used for pre-training image convolutional neural networks (CNNs) for speech
recognition [15]. Moreover, humans may have more than one tone when speaking,
and the diversity of tones represents changes in emotion [36, 37]. Under such
circumstances, the boundaries of emotional recognition become even more
complex due to human factors. On the other hand, the perception of speech in a
language depends on factors such as the speaker’s cultural background,
language, gender, and even the speaker’s age, which makes speech emotion
recognition more complex [38]. Based on the above description, the second step
of speech emotion recognition is to classify the emotional content of the
speech file through machine learning and other classification algorithms. In
most cases, classifiers are divided into two main categories: traditional
classifiers and deep learning classifiers. Traditional classifiers include
SVM, HMM, GMM, KNN, decision trees, LDA, and maximum likelihood methods [39,
40, 41]. In contrast, there has been an increasing number of studies in recent
years using deep learning classifiers such as CNN, DNN, RNN, DBN, LSTM, and
DBM for speech emotion recognition. Notably, the emotional content in speech
changes over time, so it is appropriate to use techniques that are effective
for modeling time, such as stochastic HMMs or neural networks with recursive
units, such as LSTM or GRU [42].

Despite notable progress in emotion recognition, cross-corpus speech emotion
recognition remains a significant challenge due to cultural and linguistic
disparities [43]. In the CES sub-project of the AVEC 2019 Workshop and
Challenge, Fabien and colleagues utilized a 2-layer LSTM-RNN (64/32 units) as
the time-dependent regressor for each representation of audio signals and
employed SVMs for late-stage fusion of predictions [44]. Compared to the
performance reported in the previous AVEC CES, arousal and valence scores for
German improved by 7.25% and 8.25%, respectively, while those for Hungarian
improved by 17.3% and 13.3%, respectively. However, the study still faces an
issue: individual features such as the MFCCs, eGeMAPs, and deep learning DEEP
SPECTRUM cannot achieve optimal results across multiple emotion categories due
to varying information content within each feature [45]. This is particularly
true since these features describe emotional information from different
perspectives. Therefore, there is a need to find a method for deep fusion of
features to generate new ones, aiming for better results across multiple
emotion categories.

###  2.2 Graph Representation Learning and Graph Neural Networks

While deep learning effectively captures the hidden patterns in Euclidean data
such as images, text, and videos, an increasing number of applications
represent data in graphical form [46]. For instance, in e-commerce, graph-
based learning systems can leverage relationships between users and products
to make highly accurate recommendations [47]. Graph Neural Networks (GNNs)
have garnered significant attention for their ability to handle non-Euclidean
data [48]. Thanks to their unique advantages in relationship construction,
GNNs can effectively represent multiple features of the same type as nodes in
a graph, thereby capturing both similarities and differences between data
points [49, 50, 51, 52].

The purpose of graph representation learning is to convert nodes in a graph
into low-dimensional vector representations through specific methods. This
low-dimensional vector representation can preserve the node features,
structural features, and semantic information of the original graph,
represented by a set of vertices and edges [53, 54, 55]. However, most graphs
are constructed by manually defining the vertices and the characteristics of
each edge through predefined rules, which ignores the links related to the
task and affects the performance of graph analysis [56], [57]. As an essential
component of the graph, edge features can avoid overlooking crucial feature-
relation clues by leveraging the relationships between vertices [58, 59, 60].
In recent years, there have been studies using multi-dimensional edge features
to describe specific relationships between vertices, such as using heuristic
edge features to represent relationships [61]. However, these hand-crafted
features still cannot capture specific task relationship information between
vertices [62, 63]. This requires a graph representation learning framework to
automatically generate task-specific topology and multi-dimensional edge
features for different input modalities. Some studies have utilized multiple
sub-graphs to learn rich node representations in graph-based networks,
proposing two types of edge features and effectively combining them with GAT
and GCN models for relationship extraction [64]. Prior to this, multi-
dimensional edge features have been widely used in areas [65, 66, 67, 81].
Song et al. proposed a general graph representation learning framework (called
GRATIS) that can generate strong graph representations with specific task
topologies and multi-dimensional edge features for arbitrary inputs. Research
results show that GRATIS not only greatly enhances pre-defined graphs but also
learns strong graph representations of non-graph data, resulting in
significant performance improvements across all tasks [33]. We leverage the
GRATIS graph representation learning framework as an auxiliary task, fully
utilizing information from various speech feature datasets.

##  3 Method

Figure 2: The framework of the graph-based multi-feature fusion method. For
simplicity, only the processing process of an audio file is shown in the
graph. The specific details of the modules are given in the corresponding
sections.

In this section, we propose a novel graph-based speech feature fusion method,
where multi-dimensional edge features are learned to specifically describe the
relationship between every pair of speech features.

Overview: The overall architecture is illustrated in Figure 2. First, our
Audio Feature Generation (AFG) module extracts features from the given audio
signal, including five hand-crafted features, namely eGeMAPs and MFCCs, two
bag-of-words model features, BoAW-e, and BoAW-m, as well as deep learning
features, Deep Spectrum. These five features encompass different speech
emotional information. The feature extraction is accomplished through two
layers of LSTM. Secondly, Due to the varying dimensions of diverse types of
speech features, a 2-layer LSTM-RNN (64/32 units) can be employed to achieve a
uniform feature vector dimensionality. Thirdly, the GRATIS framework generate
graph representations with task-specific topology and multidimensional edge
features, since speech features are non-graph data, the input speech features
will be represented by a set of vectors and the GRATIS framework connects them
into a matrix, by training to obtain the optimal topology. This vector,
through edge feature information, contains the association between features,
and through point feature information, represents the differences between
features. Finally, the vector is fed back into the model as an input, and the
end-to-end output is obtained through a fully connected layer.

###  3.1 Hand-crafted and Deep learning speech features extraction

Identifying emotions from audio-visual signals usually relies on feature sets,
which are extracted based on different aspects that help the model acquire
knowledge. In the first module of Figure 2 (AFG), the study achieve the
extraction of handcrafted speech features.

eGeMAPs features using the extended Geneva Minimum Acoustic Parameter Set
(eGeMAPs). This set encompasses 88 metrics covering the acoustic dimensions,
include spectral, cepstral, prosodic, and speech quality information.

MFCCs features have 13 coefficients, which have better robustness than LPCCs
based on vocal tract models, and have been proven to have good recognition
performance even when the signal-to-noise ratio decreases [68]. And the
extraction of MFCCs was accomplished using the openSMILE2 toolkit [45], [69].

Bag-of-words-eGeMAPs (BoAW-e) features are based on the bag-of-words model
features of handcrafted features, eGeMAPs LLDs are standardized in an online
manner (zero mean, unit variance) before vector quantization. The entire
cross-modal BoW (XBoW) processing is executed using the open-source toolkit
openXBOW4 chain [72].

Bag-of-words-MFCCs (BoAW-m) features are based on the bag-of-words model
features of handcrafted features, MFCCs LLDs are standardized in an online
manner before vector quantization. The entire cross-modal BoW (XBoW)
processing is executed using the open-source toolkit openXBOW4 chain [72].

Deep Spectrum features, the inspiration for Deep Spectrum features comes from
image processing. By inputting the spectral image of a speech instance into a
pre-trained image recognition CNN and extracting a set of resulting activation
values as feature vectors. Deep Spectrum features have been widely applied in
past speech emotion recognition research for their efficient extraction
ability and good recognition preparation rate [72], [77].We employed the
VGG-16 to extract Deep Spectrum features from the pre-trained CNN [78]. A
4096-dimensional feature vector is then extracted from the activations of the
second fully connected layer in VGG-16.

In summary, a total of five different features can be obtained, the LLDs
features contain acoustic features with a small number of dimensions, which
can effectively reduce the computational complexity of the subsequent pattern
recognition system; whereas the Deep Spectrum features contain more
dimensions, which have high computational complexity when inputted to the
recognition system but contain spatio-temporal features, such as temporal
information, and each has its own advantages and disadvantages. Therefore, we
expect to further improve their performance in speech emotion recognition
scenarios through feature fusion.

###  3.2 Audio-feature temporal modelling

Before feature fusion, we need to preprocess different types of features, The
method comprises two stages. The first stage involves unifying the vector
dimensions of each type of speech feature, represented as a graph comprising
speech features with the same vector dimensions. This reduces the
computational complexity of the subsequent speech emotion recognition model.
Second, to consider the temporal sequence when processing the feature vectors,
due to the time-series nature of the speech data, the LSTM is more time-
sensitive than the ordinary RNN, and it can learn the patterns and features in
the time-series data. Therefore, we choose a 2-layer LSTM-RNN to standardize
the dimension of speech feature vectors. The 2-layer LSTM layers with
different units can better capture the key information between sequential
data.

###  3.3 Audio feature fusion method

####  3.3.1 Vertex Feature Extraction

Five different speech features (eGeMAPs, MFCCs, BoAW-M, BoAW-e and DS) are
obtained from the input audio feature data,these five speech features are used
as the vertex features with the $\mu$. Firstly, The graph representation
consists of five vertex features, each representing a speech feature: the
hand-crafted features eGeMAPs and MFCCs, the two bag of words features BoAW-e
and BoAW-m, and Deep Spectrum features. Each speech feature is processed by
two LSTM layers in order to obtain a feature vector $\mu$ with a unified
dimension of $[N,K]$, the representative speech data has $N$ frames and each
feature contains $K$ dimensions. Then, a set of vertex features for each frame
can be represented as:

$\mu=\left(\mu_{i},\mu_{j},\mu_{k},\mu_{n},\mu_{l}\right)$ (1)

where the vertex ${\mu}_{i}$ represents the use of eGeMAPs features processed
into feature vectors by two-layer LSTM; The vertex ${\mu}_{j}$ represents the
use of MFCCs features processed into feature vectors by two-layer LSTM; The
vertex ${\mu}_{k}$ represents the use of BoAW-M features processed into
feature vectors by two-layer LSTM; The vertex ${\mu}_{n}$ represents the use
of BoAW-e features processed into feature vectors by two-layer LSTM; The
vertex ${\mu}_{l}$ represents the use of Deep Spectrum features processed into
feature vectors by two-layer LSTM.

The vertex features are initially fed to a backbone, which may be any suitable
machine learning model. In the case of graph data, this may be a GNN, whereas
in the case of non-graph data, this may be a Transformer. In this study, we
have chosen to use a BiGRU model as the backbone. The composition of the
backbone is due to the fact that the BiGRU model introduces a bi-directional
structure on top of the GRU model, which better captures the bi-directional
dependencies of sequential data, such as speech features. In our
implementation, we use GRU as the backbone, to directly extract the global
contextual representation$\mathbf{\/Y}\in{\mathbb{R}}^{A\times K\times N}$
from the input data, where A denotes the number of vertices. The global
contextual representation $Y$ is defined as:

$Y=BiGRU{\left(\mu\right)}$ (2)

####  3.3.2 Graph Definition

We need to use the vertex features $\mu$, the basic adjacency matrix
$\mathcal{A}$ and edge features $\beta$ to build its graph representation
$\mathcal{G}^{A}(\mu,\beta)$. The number of vertex feature extractors is $i$,
each of which consists of a fully connected layer (FC). The vertices set $\mu$
contains vectors describing five audio features (each has $K$ dimensions). We
treat each vector directly as a vertex, and the basic adjacency matrix
concatenating all vectors as the global contextual representation $Y$. We
define edge presence in the basic graph $\mathcal{G}^{A}$ according to
$\mathcal{R}ule$, These can be formulated as:

$\begin{matrix}\mathcal{G}=\mathcal{G}^{A}(\mu,\beta)\\\
\beta\subseteq\left\\{\mathbf{e}_{i,j}=1\mid\mathbf{\mu}_{i},\mathbf{\mu}_{j}\in\mu\mathrm{\
and\ }\mathcal{A}_{i,j}=1\right\\}\\\ \end{matrix}$ (3)

which is subject to:

$\mathcal{A}_{i,j}=\left\\{\begin{matrix}1&\left\\{\mathbf{\mu}_{i},\mathbf{\mu}_{j}\right\\}\in\mathcal{R}ule\\\
0&\mathrm{\ Otherwise\ }\\\ \end{matrix}\right.$ (4)

where $\mathbf{e}_{i,j}$ represents the vertex ${\mu}_{i}$ effect on its
adjacent vertex ${\mu}_{j}$ is controlled by a single value. $\mathcal{R}ule$
represents the basic edge representation between a pair of connected vertex
$i$ and vertex $j$ is defined as 1.

####  3.3.3 The task-specific topology and vertex features(TTF)

This section takes the global contextual representation $Y$, the basic
adjacency matrix $\mathcal{A}$ and vertex features ${\mu}$ as the input, and
generates the task-specific adjacency matrix topology and vertex features.

Firstly, we consider the feature vectors as five node features and define the
connectivity (vertex-to-vertex relationship) and the connectivity between a
pair of nodes $\boldsymbol{\mu}_{i}$ and $\boldsymbol{\mu}_{j}$ by the
similarity of the
features($Sim(i,j)=\boldsymbol{\mu}_{i}^{T}\boldsymbol{\mu}_{j}$). At the same
time, we select the K nearest neighbors of each node as its neighbors, thus
completing the definition of the graph topology. Then, a GCN layer is employed
to jointly update all vertexs from the produced graph, where the $i_{th}$
node’s task-specific vertex feature $\boldsymbol{\hat{\mathcal{\mu}_{i}}}$ is
generated by $\boldsymbol{\mu}_{i}$ and its connected nodes as:

$\boldsymbol{\hat{\mathcal{\mu}_{i}}}=Relu\left[\boldsymbol{\mu}_{i}+p\left(\boldsymbol{\mu}_{i},\sum_{j=1}^{A}q\left(\boldsymbol{\mu}_{j},a_{i,j}\right)\right)\right]$
(5)

where ${Relu}$ is the nonlinear activation function, ${p}$ and ${q}$ denote
functions of the GCN layer, ${a_{i,j}}$ represents the relationship between
two vertices.

After obtaining the optimal task-specific vertex feature representation, we
then use this to calculate the vertex-to-vertex similarity to obtain the task-
specific adjacency matrix topology $\mathcal{\hat{A}}_{i,j}$.

$\mathcal{\hat{A}}_{i,j}=\boldsymbol{\hat{\mathcal{\mu}_{i}}}\boldsymbol{\hat{\mu}_{j}^{T}}$
(6)

####  3.3.4 Audio-Feature Multi-dimensional Edge Feature Learning (AMEF)

To further learn various cues that describe the relationship between speech
features and specific tasks, each learned edge is a 1$\times$N dimension,
which assign a multi-dimensional to each learned edge feature
${\hat{\mathbf{e}}}_{i,j}\in\mathbb{R}^{1\times N}$. This module is trained
under the supervision of target graph analysis tasks. Due to the complex
information dimension of speech features, including mathematical information
and temporal information, we assume that the relationship between the
connected multi-dimensional vertices $\hat{\mathbf{\mu}}_{i}$ and
$\hat{\mathbf{\mu}}_{j}$ cannot be best described by one-dimensional edge
features. In addition, this relationship clue is not only contained in their
vertex features, but also reflected in the global context of the graph. The
fusion method takes various speech features $\mu$ and global context
representation $Y$ as inputs and outputs a multi-dimensional edge feature
vector.

The Audio-feature Vertex-Context Relationship Modelling (AVCR) block takes
vertex features $\hat{\mathbf{\mu}}_{i}$ and $\hat{\mathbf{\mu}}_{j}$ and the
global contextual representation $Y$ as input. It conducts cross attention
between $\hat{\mathbf{\mu}}_{i}$ and $Y$ as well as $\hat{\mathbf{\mu}}_{j}$
and $Y$. Before that, the global contextual representation $Y$ goes through a
reshape operation into $\hat{Y}$ to keep the shape consistent with that of
$\hat{\mathbf{\mu}}_{i}$ and $\hat{\mathbf{\mu}}_{j}$. The vertex features
$\hat{\mathbf{\mu}}_{i}$ and $\hat{\mathbf{\mu}}_{j}$ are independently used
as queries to locate vertex-context relationship features $\mathcal{F}_{i,x}$
and $\mathcal{F}_{j,x}$ in $\hat{Y}$. Mathematically speaking, this process
can be represented as:

$\begin{matrix}\mathcal{F}_{i,x}=AVCR(\hat{\mathbf{\mu}}_{i},\hat{Y})\\\
\mathcal{F}_{j,x}=AVCR(\hat{\mathbf{\mu}}_{j},\hat{Y})\end{matrix}$ (7)

with the cross-attention operation in AFR defined as:

$AVCR{\left(A,B\right)}=softmax{\left(\frac{AS_{q}\left(BS_{k}\right)^{T}}{\sqrt{f_{k}}}\right)}BS_{\mu}$
(8)

where ${S_{q}}$,${S_{k}}$,${S_{\mu}}$ are learnable weight matrices for the
encoding of the query, the key and the values, and ${f_{k}}$ is a scaling
factor.

The Audio-feature Vertex-Vertex Relationship Modelling (AVVR) block further
extracts the features. It also conducts the cross-attention (has the same form
as Eq. 10 but independent weights) produces features $\mathcal{F}_{i,j,x}$ and
$\mathcal{F}_{j,i,x}$, where $\mathcal{F}_{i,j,x}$ is generated by using
$\mathcal{F}_{j,x}$ as the query and $\mathcal{F}_{i,x}$ as the key and value.
Finally, we feed $\mathcal{F}_{i,j,x}$ and $\mathcal{F}_{j,i,x}$ to a fully-
connected (FC) layer to obtain multi-dimensional edge feature vectors
$\boldsymbol{e}_{i,j}$ and $\boldsymbol{e}_{j,i}$, respectively. This process
can be represented as:

$\boldsymbol{e}_{i,j},\boldsymbol{e}_{j,i}=FC\left(AVVR\left(\mathcal{F}_{j,x},\mathcal{F}_{i,x}\right),AVVR\left(\mathcal{F}_{i,x},\mathcal{F}_{j,x}\right)\right)$
(9)

In order to better integrate the multidimensional edge feature learning method
for multiple speech features, The graph was constructed using the GCN model
after the multidimensional edge features were obtained. These features were
then combined with the previously obtained task-specific topology and vertex
features. The task-specific vertex features were used to create the task-
specific topology, which was then used as input.The task-specific topology is
unified with the multidimensional edge feature through multiplication. Prior
to this, the task-specific topology is reshaped once, with the reshaped
dimension matching that of the multidimensional edge features.

$\boldsymbol{\hat{e}}=\boldsymbol{{e}}\mathcal{\hat{A}}$ (10)
$\mathbf{\hat{\mu}^{AMEF}},\mathbf{\hat{e}^{AMEF}}=\mathbf{GCN}\left(\mathbf{\hat{\mu}},\mathcal{\boldsymbol{\hat{e}}}\right)$
(11)

where $\hat{\mu}$ represents the set of all task-specific vertex features
obtained in the TTF section, $\mathcal{\hat{A}}$ represents the task-specific
topology between all vertices.

###  3.4 Feature fusion and emotion recognition

Once the audio-feature relation graph
$\boldsymbol{G}^{L}=\left(\boldsymbol{\mu}^{L},\boldsymbol{e}^{L}\right)$ that
consists of $A$ node features and $A\times A$ multi-dimensional directed edge
features is learned, we feed this into the GCN model to jointly recognise all
audio data. The next step is the feature fusion, where graphs containing
multiple classes of features are fed to through the employed GCN network, and
then a final fusion vector is obtained. Specifically, in the reasoning phase,
the graph consists of multiple types of speech features, using the number of
nodes to determine the structure and size of the output layer, this leads to
feature fusion. In initialising the weights, the number of nodes $Num$ is used
to initialise the weight matrix. In the step of batch normalisation layer
operation, the layers normalised against node features, their dimension is set
to $Num$ and in the layers normalised against edge features, their dimension
is set to $Num\times Num$ as the feature matrix of the edges is two
dimensional. In the update step of the nodes, $Num$ is used to normalise the
aggregated edge features, i.e. divide by $Num$. This is because when
calculating the update for each node, it is necessary to take into account how
many edges point to that node. Compared to traditional feature fusion methods,
our method first fuse node features by applying convolution operation on the
graph structure, through which not only the features of the node itself are
fused, but also the features of the neighbouring nodes are taken into account,
and then the edge information is preserved by the method of multidimensional
edge features. Secondly, the output layer of our method usually uses a fully
connected layer whose output dimension is equal to $Num$, and then, the
features of each node are mapped to different class probabilities. Finally, in
contrast to approaches that perform fusion at the feature level and fusion at
the decision level, our approach allows dynamic feature fusion at different
levels of the graph structure, which provides more flexibility to the model.

After completing the above updates, we obtained the final output of the AMEF
module $\boldsymbol{\mu}^{L}$, which is a $N\times Num$ feature vector, where
N is the dimension of the audio data and $Num$ represents the number of
classifications set by the model. At the end of the model, we add a linear
layer with the purpose of transforming $Num$ in the output of the AMEF module
into a feature vector with a value of 1.This step is the fusion of speech
features, we fuse five features into a new feature vector and use the feature
vector to complete the task of speech emotion recognition.

###  3.5 Training scheme

The training set and validation set are divided according to cultural types,
where the training set is divided into monolingual data sets and multilingual
data sets. We use two types of data sets to complete the experiment, and input
the fused feature vectors into the 2-layer LSTM-RNN. For time series
prediction problems, it is equivalent to setting sequence length to 1. The
prediction results are processed by squeeze dimensionality reduction,
performing standardization of the prediction results $r_{i}$ and label values
$r_{j}$. Calculate the mean and variance of the output prediction values and
label values. As shown in equations (12) and (13):

$\bar{r}=\frac{\sum_{n=1}^{N}r}{N}$ (12)
$r\\_val=\frac{\sum_{n=1}^{N}(r-\bar{r})^{2}}{N}$ (13)

where prediction results is $r_{i}$ and label values is $r_{j}$, The N
represents the number of data points. The correlation coefficient
$\sigma_{r_{i}r_{j}}$ is calculated as follows:

$\sigma_{r_{i}r_{j}}=\left(r_{i}-{\bar{r}}_{i}\right)\cdot\left(r_{j}-{\bar{r}}_{j}\right)$
(14)

The loss function for training and validation is based on the consistency
correlation coefficient (CCC) as shown below:

$ccc\\_loss=C-\frac{2\cdot\sigma_{r_{i}r_{j}}}{\left({\bar{r}}_{i}-{\bar{r}}_{j}\right)^{2}+r_{i_{-}var}+r_{j_{-}var}}$
(15)

where $C$ represents a constant, and in this study, and $C$ is taken as 2 when
calculating CCC loss. The objective of this loss function is to update the
model in order to identify the model with the highest CCC score.

##  4 Experiment

This section introduces the experimental details and results. Section 4.1
first presents the dataset. Then, Sections 4.2 and 4.3 introduce Evaluation
Metrics and Implementation Details, respectively. Subsection 4.4 explains the
advantages of our method compared to the baseline. Finally, subsection 4.5
shows ablation experiments and analyze the accuracy of speech emotion
recognition with different fusion features and our fusion features.

###  4.1 Dataset

We conducted extensive experiments on a cross-cultural speech dataset: the
SEWA dataset, which was officially authorized for access and use [80]. The
SEWA database comprises 199 experimental sessions, involving 398 participants
from six distinct cultures (UK, Germany, Hungary, Greece, Serbia, and China).
We selected the SEWA data used in the AVEC 2019 Workshop and Challenge,
encompassing the German and Hungarian cultures, the emotional dimensions
arousal and valence, and liking, were independently assessed by several native
speakers [44].

Number of subjects and duration of the video chats contained in the SEWA
database [80].  Culture Partition # Subjects Duration [h:min:s] German
Training 34 1:33:12 German Devel 14 0:37:46 Hungarian Training 34 1:08:24
Hungarian Devel 14 0:28:42 All 96 3:48:04

###  4.2 Evaluation Metrics

To ensure comparable results, we used a training and validation set consistent
with Ringeval et al [44]. These datasets were taken from the SEWA dataset, and
the training set contained both German and Hungarian cultures, and for
training and validation we used three combinations of data: data from the
German culture only, data from the Hungarian culture only, and data from both
the German and Hungarian cultures for training. cultures, as well as adding
new cultures to the test set: China and the UK, in order to evaluate the
potential of our approach in cross-cultural speech emotion recognition tasks.
In all experiments, we evaluated our model and baseline using a widely adopted
evaluation protocol. The consistent correlation coefficient (CCC) was used as
the evaluation metric. as the evaluation metric.

###  4.3 Implementation Details

The operating system we used in the experimental environment configuration is
Ubuntu 18.04, the development language is python3.9.0, the framework is
Pytorch1.11.0+CUDA 11.1, the CPU is AMD EPYC 7742 64-Core Processor, and the
GPU is NVIDIA A100-SXM4-40GB. The experimental part is divided into three
parts: feature extraction, feature fusion, and emotion recognition. In the
detail section of the feature extraction process, the settings proposed by
Ringeval et al. were employed in conjunction with the experimental apparatus
[44].We adopted a 2-layer LSTM-RNN (64/32 units) as the time-dependent
regressor. We utilized the RMSprop optimizer, implementing a 10% dropout rate
over 50 epochs. The model parameter settings were aligned with the literature
[72]. And the detailed part of feature fusion and emotion recognition, the
network configuration and learning settings of the proposed method are as
follows:

  * •

GRU is used as the backbone network to extract features from the input graph
representation network. GCN is updated during the end-to-end training of the
entire system.

  * •

The end-to-end training of the entire network, and batch normalization is
applied to each fully connected (FC) layer. RMSprop is used as the optimizer,
the initial parameter learning rate was set at 0.005 and after obtaining the
experimental results, it was subsequently increased by 0.001 for each
experiment up to 0.01. This process was repeated a number of times to
determine the best results.

  * •

The final classifier consists of a fully connected layer.

  * •

The method we propose is trained in an end-to-end manner, using the fused
feature vectors as inputs.

###  4.4 Comparison with existing approaches

We examined the performance of the following model. The baseline algorithm
primarily draws inspiration from the studies by Ringeval et al, MFCCs,
eGeMAPS, BoAW-M, BoAW-e and DS [44]. The reason for using the above results as
baseline is that the experimental results of this study are among the better
ones in the speech emotion recognition literature using the SEWA dataset in
the last few years [70, 71].

Baseline results evaluated with CCC for the AVEC 2019 CES(MFCCs, eGeMAPs,
BoAW-M, BoAW-e and DS)[44]; Our approach; best result on the test partition is
highlighted in bold.  Culture MFCCs[73] eGeMAPS[74] BoAW-M [75] BoAW-e [75] DS
[76] Ours _Arousal_ DE+HU 0.326 0.371 0.298 0.398 0.312 0.408 DE 0.389 0.396
0.323 0.434 0.380 0.509 HU 0.236 0.305 0.237 0.291 0.156 0.339 _Valence_ DE+HU
0.187 0.286 0.134 0.352 0.233 0.327 DE 0.344 0.405 0.190 0.455 0.317 0.460 HU
0.017 0.073 0.042 0.135 0.084 0.312 _Liking_ DE+HU 0.144 0.159 0.074 0.138
0.142 0.176 DE 0.159 0.136 0.140 0.003 0.164 0.177 HU 0.115 0.192 -0.027 0.253
0.121 0.240

Table 2 shows the overall performance compared to the baseline. From the
results, we observe that the proposed model consistently outperforms all the
baselines, indicating the effectiveness of the fusion method in speech feature
fusion. More precisely, the method proposed in this paper improves upon the
baseline in the AVEC 2019 Workshop and Challenge: on the SEWA dataset, the CCC
scores for German are improved by 17.28% for arousal and 7.93% for liking, and
for Hungarian, the CCC scores are improved by 11.15% for arousal and 131.11%
for valence. It is noteworthy that the efficacy of our method is less
pronounced in the ’DE+HU’ dataset, with a mere 2.5% improvement observed in
Arousal and a more pronounced decline in Valence compared to the previous
baseline. This may be attributed to the existence of significant linguistic
differences between languages, which are not adequately captured by the
multidimensional edge features. Consequently, the resulting feature
representation may be less accurate. This indicates that by integrating speech
features through GRU and multi-dimensional edge feature learning, potential
feature information can be extracted, thereby enhancing prediction accuracy.

###  4.5 Ablation analysis

Additionally, a series of ablation studies were conducted to investigate
various aspects of the proposed approach. Five distinct methods for fusing
audio features were considered, as illustrated in the ablation model below:

A fusion model based on five audio features, in which the feature vectors are
averaged pairwise (Feature fusion-AVE).

Feature Fusion (FC) using fully connected networks (Feature fusion-FC)

Only use one-dimensional edge to complete the construction of the graph, and
perform feature fusion in vector dimensions (GNN+OD-edge)

Use vector stitching to complete feature fusion, and the output dimension of
the vector changes from $N\times K$ to $N\times(K*5)$(GNN+ST).

Based on the five types of audio features, the output prediction values are
averaged to complete the calculation of the result. (Baseline-ave).

Figure 3: The values on the vertical axis are the average CCC scores of the
three emotional dimensions: arousal, valence, and linking. The horizontal axis
represents the ablation model and our method.

To more accurately assess the extent to which the different modules influence
the final result, three distinct data sources were employed for the purpose of
speech emotion recognition. The first approach utilises solely the task-
specific vertex features derived from the TTF part for linear layer
processing, thereby completing the prediction. The second approach employs the
original vertex features that have not undergone processing by the TTF module,
in conjunction with the initial topology, to complete the AMEF module.
Additionally, the new feature vectors derived from the aforementioned training
are subjected to linear layer processing, thereby completing the prediction.
The training data are subjected to linear layer processing and complete the
prediction. The third method is based on the aforementioned complete method
and uses the task-specific topology and vertex features as inputs. The new
feature vectors obtained after training through the AMEF module are processed
in linear layers and complete the prediction.

CCC Score results: As shown in the Table 3, the results indicate that the
fusion method consistently outperforms the models proposed in ablation studies
in terms of CCC Scores. The fusion method further improves the fusion of all
speech features, most notably by accomplishing dimensionality unification, and
the new features do not increase the computational complexity of the emotion
recognition model as deep spectral features do. To compare data more
intuitively, this study averaged the three emotion classification results
(Arousal, Valence, and Liking) of the ”DE+HU” cross-cultural dataset.

Ablation analysis results evaluated with CCC Score.  Culture Feature Feature
GNN+ GNN+ Baseline- TTF AMEF TTF+AMEF fusion-AVE fusion-FC ST OD-edge ave
Feature Feature Feature _Arousal_ DE+HU 0.169 0.361 0.347 0.361 0.341 0.374
0.372 0.408 DE 0.270 0.398 0.476 0.466 0.384 0.483 0.491 0.509 HU 0.045 0.308
0.291 0.293 0.257 0.324 0.319 0.339 _Valence_ DE+HU 0.144 0.273 0.292 0.328
0.238 0.327 0.323 0.330 DE 0.204 0.416 0.409 0.388 0.342 0.436 0.436 0.460 HU
0.053 0.129 0.300 0.257 0.070 0.306 0.278 0.312 _Liking_ DE+HU 0.032 0.097
0.074 0.167 0.131 0.124 0.127 0.176 DE 0.004 0.086 0.077 0.136 0.120 0.139
0.131 0.177 HU 0.108 0.058 0.124 0.196 0.131 0.240 0.195 0.220

As shown in Figure 3, based on the average value, it can be found that our
fusion method is 13% better than the results GNN+OD-edge methods. This also
reflects that these simple series and parallel methods result in a decrease in
model accuracy due to the neglect of edge features. The GNN+ST method and
GNN+OD-edge method has considered the importance of edge features, so their
results are better than the previously mentioned Feature fusion-FC method that
only considers point features. However, our multi-dimensional edge feature
fusion results are still better than these single-dimensional edge feature
fusion methods, this could be because multi-dimensional edge features can
express key information between different speech features, highlighting the
role of vertex and edge relationship information in speech emotion
recognition.

t-SNE results: As shown in Figure 4, to demonstrate the efficacy of our
method, we utilize t-SNE to illustrate the impact of the aforementioned
approach in the context of an ablation experiment. To this end, we employ a
dataset comprising 40,032 frames of speech belonging to the ”DE+HU” group.
Each data point is represented by a 10-dimensional output vector, which is
subsequently transformed into two dimensions through the use of a
dimensionality reduction tool. This transformation allows us to represent the
horizontal and vertical coordinates of the points in the graph, respectively.
The color of the point is represented by the label value of the point’s
sentiment category (arousal, validity, and liking). The larger the value, the
redder the color, and the smaller the value, the bluer the color. As can be
seen from the figure, our method has better clustering results, as the darker
points (indicating strong emotions) have higher overlapping coordinates in the
plot. In contrast, for methods with suboptimal experimental results, such as
the “Feature fusion-AVE” group, the distribution of points with strong
emotions is more dispersed in the t-SNE plots of arousal and validity.

Figure 4: Ablation experiments on the DE+HU cultural dataset with t-SNE.

Different combinations of audio features results evaluated with CCC Score.
Audio features Ger. + Hun. German Hungarian TTF AMEF TTF AMEF TTF AMEF Arousal
Arousal Arousal MFCC+eGeMAPs+BoAW_e+DS 0.370 0.366 0.443 0.450 0.372 0.420
MFCC+eGeMAPs+BoAW_e 0.283 0.341 0.404 0.399 0.364 0.358 eGeMAPs+BoAW_e 0.363
0.284 0.370 0.374 0.316 0.305 Valence Valence Valence MFCC+eGeMAPs+BoAW_e+DS
0.261 0.292 0.433 0.338 0.198 0.326 MFCC+eGeMAPs+BoAW_e 0.269 0.291 0.414
0.350 0.339 0.272 eGeMAPs+BoAW_e 0.328 0.252 0.352 0.397 0.352 0.178 Linking
Linking Linking MFCC+eGeMAPs+BoAW_e+DS 0.174 0.120 0.183 0.115 0.182 0.199
MFCC+eGeMAPs+BoAW_e 0.134 0.121 0.161 0.097 0.151 0.149 eGeMAPs+BoAW_e 0.151
0.15 0.153 0.161 0.129 0.155

In summary, we found that the TTF feature and the AMEF feature achieved good
results in the results, while the TTF feature was superior to the AMEF
feature. In order to test our research hypothesis that TTF and AMEF can better
capture the best relational cues between different types of speech features,
we conducted an experiment where TTF features and AMEF features were fused
with two, three, and four sets of speech features, respectively.In this
fusion, we discarded the speech features that yielded the worst results: three
(MFCC, BoWA-M, DS), two (BoWA-M, DS), and one (BoWA-M).By examining the CCC
scores of the prediction outcomes, we observed that the scores decreased as
the number of retained feature types diminished.This suggests that even when
the prediction performance of a single type of speech feature is below
average, our method can still identify relational clues among different types
of speech features that are beneficial for enhancing prediction performance.
Consequently, our approach can further improve prediction performance.

##  5 Conclusion

This paper presents a novel approach to speech feature fusion that employs
graphical representation learning and graphical neural networks to model
speech features derived from disparate datasets.The objective of this study is
to investigate the impact of various speech features on speech emotion
recognition in the SEWA dataset. Initially, features are extracted from each
frame of the input audio in order to obtain five features that capture
different aspects of speech emotion information. Subsequently, these features
are processed through a two-layer long short-term memory (LSTM) network to
extract vectors. This paper presents an innovative approach to speech feature
fusion that uses the multidimensional edge feature approach. This approach
combines speech feature vectors into a unified representation that captures
the interactions between features through edge information and the uniqueness
of features through point information. Furthermore, ablation studies were
conducted using different feature merging and fusion methods. The results of
the experiments conducted on the SEWA dataset demonstrate the accuracy of the
method. Furthermore, two sets of experiments, t-SNE and different
combinations, were performed to illustrate the effectiveness of the method.

The main limitations of this study are: (1) the dataset utilised is limited to
the SEWA dataset; and (2) the data features of other modalities, such as
video, have not been explored. In the future, we intend to incorporate a wider
range of speech features and cross-corpus datasets to further optimise our
method.

## Acknowledgments

(Portions of) the research in this paper uses the SEWA Database collected in
the scope of SEWA project financially supported by the European Union’s
Horizon 2020 Research and Innovation Programmer under Grant agreement No.
644869 (SEWA)

This work was supported by Science and Technology Commission of Shanghai
Municipality (22692108300) and the National Natural Science Foundation of
China (71672128).

## References

  * [1] C.-M. Chang, G.-Y. Chao and C.-C. Lee, Enforcing semantic consistency for cross corpus emotion prediction using adversarial discrepancy learning, IEEE transactions on affective computing (2021).
  * [2] S. Melumad, Vocalizing search: How speech technologies alter consumer search processes and satisfaction, Journal of Consumer Research (2023) p. ucad009.
  * [3] M. Chen, Y. Zhang, Y. Li, M. M. Hassan and A. Alamri, Aiwac: Affective interaction through wearable computing and cloud technology, IEEE Wireless Communications 22(1) (2015) 20–27.
  * [4] S. S. Rautaray and A. Agrawal, Vision based hand gesture recognition for human computer interaction: a survey, Artificial intelligence review 43 (2015) 1–54.
  * [5] X. Xia, D. Jiang and H. Sahli, Learning salient segments for speech emotion recognition using attentive temporal pooling, IEEE Access 8 (2020) 151740–151752.
  * [6] J. Liu and H. Wang, Graph isomorphism network for speech emotion recognition., in Interspeech (2021) pp. 3405–3409.
  * [7] K. Wang, N. An, B. N. Li, Y. Zhang and L. Li, Speech emotion recognition using fourier parameters, IEEE Transactions on affective computing 6(1) (2015) 69–75.
  * [8] K. Jagadeeshwar, T. Sreenivasarao, P. Pulicherla, K. Satyanarayana, K. M. Lakshmi and P. M. Kumar, Asernet: Automatic speech emotion recognition system using mfcc-based lpc approach with deep learning cnn, International Journal of Modeling, Simulation, and Scientific Computing 14(04) (2023) p. 2341029.
  * [9] R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar and T. Alhussain, Speech emotion recognition using deep learning techniques: A review, IEEE Access 7 (2019) 117327–117345.
  * [10] P. Mehra and S. K. Verma, Beris: An mbert-based emotion recognition algorithm from indian speech, Transactions on Asian and Low-Resource Language Information Processing 21(5) (2022) 1–19.
  * [11] L. Zhang, J. Han and S. Deng, Unsupervised temporal feature learning based on sparse coding embedded boaw for acoustic event recognition., in INTERSPEECH (2018) pp. 3284–3288.
  * [12] I. McLoughlin, H. Zhang, Z. Xie, Y. Song and W. Xiao, Robust sound event classification using deep neural networks, IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(3) (2015) 540–552.
  * [13] X. Li, Z. Li, H. Qiu, G. Chen and P. Fan, Soil carbon content prediction using multi-source data feature fusion of deep learning based on spectral and hyperspectral images, Chemosphere (2023) p. 139161.
  * [14] J. Ye, Y. Yu, Q. Wang, W. Li, H. Liang, Y. Zheng and G. Fu, Multi-modal depression detection based on emotional audio and evaluation text, Journal of Affective Disorders 295 (2021) 904–913.
  * [15] S. Amiriparian, T. Hübner, V. Karas, M. Gerczuk, S. Ottl and B. W. Schuller, Deepspectrumlite: A power-efficient transfer learning framework for embedded speech and audio processing from decentralized data, Frontiers in Artificial Intelligence 5 (2022) p. 856232.
  * [16] S. Mirsamadi, E. Barsoum and C. Zhang, Automatic speech emotion recognition using recurrent neural networks with local attention, in 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP) (2017) pp. 2227–2231.
  * [17] D. Wang, J. Dong, D. Zhou, X. Wei and Q. Zhang, Speech emotion recognition based on image enhancement, in 2019 IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering (ISKE) (2019) pp. 1126–1132.
  * [18] Y. Li, T. Zhao, T. Kawahara et al., Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning., in Interspeech (2019) pp. 2803–2807.
  * [19] T. L. Nwe, S. W. Foo and L. C. De Silva, Speech emotion recognition using hidden markov models, Speech communication 41(4) (2003) 603–623.
  * [20] M. Jain, S. Narayan, P. Balaji, A. Bhowmick, R. K. Muthu et al., Speech emotion recognition using support vector machine, arXiv preprint arXiv:2002.07590 (2020).
  * [21] Z.-J. Chuang and C.-H. Wu, Emotion recognition using acoustic features and textual content, in 2004 IEEE International Conference on Multimedia and Expo (ICME)(IEEE Cat. No. 04TH8763), Vol. 1 (2004) pp. 53–56.
  * [22] A. Hashem, M. Arif and M. Alghamdi, Speech emotion recognition approaches: A systematic review, Speech Communication (2023) p. 102974.
  * [23] M. S. Hossain and G. Muhammad, Emotion recognition using deep learning approach from audio–visual emotional big data, Information Fusion 49 (2019) 69–78.
  * [24] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz and J. G. Taylor, Emotion recognition in human-computer interaction, IEEE Signal processing magazine 18(1) (2001) 32–80.
  * [25] D. Liu, S. Wang, J. Ren, K. Wang, S. Yin, H. Deng and Q. Zhang, Trap of feature diversity in the learning of mlps, arXiv preprint arXiv:2112.00980 (2021).
  * [26] J. Kim and R. A. Saurous, Emotion recognition from human speech using temporal information and deep learning., in Interspeech (2018) pp. 937–940.
  * [27] S. R. Bandela, S. Siva Priyanka, K. Sunil Kumar, Y. Vijay Bhaskar Reddy, A. A. Berhanu et al., Stressed speech emotion recognition using teager energy and spectral feature fusion with feature optimization, Computational Intelligence and Neuroscience 2023 (2023).
  * [28] B. Paul, S. Bera, T. Dey and S. Phadikar, Machine learning approach of speech emotions recognition using feature fusion technique, Multimedia Tools and Applications (2023) 1–26.
  * [29] Y. Guo, Y. Zhou, X. Xiong, X. Jiang, H. Tian and Q. Zhang, A multi-feature fusion speech emotion recognition method based on frequency band division and improved residual network, IEEE Access (2023).
  * [30] C. Akalya devi and D. Karthika Renuka, Multimodal emotion recognition framework using a decision-level fusion and feature-level fusion approach, IETE Journal of Research (2023) 1–12.
  * [31] Q. Zhang, H. Zhang, K. Zhou and L. Zhang, Developing a physiological signal-based, mean threshold and decision-level fusion algorithm (pmd) for emotion recognition, Tsinghua Science and Technology 28(4) (2023) 673–685.
  * [32] Z. Lian, J. Tao, B. Liu, J. Huang, Z. Yang and R. Li, Conversational emotion recognition using self-attention mechanisms and graph neural networks., in INTERSPEECH (2020) pp. 2347–2351.
  * [33] S. Song, Y. Song, C. Luo, Z. Song, S. Kuzucu, X. Jia, Z. Guo, W. Xie, L. Shen and H. Gunes, Gratis: Deep learning graph representation with task-specific topology and multi-dimensional edge features, arXiv preprint arXiv:2211.12482 (2022).
  * [34] A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * [35] K. Aghajani and I. Esmaili Paeen Afrakoti, Speech emotion recognition using scalogram based deep structure, International Journal of Engineering 33(2) (2020) 285–292.
  * [36] S. Begeer, D. Mandell, B. Wijnker-Holmes, S. Venderbosch, D. Rem, F. Stekelenburg and H. M. Koot, Sex differences in the timing of identification among children and adults with autism spectrum disorders, Journal of autism and developmental disorders 43 (2013) 1151–1156.
  * [37] M. J. Al-Dujaili and A. Ebrahimi-Moghadam, Speech emotion recognition: a comprehensive survey, Wireless Personal Communications 129(4) (2023) 2525–2561.
  * [38] A. Haider, C. Zhang, F. L. Kreyssig and P. C. Woodland, A distributed optimisation framework combining natural gradient with hessian-free for discriminative sequence training, Neural Networks 143 (2021) 537–549.
  * [39] L. Sun, B. Zou, S. Fu, J. Chen and F. Wang, Speech emotion recognition based on dnn-decision tree svm model, Speech Communication 115 (2019) 29–37.
  * [40] R. B. Lanjewar, S. Mathurkar and N. Patel, Implementation and comparison of speech emotion recognition system using gaussian mixture model (gmm) and k-nearest neighbor (k-nn) techniques, Procedia computer science 49 (2015) 50–57.
  * [41] C. Torres-Valencia, M. Álvarez-López and Á. Orozco-Gutiérrez, Svm-based feature selection methods for emotion recognition from multimodal data, Journal on Multimodal User Interfaces 11 (2017) 9–23.
  * [42] E. Lieskovská, M. Jakubec, R. Jarina and M. Chmulík, A review on speech emotion recognition using deep learning and attention mechanism, Electronics 10(10) (2021) p. 1163.
  * [43] S. Hareli, K. Kafetsios and U. Hess, A cross-cultural study on emotion expression and the learning of social norms, Frontiers in psychology 6 (2015) p. 1501.
  * [44] F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner et al., Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition, in Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop (2019) pp. 3–12.
  * [45] L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan and K. P. Truong, The geneva minimalistic acoustic parameter set (gemaps) for speech research and affective computing, IEEE Transactions on Affective Computing 7(2) (2016).
  * [46] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher and T. Eliassi-Rad, Collective classification in network data, AI magazine 29(3) (2008) 93–93.
  * [47] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang and S. Y. Philip, A comprehensive survey on graph neural networks, IEEE transactions on neural networks and learning systems 32(1) (2020) 4–24.
  * [48] Y. Ma, S. Wang, C. C. Aggarwal, D. Yin and J. Tang, Multi-dimensional graph convolutional networks, in Proceedings of the 2019 siam international conference on data mining (2019) pp. 657–665.
  * [49] Y. Dai, L. Shou, M. Gong, X. Xia, Z. Kang, Z. Xu and D. Jiang, Graph fusion network for text classification, Knowledge-based systems 236 (2022) p. 107659.
  * [50] L. Gong and Q. Cheng, Exploiting edge features for graph neural networks, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019) pp. 9211–9219.
  * [51] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio and Y. Bengio, Graph attention networks, arXiv preprint arXiv:1710.10903 (2017).
  * [52] G. Ciano, A. Rossi, M. Bianchini and F. Scarselli, On inductive–transductive learning with graph neural networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 44(2) (2021) 758–769.
  * [53] E. Isufi, F. Gama and A. Ribeiro, Edgenets: Edge varying graph neural networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 44(11) (2021) 7457–7473.
  * [54] Y. Bengio, A. Courville and P. Vincent, Representation learning: A review and new perspectives, IEEE transactions on pattern analysis and machine intelligence 35(8) (2013) 1798–1828.
  * [55] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio and X. Bresson, Benchmarking graph neural networks, Journal of Machine Learning Research 24(43) (2023) 1–48.
  * [56] A. K. McCallum, K. Nigam, J. Rennie and K. Seymore, Automating the construction of internet portals with machine learning, Information Retrieval 3 (2000) 127–163.
  * [57] S. Cai, L. Li, X. Han, Z.-j. Zha and Q. Huang, Edge-featured graph neural architecture search, arXiv preprint arXiv:2109.01356 (2021).
  * [58] C. Xiong, W. Li, Y. Liu and M. Wang, Multi-dimensional edge features graph neural network on few-shot image classification, IEEE Signal Processing Letters 28 (2021) 573–577.
  * [59] S. Song, Z. Shao, S. Jaiswal, L. Shen, M. Valstar and H. Gunes, Learning person-specific cognition from facial reactions for automatic personality recognition, IEEE Transactions on Affective Computing 14(4) (2023) 3048–3065.
  * [60] Z. Shao, S. Song, S. Jaiswal, L. Shen, M. Valstar and H. Gunes, Personality recognition by modelling person-specific cognitive processes using graph representation, in proceedings of the 29th ACM international conference on multimedia (2021) pp. 357–366.
  * [61] W. Wang, J. Gao, X. Yang and C. Xu, Learning coarse-to-fine graph neural networks for video-text retrieval, IEEE Transactions on Multimedia 23 (2020) 2386–2397.
  * [62] C. Luo, S. Song, W. Xie, L. Shen and H. Gunes, Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition, arXiv preprint arXiv:2205.01782 (2022).
  * [63] Y. Hou, S. Song, C. Yu, Y. Song, W. Wang and D. Botteldooren, Multi-dimensional edge-based audio event relational graph representation learning for acoustic scene classification, arXiv preprint arXiv:2210.15366 (2022).
  * [64] A. Mandya, D. Bollegala and F. Coenen, Contextualised graph attention for improved relation extraction, arXiv preprint arXiv:2004.10624 (2020).
  * [65] X. Liu, S. Meng, Q. Li, L. Qi, X. Xu, W. Dou and X. Zhang, Smef: Social-aware multi-dimensional edge features-based graph representation learning for recommendation, in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (2023) pp. 1566–1575.
  * [66] K. Wang, S. C. Han, S. Long and J. Poon, Me-gcn: multi-dimensional edge-embedded graph convolutional networks for semi-supervised text classification, arXiv preprint arXiv:2204.04618 (2022).
  * [67] Y. Tao, Y. Li, S. Zhang, Z. Hou and Z. Wu, Revisiting graph based social recommendation: A distillation enhanced social graph network, in Proceedings of the ACM Web Conference 2022 (2022) pp. 2830–2838.
  * [68] K. Murty and B. Yegnanarayana, Combining evidence from residual phase and mfcc features for speaker recognition, IEEE Signal Processing Letters 13(1) (2006) 52–55.
  * [69] F. Eyben, F. Weninger, F. Gross and B. Schuller, Recent developments in opensmile, the munich open-source multimedia feature extractor, in Proceedings of the 21st ACM international conference on Multimedia (2013) pp. 835–838.
  * [70] F. Ringeval, B. Schuller, M. F. Valstar, J. Gratch, R. Cowie, S. Scherer, S. Mozgai, N. Cummins, M. Schmitt and M. Pantic, Avec 2017: Real-life depression, and affect recognition workshop and challenge, Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge (2017).
  * [71] T. Dang, B. Stasak, Z. Huang, S. Jayawardena, M. Atcheson, M. Hayat, P. Le, V. Sethu, R. Goecke and J. Epps, Investigating word affect features and fusion of probabilistic predictions incorporating uncertainty in avec 2017, in Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, AVEC ’17 (Association for Computing Machinery, New York, NY, USA, 2017) p. 27-35.
  * [72] F. Ringeval, B. Schuller, M. Valstar, R. Cowie, H. Kaya, M. Schmitt, S. Amiriparian, N. Cummins, D. Lalanne, A. Michaud et al., Avec 2018 workshop and challenge: Bipolar disorder and cross-cultural affect recognition, in Proceedings of the 2018 on audio/visual emotion challenge and workshop (2018) pp. 3–13.
  * [73] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan and K. P. Truong, The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing, IEEE Transactions on Affective Computing 7(2) (2016) 190–202.
  * [74] F. Eyben, F. Weninger, F. Gross and B. Schuller, Recent developments in opensmile, the munich open-source multimedia feature extractor, in Proceedings of the 21st ACM International Conference on Multimedia, MM ’13 (Association for Computing Machinery, New York, NY, USA, 2013) p. 835-838.
  * [75] M. Schmitt and B. W. Schuller, openXBOW - Introducing the Passau Open-Source Crossmodal Bag-of-Words Toolkit, arXiv e-prints (May 2016) p. arXiv:1605.06778.
  * [76] S. Amiriparian, M. Gerczuk, S. Ottl, N. Cummins, M. Freitag, S. Pugachevskiy, A. Baird and B. Schuller, Snore sound classification using image-based deep spectrum features, Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2017-August (2017) 3512–3516, Publisher Copyright: Copyright © 2017 ISCA.; 18th Annual Conference of the International Speech Communication Association, INTERSPEECH 2017 ; Conference date: 20-08-2017 Through 24-08-2017.
  * [77] S. Amiriparian, M. Gerczuk, S. Ottl, N. Cummins, M. Freitag, S. Pugachevskiy, A. Baird and B. Schuller, Snore sound classification using image-based deep spectrum features (2017).
  * [78] K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).
  * [79] K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition (2016) pp. 770–778.
  * [80] J. Kossaifi, R. Walecki, Y. Panagakis, J. Shen, M. Schmitt, F. Ringeval, J. Han, V. Pandit, A. Toisoul, B. Schuller et al., Sewa db: A rich database for audio-visual emotion and sentiment research in the wild, IEEE transactions on pattern analysis and machine intelligence 43(3) (2019) 1022–1040.
  * [81] T. Xu, M. Spitale, H. Tang, L. Liu, H. Gunes and S. Song, Reversible graph neural network-based reaction distribution learning for multiple appropriate facial reactions generation, arXiv preprint arXiv:2305.15270 (2023) (2019) 1022–1040.