[1,2]

[1]This document is the results of the research project funded by the National
Science Foundation.

[2]The second title footnote which is a longer text matter to fill through the
whole text width and overflow into another line in the footnotes area of the
first page.

[type=editor, auid=000,bioid=1, orcid=0000-0001-8466-7520]

[1]

[1]

1]organization=Zhejiang University, addressline=West Lake District 38 Zhejiang
University Road, city=HangZhou, postcode=315000, country=China

[style=chinese]

[role=Co-ordinator, ] [2] Data curation, Writing - Original draft preparation

[cor1]Corresponding author [cor2]Principal corresponding author

[fn1]This is the first author footnote. but is common to third author as well.
[fn2]Another author footnote, this is a very long footnote and it should be a
really long footnote. But this footnote is not yet sufficiently long enough to
make two lines of footnote text.

This note has no numbers. In this work we demonstrate $a_{b}$ the formation
Y_1 of a new type of polariton on the interface between a cuprous oxide slab
and a polystyrene micro-sphere placed on the slab.

# Let Segment Anything Help Image Dehaze

Zheyan Jin<EMAIL_ADDRESS>[  Shiqi Chen  Yueting Chen<EMAIL_ADDRESS>Zhihai Xu<EMAIL_ADDRESS>Huajun Feng<EMAIL_ADDRESS>
###### Abstract

The large language model and high-level vision model have achieved impressive
performance improvements with large datasets and model sizes. However, low-
level computer vision tasks, such as image dehaze and blur removal, still rely
on a small number of datasets and small-sized models, which generally leads to
overfitting and local optima. Therefore, we propose a framework to integrate
large-model prior into low-level computer vision tasks. Just as with the task
of image segmentation, the degradation of haze is also texture-related. So we
propose to detect gray-scale coding, network channel expansion, and pre-dehaze
structures to integrate large-model prior knowledge into any low-level
dehazing network. We demonstrate the effectiveness and applicability of large
models in guiding low-level visual tasks through different datasets and
algorithms comparison experiments. Finally, we demonstrate the effect of
grayscale coding, network channel expansion, and recurrent network structures
through ablation experiments. Under the conditions where additional data and
training resources are not required, we successfully prove that the
integration of large-model prior knowledge will improve the dehaze performance
and save training time for low-level visual tasks.

###### keywords:

image dehazing image segmentation large model network training

We discovered and demonstrated the emergence of anti-fog capabilities in
large-scale image segmentation models, which were not innately present in the
dataset or training process but are achieved through large-scale datasets and
large-scale models.

Through grayscale coding and channel expansion, we propose a new framework for
transferring the advantages of the large model to the low-level visual dehaze
task engaging with small-scale data and small models, which accelerate the
adaptation of specific dehaze results.

We carry out a comprehensive experiment for evaluating the proposed method and
comparing the impact of different model sizes on the final dehaze results
under different fog scenarios.

Figure 1: Main pipeline. (a) Segment model: Put haze image input into a large-
scale segmentation model, and output the segmentation result in grayscale
encoding. By utilizing the emergence capability of large models, it can also
handle haze images that have not been trained before. (b) Size comparison:
Both in terms of network size and dataset size, the largest existing dehazing
model and segmentation large model are several orders of magnitude apart.
Therefore, the emergence transparency ability of large models and the image
domain translation ability of the dehazing model can help each other. (c)
Dehaze: grayscale encoded segmentation mask is added to the encoder part of
the image dehazing network through the encoder-decoder structure.

##  1 Introduction

Image dehazing is one of the important computer vision tasks, which removes
the haze interference by using image restoration algorithms, allowing for
better subsequent calculations. As the research on dehazing algorithms
continues to advance, such as the appearance of complex scenes with thick
haze, non-uniform haze, and complex lighting conditions, which small-sized
models are difficult to handle well. The mainstream progress in dehazing
algorithms is mainly based on transformer models with enhanced parameter
counts Song et al. (2022) and image-domain adaptation methods Yi et al. (2022)
respectively. These algorithms often require a large number of datasets.
However, dehazing models often lack reliable real-world data sets, and it is
also difficult to separately train different large models for different
dehazing tasks.

With the continuous development of large-scale models, the emergence of many
abilities beyond data itself in large language and large segmentation models.
These abilities have been achieved through the joint improvement of data
quantity and network scale. However, image dehazing can not achieve
significant improvements through large-scale and high-quality model training.
It is difficult for the low-level vision dehazing model to benefit from the
large model with large data size and continuous rapid development. Therefore,
we hope that image dehazing networks based on small models and small data can
be improved through the large image segmentation models. This allows large
models to enhance the ability of image dehazing.

We have discovered an emergent self-adaptive ability of large-scale image
segmentation models in the image domain. Even if the training dataset does not
contain images specifically for fog, by increasing the network parameter, the
large model can quickly compensate for the performance impact of fog on
segmentation. Therefore, for various fog scenarios, we use different
parameter-count large-scale segmentation models to guide the encoding image
dehazing model. And through the method of grayscale coding and channel
expansion, the small dehazing network can learn the dehazing ability of the
large segmentation model. The above content enables the application of
powerful generalization anti-haze capabilities of large models in small dehaze
networks.

Our main innovations and contributions are as follows:

  * •

We discovered and demonstrated the emergence of anti-fog capabilities in
large-scale image segmentation models, which were not innately present in the
dataset or training process but are achieved through large-scale datasets and
large-scale models.

  * •

Through grayscale coding and channel expansion, we propose a new framework for
transferring the advantages of the large model to the low-level visual dehaze
task engaging with small-scale data and small models, which accelerate the
adaptation of specific dehaze results.

  * •

We carry out a comprehensive experiment for evaluating the proposed method and
comparing the impact of different model sizes on the final dehaze results
under different fog scenarios.

##  2 Related Works

###  2.1 Image dehazing

Image dehazing is a low-level computer vision task. It aims to eliminate the
negative effects of haze scattering on images. Tang et al. (2014) used a
random forest regression model to estimate the amount of haze. He (2009) was
the first to use a dark channel prior to removing haze. Deep learning-based
dehazing methods can be divided into two main categories: calculation of
intermediate parameters and direct end-to-end training. The former estimates
intermediate parameters and then inputs them into an atmospheric degradation
model to calculate the final clean image. Later models tend to learn the
mapping from hazy to clean images directly. Cai et al. (2016) presented a full
convolution neural network (CNN) called DehazeNet for image dehazing. The
model accepts hazy images as input and produces a transmission map as its
output. Ren et al. (2020) proposed a multi-scale deep neural network to
estimate the transmission value. Chen et al. (2019) proposed a threshold
fusion network that utilizes a generative adversarial network (GAN) for image
dehazing, solving the common unreal unreality issue. Song et al. (2022)
applied a larger parameter-count transformer structure to the dehazing field
and achieved better results. Zheng et al. (2023) used the domain changes
between different hazy images from the same dataset for image dehazing.

Compared to normal image dehazing, the thick haze, non-uniform haze, and night
scene haze are more complex, and the research started later. Ancuti et al.
(2019) released a real-world dataset of thick haze outdoors. Ancuti et al.
(2020) released a real-world dataset of non-uniform haze outdoors. Jing et al.
(2014) proposed the NDIM algorithm, which performs colour correction after
estimating the colour features of the incident light. Yu et al. (2015)
distinguished between atmospheric light, haze light, glow light, and different
types of sources of light, and proposed an algorithm based on glow special
processing and night-time different source recognition. Ancuti et al. (2016)
proposed a multi-scale artificial light patch pyramid network to adapt to
night-time haze environments. Jing et al. (2017) believed that the local
maximum intensity of each colour channel in night-time images was mainly
contributed by environmental lighting, and proposed the MRP algorithm with
maximum reflection first-order prior. Zhang et al. (2020) based on scene
geometry, then simulated the light and object reflection rates in two
dimensions. A new method and benchmark testing method for haze rendering
images were proposed.

###  2.2 Segmentation and large language model

Unlike image dehazing, which is a low-level task in computer vision, semantic
segmentation is a high-level task in computer vision to classify each pixel in
an image into a particular class or object. The goal is to generate a dense
pixel segmentation map, where each pixel is assigned to a specific class or
object. Some examples of benchmark datasets for this task include Cityscapes
Cordts et al. (2016), PASCAL VOC Everingham et al. (2010), and ADE20K Zhou et
al. (2017). Models are typically evaluated using metrics such as average
intersection over union (average IoU) and pixel accuracy measures.

Recently, some studies have explored the relationship between image coding
models and large language models. Pretrained large language models on massive
datasets available online have revolutionized the generalization of natural
language processing (NLP) Brown et al. (2020a). These models can generalize to
properties beyond the distribution of the data. These models can even defeat
end-to-end trained fine-tuning models with a certain probability when there is
a lack of dataset. Empirical trends indicate that this behavior improves as
the size of the model increases, and the size of the dataset and the scale of
the training model also significantly affect the performance of the model.
Aligning pairs of text and images online is an excellent work, for example,
CLIP Radford et al. (2021) and ALIGN Jia et al. (2021) use contrastive
learning to train text and image encoders that align both pairs. After
improving, such encoding methods can generalize to zero-sample scenarios for
new image contexts and data. Such coding methods also effectively collaborate
with other modules that enable fundamental image tasks, such as image
generation by DALL·E Ramesh et al. (2021).

###  2.3 The haze effects on semantic segmentation

There have been previous studies that focus on the relationship between image
dehazing and image segmentation. Haze can cause degradation in the
segmentation capabilities of existing models.

Some research focuses on the segmentation performance of the network can be
improved after image dehazing. After passing the dehazing pipeline, the
distribution of the image is closer to the distribution of the normal
segmentation training dataset, thereby improving segmentation and detection
capabilities. Li et al. (2017) found that dehaze images provided better
detection and segmentation results.

Other researchers hope to adapt the data environment of fog conditions by
directly improving detection and segmentation networks. Lee et al. (2022)
proposed a region mapping filter to improve the segmentation capabilities of
images in haze-contaminated scenes. Sakaridis et al. (2018) and Hahner et al.
(2019) respectively proposed a method of training segmentation networks in
haze-contaminated scenes by synthesizing haze-contaminated pipelines.

##  3 Methodology

###  3.1 Large model can overcome haze degradation

We believe that large-scale models can emerge with unexpected capabilities.
For example, Brown et al. (2020b) generative pre trained transformer (GPT)
trained extensively can perform text interaction such as dialogues. Due to
their sufficient parameters and wide distribution of image color and contrast
participation, large-scale segmentation networks can potentially achieve some
degree of domain adaptability to haze. As the previous paragraph mentioned,
other researchers have used methods such as image dehaze or domain adaptation
to achieve the same results. However, the anti-haze capabilities of large
models are automatically obtained, rather than manually optimized. The
existing segmentation large models and dehaze models exhibit several orders of
magnitude differences in their model parameters and training datasets. In
terms of image numbers, the dataset size of segmentation large models is four
orders of magnitude larger than that of dehaze models. Meanwhile, the
parameter size of segmentation large models is two orders of magnitude larger
than that of dehaze networks. The detailed comparison is shown in Fig.1. We
use the segment anything model (SAM) Kirillov et al. (2023) as our large
segmentation model. Its training data volume exceeds 1 billion masks, and
network parameters are close to 3GB, and the occupied video memory is close to
50GB when tested on the dehaze dataset. If large models can perform precise
segmentation under haze conditions, this can assist small dehaze networks in
various types of haze.

Figure 2: The relationship between segmentation and texture in haze images.
The figure displays different types of haze scenarios, input haze images,
segmentation masks, residuals, and no haze images. The segmentation result can
separate similar haze degradation areas under different haze conditions. The
numbers in parentheses in the figure indicate the image position of the
dataset.

###  3.2 Haze, image texture and segmentation

The traditional dehaze formula is as follows,

$I(x)=R(x)t(x)+L(x)(1-t(x)),$ (1)

where $x$ is the position of the pixel, $I(x)$ is the signal received by the
camera pixel, $R(x)$ is the signal emitted by the object itself, $L(x)$ is the
atmospheric global illumination, and $t(x)$ is a transmission rate. The
transmission rate formula is as follows,

$t(x)=e^{-\beta\cdot d(x)},$ (2)

where $d(x)$ is the distance from the object to the camera, $\beta$ is the
attenuation coefficient, and $e$ shows that the attenuation is in exponential.

However, many haze images are not homogeneous and the transmittance is not
linearly related to depth, the $t(x)$ is more complex. Additionally, $L(x)$ is
influenced by the properties of the fog and the light. The variable
environment of fog makes Eq.2 often fail, and it is difficult to directly
calculate $R(x)$ from the input of the sensor $I(x)$ in a linear manner.

If a large model can output precise segmentation results, Eq.(1) can be
transformed into Eq.(3).

$I(x)=R_{mask}(x)R_{mask2texture}(x)t(x)+L(x)(1-t(x)),$ (3)

where the $R_{mask}(x)$ represents the segmentation mask output by a large-
scale segmentation model under ideal conditions, while $R_{mask2texture}(x)$
represents the transition matrix from the segmentation mask to the no haze
image. Since we ultimately need $R(x)$, once we have the guidance of
$R_{mask}(x)$, it will be easier to fit and calculate $R_{mask2texture}(x)$
using Eq.(3). Then calculate $R(x)$ using Eq.(4),

$R(X)=R_{mask}(x)R_{mask2texture}(x).$ (4)

In addition to relying on the dehaze formula, we can also subjectively
understand the relationship between image segmentation and image texture
degradation under fog conditions. As shown in Fig.2, different objects often
have different textures, and different textures will suffer from different
degeneration under the effect of fog. The segmentation mask provides texture
edge information and segments similar degeneration parts together, thus better
guiding small networks to learn similar degeneration processes.

###  3.3 Grayscale coding of segmentation results

The output of the segmentation model is a digital number corresponding to the
segmentation of each pixel in the image. In other segmentation projects, the
segmentation result is often visualized by converting the image to color, with
different colors used to represent different types of segments in an image.
Converting segments in an image to color requires three channels, which
increases the computational complexity of subsequent small models for
dehazing. Therefore, we propose to combine segmentation on a grayscale channel
as shown in Fig.1(a). Since large segmentation models can segment all objects
in the image, there often exist many objects in an image. As a result, an
image is often divided into many parts in segmentation result. According to
our experiments on a dehazing dataset, an image can be segmented into more
than 130 parts at most, while the majority of images have a segmentation
number between 30 and 127 (half of 255). We propose a grayscale coding method,
shown in Algorithm 1, which converts the segmentation result into a grayscale
image. There are several advantages to this method. First, try to fill the
1-255 grayscale space (0 is the result of no segmentation). From dark to
bright, it can be clearly understood the output order of each segmentation
result. There are brighter areas in the segmentation results of the same
image, indicating more segmentation, and the network performance is better.
Using grayscale coding helps to feel the segmentation effect more
subjectively, as shown in Fig.4.

Figure 3: Network structure of image dehaze model with mask. The figure shows
the network structure of different methods compared in our experiments. Each
color block of the network structure has a one-to-one correspondence with
Fig.1.

Input: The output results of the segmentation network $masks$, including
segmentation types $id$ and segmentation pixel distribution data $area$.

Output: Grayscale coding segment mask $segmask$

for  _ $id$, $area$ in enumerate($masks$)_ do

$area$ becomes a matrix with a segmentation result of 1;

if  _$id$ < 127_ then

$segmask=segmask+area\times 2\times(id+1)$;

end if

else

$segmask=segmask+area\times(2\times(255-id)-1)$;

end if

end for

$segmask$ from matrix to single channel image;

return $segmask$;

Algorithm 1 Grayscale coding of segmentation

###  3.4 Image dehaze model with segmask

Common dehazing models typically have an input and output of three channels.
The input fog image has RGB three channels, and the output is the dehazed
image has RGB three channels. To enable the model to perceive grayscale
encoded segmentation masks, we need to expand the input channels to four
channels by adding a grayscale channel and putting the segmentation masks in
the new grayscale channel. The output channels remain at three channels. This
causes the encoder part of the network to have an additional channel and
expand in size, while the decoder part remains unchanged, as shown in
Fig.1(c). It is important to note that the two encoders are not separate, but
rather have different channels.

This expansion channel method can be used by most dehazing networks, but after
increasing the number of channels, the parameters increase, and the original
network structure and the improved structure have different parameters, making
direct comparison unfair. To remove the impact of parameter changes, we
generated grayscale images from the RGB channels into the new grayscale
channel as the baseline. Since the concentration and distribution of haze are
not uniform, the difficulty of different dehazing datasets is different, and
this baseline also serves as a baseline for evaluating the difficulty of the
dataset itself. We placed the fourth channel of the network’s encoder into the
following types, all corresponding network structures are shown in Fig.3.

We will carry out the following types of dehazing network structures with
different masks, where gray refers to the grayscale image from the RGB image.
Among them, small/middle/large refers to different model sizes in the SAM
model. Large refers to vit_h, which is the largest model of the SAM model.
Middle refers to vit_l, medium scale. Small refers to vit_b, which is the
smallest model in SAM.

  * •

haze gray: Baseline. The grayscale of the haze image is directly sent to the
network as a segmask.

  * •

nohaze gray: Directly send the grayscale image of the fog-free image as
segmask to the network. Is the absolute ideal ceiling for experiments. Used to
evaluate data and network capabilities.

  * •

haze + small/middle/large: After the fog image is segmented by different size
models(small/middle/large), it is sent to the network as a segmask.

  * •

nohaze + small/middle/large: After the no-haze image is segmented by different
size models, it is sent to the network as a segmask. Used in evaluating
segmentation network performance and dehazing training.

  * •

dehaze + small/middle/large: Dehazing first, and the result of dehazing image
segmentation by different size models is sent to the network as a segmask.
Used in the actual dehazing test.

Figure 4: Different concentrations of haze and the performance of segmentation
networks. The images in the top row are input images with haze, and the
numbers in the lower right corner are the fog concentration. The images in the
bottom row are gray scale masks obtained after passing the segmentation
network, and the numbers in the lower right corner are the percentage of
segmentations compared to the no-haze case. As the fog concentration
increases, the order of segmentation changes, and the mask that is segmented
last is brighter. It is worth noting that in the red box, even if the fog is
not evenly distributed, the model can still correctly segment each window
position. Large-model segmentation networks can provide texture guidance for
image dehazing.

##  4 Data Preparation

There are many existing dehaze datasets available, and we need to choose data
that covers various scenarios such as indoor and outdoor, thick fog, thin fog,
and non-uniform fog. This will allow us to test the effectiveness of our
method under different fog conditions. At the same time, we should strive to
use real-shot datasets as much as possible to achieve a closer match to real-
world scenarios. We have chosen RESIDE to represent simulated thin fog
outdoors, and REVIDE Zhang et al. (2021) to represent thick fog indoors.
NTIRE2018I Ancuti et al. (2018a) and NTIRE2018O Ancuti et al. (2018b)
represent real-world fog captured indoors and outdoors, while NTIRE2019 Ancuti
et al. (2019) represents outdoor thick fog and NTIRE2020 Ancuti et al. (2020)
represents outdoor non-uniform haze.

Due to the existence of different types of haze, there is a relationship
between the deterioration of different haze types. We analyze the guidance
ability of large segment models for small dehaze models based on different
types of dehaze data. The dehaze result in different fog scenarios can show
the relationship between the degradation of different fog types and the
segmentation mask.

To compare the impact of non-uniform haze and different haze concentrations on
the segmentation network, we used Unreal Engine 5 EPIC (2022b) to render and
generate different concentrations of fog to evaluate the segmentation ability
of the network under haze conditions. The simulation images were rendered
based on the volume fog EPIC (2022a), and the strength of the fog
concentration varied with the intensity of the volume fog. As the fog
concentration increased, the image became increasingly polluted.

Figure 5: Results of different types of segmentation masks for different
dehazing datasets. The figure presents the segmentation results of hazy images
in different dehazing datasets from top to bottom. The selected images
correspond to the dehazing results in Fig.7. The results from left to right
are the hazy input images, different size models segmentation results with
haze and dehazing.

##  5 Experimental Assessment

###  5.1 The haze effects on segmentation

The general detection and segmentation datasets do not use datasets with haze
for training. The dataset used for the SAM large model training does not have
any data related to haze. Therefore, the degradation caused by haze will have
a negative impact on the performance of the segmentation network.

We quantitatively and qualitatively measured the negative impact of haze
concentration on the performance of the segmentation network. Since the
existing dehaze datasets do not contain different concentrations and non-
uniform haze images, we used Unreal Engine 5 EPIC (2022b) to render different
concentrations of smog data. We passed these different concentrations of data
through the segmentation large model separately. The baseline was an image
without fog. As the fog concentration increased from 0 to 0.2, which
corresponds to going from no haze to very thick haze, the detection rate
gradually decreased from 100% to 88%. As shown in the red box area in Fig.4,
due to the ability of large models, in severe conditions of uneven thick fog,
a large-scale segmentation model can still correctly segment each glass of the
window.

Table 1: Quantitatively compare the segmentation performance of models of different sizes under different datasets Method | nohaze small | hurtle | haze middle | nohaze large | haze large |
---|---|---|---|---|---|---
RESIDE | 80 (61%) | 77 (58%) | 131 (99%) | 127 (96%) | 132 (100%) | 130 (98%)
REVIDE | 55 (65%) | 50 (59%) | 82 (96%) | 74 (87%) | 85 (100%) | 75 (88%)
NTIRE2018O | 53 (76%) | 51 (73%) | 69 (98%) | 45 (64%) | 70 (100%) | 47 (67%)
NTIRE2018I | 52 (54%) | 48 (50%) | 88 (92%) | 87 (91%) | 96 (100%) | 90 (94%)
NTIRE2019 | 32 (49%) | 8 (12%) | 61 (94%) | 15 (23%) | 65 (100%) | 16 (25%)
NTIRE2020 | 47 (52%) | 33 (36%) | 94 (103%) | 59 (65%) | 91 (100%) | 57 (63%)

###  5.2 Model size on haze image segmentation

We compared the segmentation ability of different size models for different
haze datasets in Fig.5. The left side represents different types of haze,
including indoor thick fog REVIDE, outdoor non-uniform fog NTIRE2020, outdoor
thick fog NTIRE2019, and thin fog RESIDE. As the size of the segmentation
model increases, the number of segmentation categories and accuracy will also
increase. A mask that appears more white in the image represents more segments
that can be extracted. The right side represents the segmentation ability of
different size segmentation without haze. We found that for RESIDE, a
relatively simple haze data set, the segmentation results with fog are often
better than those without fog. In more difficult haze scenarios, the larger
the model size, the better the segmentation accuracy. It is worth noting that
when the parameter number of the segmentation model increases by one level, it
often directly offsets the effects of haze. For example, the effect of haze +
middle is often better than nohaze + small. This subjectively demonstrates the
role of the emerging anti-haze ability of the segmentation network in various
complex haze scenarios. For the large model performance on RESIDE is poor,
mainly because the small model can already perform effectively in less severe
haze scenarios, which is also repeated in the comparison of dehaze results
later.

Figure 6:  The use of large models can accelerate the dehaze model training.
The above figure shows the convergence results of different data sets when
trained with and without large segment model masks. The below figure shows the
convergence results of the same data set when trained with different sizes of
large segment model.

We also conducted quantitative performance statistics on all dehaze datasets
under different size segmentation models, as shown in Table 1. All data is the
average number of segments per image after the segmentation network segments
the entire dehaze dataset. In general, small segmentation models are
sufficient for simple fog scenes. For moderate-difficulty scenes, increasing
the model scale can overcome the side effects of haze. For extremely difficult
fog scenes, increasing the model scale can alleviate the side effects of haze.
The fog changes on the RESIDE dataset are simple. Simple haze cannot cause the
degradation of large-model segmentation results. The large-model segmentation
results of haze images and clear images are not different. At the same time,
neither size of the segmentation large model shows a significant improvement
in subsequent image dehazing. REVIDE, NTIRE2018O, and NTIRE2018I maintain
similar segmentation results, indicating that fog can cause some impact on
segmentation, but increasing the scale of the segmentation network can
overcome the negative effects of fog. NTIRE2019 and NTIRE2020 show the
uniqueness of these two datasets. NTIRE2019 is a dataset of dense fog, and the
difference between the presence and absence of fog is very dramatic. No matter
how the scale increases, the segmentation model cannot use extremely weak or
even nonexistent signals. However, as the size of the network increases, some
weaker but still detectable signals are still correctly segmented by the large
model. Segmentation results can improve by 100% or more. NTIRE2020 is an
unevenly distributed dehaze dataset. Uneven fog is a challenge for small
dehaze models, but for large segmentation models, uneven fog interference has
almost no effect, as shown in Fig.5. The gradient haze border does not affect
the segmentation effect. At the same time, the performance of large models
directly segmenting uneven fog is also superior to the performance of small
models after dehaze processing.

Figure 7:  Small network dehazing results guided by different types of segmentation masks. All of the images correspond to Fig.5 in the previous paragraph, which is a magnified version of the previous image. From top to bottom, they represent different datasets of different types of fog. From left to right, they show different dehaze results. The red boxes highlight relevant areas that are worth attention. In scenes that are difficult for existing dehaze networks to handle, it can be seen that as the scale of the segmentation model increases, the quality of the dehaze results improves significantly. Table 2: Quantitative results comparing the universality of different datasets (best and second best) Dataset | gray | small | middle | large
---|---|---|---|---
RESIDE (haze) |  33.81(0%) / 0.988 |  33.72(1%) / 0.988 | 32.46(17%) / 0.988 | 32.65(14%) / 0.982
RESIDE (nohaze/dehaze) | 57.12 / 0.999 | 33.67(16%) / 0.988 | 33.70(13%) / 0.988 | 32.84(12%) / 0.988
NTIRE2018I (haze) | 28.42(8%) / 0.938 |  29.05(0%) / 0.946 | 27.97(13%) / 0.961 | 27.73(16%) / 0.956
NTIRE2018I (nohaze/dehaze) | 53.38 / 0.998 |  28.47(7%) / 0.914 | 27.35(22%) / 0.951 | 27.31(22%) / 0.929
NTIRE2018O (haze) | 26.03(1%) / 0.795 | 26.06(1%) / 0.796 | 26.08(1%) / 0.794 | 26.01(2%) / 0.794
NTIRE2018O (nohaze/dehaze) | 49.90 / 0.997 | 26.09(1%) / 0.793 |  26.14(0%) / 0.793 |  26.15(0%) / 0.796
REVIDE (haze) | 22.03(17%) / 0.878 | 22.42(12%) / 0.880 | 22.66(9%) / 0.880 | 22.61(10%) / 0.883
REVIDE (nohaze/dehaze) | 42.66 / 0.998 | 22.58(10%) / 0.873 |  22.86(6%) / 0.882 |  23.41(0%) / 0.888
NTIRE2020 (haze) | 19.90(13%) / 0.658 | 20.52(5%) / 0.652 | 20.54(5%) / 0.652 |  20.69(3%) / 0.659
NTIRE2020 (nohaze/dehaze) | 50.11 / 0.999 | 20.33(7%) / 0.655 | 20.68(3%) / 0.655 |  20.94(0%) / 0.658
NTIRE2019 (haze) | 16.46(17%) / 0.415 | 17.15(8%) / 0.578 | 17.32(6%) / 0.575 |  17.62(2%) / 0.575
NTIRE2019 (nohaze/dehaze) | 49.51 / 0.999 | 17.26(7%) / 0.545 | 17.38(5%) / 0.587 |  17.84(0%) / 0.587

###  5.3 Segmentation mask accelerates training

Due to the ability of large model segmentation networks to emerge with anti-
fog performance, they can enhance the effectiveness of dehaze. We also found
the segmentation mask can improve the convergence speed of dehaze network
training, as shown in Fig.6. We believe that the segmentation results produced
by large models can guide small dehaze networks. Large models segment the same
texture variations into different regions, helping dehaze networks accelerate
training. All of our experimental structures demonstrate this property. As
shown in the blue and red points in the upper figure of Fig.6, compared to the
other curves, the loss of the blue and red points converges slowly,
representing more complex and difficult to process dehaze datasets under fog
conditions. However, red dots fall faster than blue dots, the segmentation
mask can significantly accelerate training. Moreover, this acceleration effect
is also present even on simpler dehaze datasets. Enlargement of the blue and
green lines in the upper figure of Fig.6 gives the result in the lower part.
Different-scale segmentation models can generate slight perturbations, but in
general, the segmentation mask of large models accelerates faster than small
models and the baseline.

###  5.4 Comparison of qualitative dehazing results

Fig.7 shows an example of the results of dehazing by different masks. The
images in the Fig.7 are the local enlargements of the corresponding regions in
Fig.5. The segmentation masks in Fig.5 can also be compared with the
corresponding regions in Fig.7. For simple datasets such as RESIDE, existing
dehaze networks can easily perform well. However, when faced with thick fog
and non-uniform haze scenes, our method shows significant subjective
improvements. As the quality of the segmentation mask improves, the quality of
the dehaze processing also rapidly increases. Both the color and the details
of the edge are restored better. In REVIDE, the green vase in the red box and
the air conditioning socket are better dehaze due to the segment mask
guidance, resulting in better texture details and colors. In NTIRE2020, non-
uniform haze is prone to causing errors in dehaze restoration, such as the
trunk in Fig.7 third row being unable to maintain the same color by haze+gray
baseline. However, with the guidance of the segmentation model, the trunk can
be correctly dehaze. After the pre-dehaze step and segmentation, the texture
detail de-fog network can also perform some restoration of the texture in the
third row of NTIRE2020 red box. In the NTIRE2019 three rows of displayed
images, due to the guidance of the segmentation model, the bicycle frame, the
yellow chair, and the difficult to process branch nodes are also better
resolved. It is worth noting that once the pre-de-fog step is incorrect, it
can also have side effects. For example, in the third row of NTIRE2019
dehaze+small, if the pre-dehaze step is incorrect, like haze+gray, the
segmentation network will also be incorrect and ultimately lead to incorrect
dehaze results. However, from a subjective visual perspective, the dehaze
results benefit from the anti-fog segmentation ability of large models.

###  5.5 Comparison of quantitative dehazing results

We also quantitatively compared the improvement effects of our method under
different levels of fog. Overall, the results were consistent with subjective
perceptions, as shown in Table2. The data was sorted from high to low based on
the haze gray mask baseline’s PSNR, representing the difficulty of different
datasets. The olive color represents the best result, while the teal color
represents the second-best result. It can be clearly seen that the color
results are diagonally distributed, indicating that our method performs better
and improves more for more difficult dehaze datasets and relatively little for
easier dehaze datasets. For the simple RESIDE, there was no improvement
because the dehazing small network itself can perform well. The NTIRE2018I and
NTIRE2018O were relatively moderate, and our method could improve by 1% to 8%.
For medium-level difficulty dehaze datasets, larger-sized segmentation
networks quickly produce saturated effects and cannot continue to provide
effective texture segmentation. For NTIRE2020, NTIRE2019, and REVIDE, which
were more difficult scenarios, our method could improve by 13% to 17%. The
increase in segmentation model scale can improve dehazing performance.

Additionally, we also studied the optimal effect of using large models to
guide dehazing, as shown in the bold part of Table 2. We applied a grayscale
image without haze as a mask to the network for dehazing, and the network
could quickly learn and accept it. It appears that there is much potential to
explore with existing large models. This method is also used in the following
section to evaluate the learning ability of different small dehaze networks.

###  5.6 Mask awareness of dehazing network

To fully demonstrate the network’s learning ability and independence from
prior knowledge of dehazing, we did not choose various custom-designed models
specifically for dehazing, such as Dehazeformer Song et al. (2022) and C2PNet
Zheng et al. (2023). We believe these networks may be more suitable for
certain types of hazy scenes. Instead, we selected general models for image
restoration algorithms with different sizes, represented by Unet Ronneberger
et al. (2015), Uformer Wang et al. (2022), and Restormer Zamir et al. (2022).

The size of the model is crucial for the ability of a large-scale segmentation
model, and it is also important for a small-scale dehaze model to accept the
ability of a large-scale segmentation model. Although large-scale segmentation
models can transmit anti-fog ability to defog networks, different scales of
defog networks exhibit different learning abilities under the same training
resources. As shown in Table3, 10M, 60M, and 100M represent the scale of
different models. For simple dehaze datasets, the improvement caused by the
segmentation mask is not apparent regardless of the scale of the dehaze
network. For more complex scenarios of haze, bigger size and more complex
small-scale dehazing models that have better fitting abilities can often learn
the anti-haze performance of large-scale segmentation networks. Furthermore,
larger-scale dehaze models can better perceive the improvement of large-scale
segmentation models. To fully reflect the learning relationship between large
and small models, the previous demonstrations have all been tested using the
largest-scale dehaze model.

Table 3: Quantitative results comparing the universality of different networks (best and second best) RESIDE | nohaze | haze gray | haze small | de. small | haze mid. | de. midd. | haze large | de. large
---|---|---|---|---|---|---|---|---
Unet10M | 41.79 |  28.05(2%) | 27.78(5%) | 27.65(7%) | 27.91(4%) |  28.22(0%) | 27.43(10%) | 27.32(11%)
Uformr60M | 45.64 |  30.24(4%) | 29.55(13%) |  30.59(0%) | 29.55(13%) | 29.73(10%) | 29.88(8%) | 30.11(6%)
Restor.100M | 57.12 |  33.81(0%) |  33.72(1%) | 33.67(2%) | 32.46(17%) | 33.70(1%) | 32.65(14%) | 32.84(12%)
NTIRE2019 | nohaze | haze gray | haze small | de. small | haze mid. | de. midd. | haze large | de. large
Unet10M | 25.49 | 14.15(1%) | 14.11(1%) | 14.11(1%) | 14.10(1%) | 14.13(1%) |  14.21(0%) |  14.17(0%)
Uformr60M | 26.28 | 15.04(5%) | 15.19(3%) | 15.27(2%) | 15.32(2%) |  15.45(0%) | 15.42(0%) |  15.45(0%)
Restor.100M | 49.51 | 16.46(17%) | 17.15(8%) | 17.26(7%) | 17.32(6%) | 17.38(5%) |  17.62(3%) |  17.84(0%)

##  6 Conclusion

In this article, we discover and prove the emergence of anti-fog ability in
large-dataset, large-parameter segmentation model. We apply this ability to
small dehaze models. First, we gray-scale encode the segmentation results of
the large-scale segmentation model. Then, we input these segmentation masks
into a new channel of the small dehaze model to achieve better dehaze results.
At the same time, we also discover the acceleration effect of large model
masks on the training of the dehaze network. We analyze the impact of
different fog scenarios, dehaze datasets, large-capacity segmentation model
sizes, and small dehaze model sizes on dehaze results. For complex fog scenes,
large models can help small dehaze models better handle them. A wide range of
experimental results show that our method performs well.

Overall, we enable the large semantic segmentation models to help small dehaze
models that cannot be trained with large parameters and large models. This
allows small models to enjoy the various development advantages of large
models, while also without requiring large models to be specifically optimized
for small-model tasks. We propose a new method for large models to help low-
level computer vision, allowing low-level visual tasks to benefit from the
development of large models.

## References

  * Ancuti et al. (2016) Ancuti, C., Ancuti, C.O., Vleeschouwer, C.D., Bovik, A., 2016\.  Night-time dehazing by fusion, in: IEEE International Conference on Image Processing.
  * Ancuti et al. (2019) Ancuti, C.O., Ancuti, C., Sbert, M., Timofte, R., 2019\.  Dense haze: A benchmark for image dehazing with dense-haze and haze-free images.  arXiv .
  * Ancuti et al. (2020) Ancuti, C.O., Ancuti, C., Timofte, R., 2020.  Nh-haze: An image dehazing benchmark with non-homogeneous hazy and haze-free images, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
  * Ancuti et al. (2018a) Ancuti, C.O., Ancuti, C., Timofte, R., Vleeschouwer, C.D., 2018a.  I-haze: a dehazing benchmark with real hazy and haze-free indoor images .
  * Ancuti et al. (2018b) Ancuti, C.O., Ancuti, C., Timofte, R., Vleeschouwer, C.D., 2018b.  O-haze: A dehazing benchmark with real hazy and haze-free outdoor images, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
  * Brown et al. (2020a) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020a.  Language models are few-shot learners.  Advances in neural information processing systems 33, 1877–1901.
  * Brown et al. (2020b) Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Amodei, D., 2020b.  Language models are few-shot learners .
  * Cai et al. (2016) Cai, B., Xu, X., Jia, K., Qing, C., Tao, D., 2016.  Dehazenet: An end-to-end system for single image haze removal.  IEEE Transactions on Image Processing 25, 5187–5198.
  * Chen et al. (2019) Chen, D., He, M., Fan, Q., Liao, J., Hua, G., 2019\.  Gated context aggregation network for image dehazing and deraining, in: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).
  * Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B., 2016.  The cityscapes dataset for semantic urban scene understanding, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223.
  * EPIC (2022a) EPIC, 2022a.  https://docs.unrealengine.com/4.27/en-us/building-worlds/fogeffects/.
  * EPIC (2022b) EPIC, 2022b.  https://docs.unrealengine.com/5.0/en-us.
  * Everingham et al. (2010) Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A., 2010.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision 88, 303–338.
  * Hahner et al. (2019) Hahner, M., Dai, D., Sakaridis, C., Zaech, J.N., Van Gool, L., 2019.  Semantic understanding of foggy scenes with purely synthetic data.  IEEE .
  * He (2009) He, K., 2009.  Single image haze removal using dark channel prior .
  * Jia et al. (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T., 2021\.  Scaling up visual and vision-language representation learning with noisy text supervision, in: International Conference on Machine Learning, PMLR. pp. 4904–4916.
  * Jing et al. (2017) Jing, Z., Yang, C., Shuai, F., Yu, K., Chang, W.C., 2017\.  Fast haze removal for nighttime image using maximum reflectance prior, in: IEEE Conference on Computer Vision Pattern Recognition.
  * Jing et al. (2014) Jing, Z., Yang, C., Wang, Z., 2014.  Nighttime haze removal based on a new imaging model, in: 2014 IEEE International Conference on Image Processing (ICIP).
  * Kirillov et al. (2023) Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al., 2023\.  Segment anything.  arXiv preprint arXiv:2304.02643 .
  * Lee et al. (2022) Lee, S., Son, T., Kwak, S., 2022.  Fifo: Learning fog-invariant features for foggy scene segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18911–18921.
  * Li et al. (2017) Li, B., Peng, X., Wang, Z., Xu, J., Feng, D., 2017\.  Aod-net: All-in-one dehazing network, in: Proceedings of the IEEE international conference on computer vision, pp. 4770–4778.
  * Radford et al. (2021) Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al., 2021\.  Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR. pp. 8748–8763.
  * Ramesh et al. (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I., 2021.  Zero-shot text-to-image generation, in: International Conference on Machine Learning, PMLR. pp. 8821–8831.
  * Ren et al. (2020) Ren, W., Pan, J., Zhang, H., Cao, X., Yang, M.H., 2020\.  Single image dehazing via multi-scale convolutional neural networks with holistic edges.  International Journal of Computer Vision 128, 240–259.
  * Ronneberger et al. (2015) Ronneberger, O., P.Fischer, Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer. pp. 234–241.  URL: http://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a. (available on arXiv:1505.04597 [cs.CV]).
  * Sakaridis et al. (2018) Sakaridis, Christos, Dai, Dengxin, Gool, V., Luc, 2018\.  Semantic foggy scene understanding with synthetic data.  INTERNATIONAL JOURNAL OF COMPUTER VISION .
  * Song et al. (2022) Song, Y., He, Z., Qian, H., Du, X., 2022.  Vision transformers for single image dehazing.  arXiv e-prints .
  * Tang et al. (2014) Tang, K., Yang, J., Wang, J., 2014.  Investigating haze-relevant features in a learning framework for image dehazing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Wang et al. (2022) Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H., 2022.  Uformer: A general u-shaped transformer for image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17683–17693.
  * Yi et al. (2022) Yi, X., Ma, B., Zhang, Y., Liu, L., Wu, J., 2022\.  Two-step image dehazing with intra-domain and inter-domain adaptation.  Neurocomputing 485, 1–11.
  * Yu et al. (2015) Yu, L., Tan, R.T., Brown, M.S., 2015.  Nighttime haze removal with glow and multiple light colors, in: IEEE International Conference on Computer Vision.
  * Zamir et al. (2022) Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., 2022\.  Restormer: Efficient transformer for high-resolution image restoration, in: CVPR.
  * Zhang et al. (2020) Zhang, J., Cao, Y., Zha, Z.J., Tao, D., 2020.  Nighttime dehazing with a synthetic benchmark, in: Proceedings of the 28th ACM International Conference on Multimedia, Association for Computing Machinery, New York, NY, USA. p. 2355–2363.  URL: https://doi.org/10.1145/3394171.3413763, doi:10.1145/3394171.3413763.
  * Zhang et al. (2021) Zhang, X., Dong, H., Pan, J., Zhu, C., Tai, Y., Wang, C., Li, J., Huang, F., Wang, F., 2021.  Learning to restore hazy video: A new real-world dataset and a new method, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9239–9248.
  * Zheng et al. (2023) Zheng, Y., Zhan, J., He, S., Dong, J., Du, Y., 2023\.  Curricular contrastive regularization for physics-aware single image dehazing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5785–5794.
  * Zhou et al. (2017) Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A., 2017.  Scene parsing through ade20k dataset, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633–641.

figs/pic1 Author biography with author photo. Author biography. Author
biography. Author biography.


# Iterative Feature Boosting for Explainable
Speech Emotion Recognition

Alaa Nfissi1,2,4 Wassim Bouachir1,4 Nizar Bouguila2 Brian Mishara3,4
1 Data Science Laboratory, University of Québec (TÉLUQ), Montréal, Canada 2
Concordia Institute for Information Systems Engineering, Concordia University,
Montréal, Canada 3 Psychology Department, University of Québec at Montréal,
Montréal, Canada 4 Centre for Research and Intervention on Suicide, Ethical
Issues and End-of-Life Practices, Montréal, Canada

###### Abstract

In speech emotion recognition (SER), using predefined features without
considering their practical importance may lead to high dimensional datasets,
including redundant and irrelevant information. Consequently, high-dimensional
learning often results in decreasing model accuracy while increasing
computational complexity. Our work underlines the importance of carefully
considering and analyzing features in order to build efficient SER systems. We
present a new supervised SER method based on an efficient feature engineering
approach. We pay particular attention to the explainability of results to
evaluate feature relevance and refine feature sets. This is performed
iteratively through feature evaluation loop, using Shapley values to boost
feature selection and improve overall framework performance. Our approach
allows thus to balance the benefits between model performance and
transparency. The proposed method outperforms human-level performance (HLP)
and state-of-the-art machine learning methods in emotion recognition on the
TESS dataset. The source code of this paper is publicly available at
Iterative-Feature-Boosting-for-Explainable-Speech-Emotion-Recognition.

###### Index Terms:

Feature selection, Supervised learning, Speech emotion recognition, Acoustic
analysis, Explainable AI.

##  I Introduction

Human emotions are complex and essential aspects of human behavior. They
correspond to subjective experiences characterized by physiological,
behavioral, and cognitive changes, and can be influenced by various factors,
such as social interactions, and cultural considerations. Emotions play an
important role in various aspects of human life. Therefore, emotion
recognition and interpretation is considered as a key problem in psychology,
with a wide range of AI applications, including human-computer interaction,
affective computing, and mental disorder detection [1].

To solve the speech emotion recognition (SER) problem, various machine
learning algorithms have been explored, including support vector machines [2],
hidden Markov models [3], and deep neural networks [4]. These algorithms are
trained on large datasets, and use various feature representations, such as
spectral features, prosodic features, and spectral envelope features [5].
Given the multitude of features and their several categories, one of the main
challenges is to find the appropriate feature representation for a given SER
task. In fact, the use of large predefined feature sets often leads to high-
dimensional datasets [6], making it difficult for the model to learn
effectively from data, in addition to increasing computational complexity [7].

In the literature, little attention has been paid to finding relevant features
for SER as most of existing works used principal component analysis (PCA) for
dimensionality reduction [8], while some other works relied on 1D convolutions
through end-to-end deep learning models [9]. In our work, we argue that the
performance of SER systems heavily depends on the used features. We thus
present a comprehensive approach, placing a special emphasis on the feature
extraction and selection process.

The proposed framework comprises three main components : 1) a feature boosting
module guided by the feedback loop of the third component to extract and
select features, 2) a classification module using a supervised classification
model, and 3) an explainability module where the contribution of features to
the classification decision is evaluated using SHapley Additive exPlanations
(SHAP) [10]. This explainability component serves as a feedback mechanism at
the end of each iteration, to continuously refine and boost the feature set in
the first component.

Accordingly, our main contributions can be summarized as follows:

  * •

A new SER approach with an emphasis on feature selection through iterative
feature boosting.

  * •

Incorporation of model explainability and SHAP technique for identifying the
most relevant features for the SER task, as well as for transparency purposes.

  * •

An experimental evaluation of the proposed method by comparison to human-level
performance and state-of-the-art algorithms.

  * •

The source code of our framework for reproducibility and future research on
SER.

##  II RELATED WORKS

Figure 1: The proposed method

Modern SER methods are mostly based on supervised learning approaches. They
can be broadly divided into two categories: traditional machine learning
methods [11] [12], and deep learning methods [13] [14] [9]. Several techniques
related to feature extraction have been explored in previous works. They are
mostly based on handcrafted features, which are designed by incorporating
expert knowledge or domain-specific insights, and using traditional feature
selection techniques. One of the major challenges is handling the high
dimensionality of the data and ensuring that the used features are meaningful
and relevant. This is because a large number of features are often extracted
from signals without considering their practical importance and suitability
for the emotion classification task.

In [11], the authors propose to use Mel Frequency Cepstral Coefficients (MFCC)
for feature extraction and Support Vector Machines (SVM) for classification.
MFCCs are calculated through a series of steps including sampling and
quantization, windowing, discrete Fourier transform, and a Mel filter bank.
The resulting MFCCs are then used for emotion classification using an SVM
classifier. A sensitivity analysis is also conducted to evaluate the impact of
different feature combinations on the classification performance.

In [13], the authors present two-way approach for SER. The first involves the
direct extraction of features from the audio dataset using a combination of
mel-scale related features. Then, PCA is used to reduce data dimensionality
and eliminate correlated variables. The resulting features are then fed into a
deep neural network (DNN) for classification. The authors observed that PCA
allowed to significantly reduce overfitting, which in turn leads to a more
effective training of the DNN. Their second approach is based on using 2D
representation of spectrograms considered as images for classification, which
is then carried out by the VGG16 CNN [15] model retrained on Mel-Spectrogram
images to classify emotions.

In [16], the authors present the robust discriminative sparse regression
(RDSR) approach to deal with feature selection and emotion classification in a
joint learning framework. The RDSR algorithm is designed to select the most
discriminative feature subset from the original high-dimensional feature set.
It uses sparse regression to improve model robustness to outliers and noise,
and introduces a feature selection regularization constraint to select the
most relevant features.

In [17], an SER model based on continuous hidden Markov model (CHMM) was
proposed, as the random generation of states in HMMs allows for statistical
modeling of the sequential nature of the data. The model extracts
33-dimensional feature parameters based on temporal sequence and uses PCA to
reduce the dimensionality of initial feature set. The experimental results
showed that the PCA-CHMM model improves emotion recognition performance
compared to the standard HMM model using the entire feature set.

In [9], the authors propose a hybrid end-to-end deep learning model for
feature extraction and emotion classification. The proposed model consists of
two main components: one-dimensional convolutional neural network (1D-CNN) and
Gated Recurrent Unit (GRU). The 1D-CNN serves for spatial features extraction
through convolutional layers, while the GRU component captures time-
distributed features due to memory and gate principles.

Previous research has shown that extracting efficient acoustic characteristics
is crucial for capturing various emotional aspects of speech in SER. However,
existing works mostly relied on predefined features, without fully
investigating their relevance for SER, and how they can be used to improve
performance. Some deep learning-based works attempted to address the feature
selection issue implicitly through 1D convolutions, while other supervised
learning methods aimed to handle the high dimensionality problem by merely
applying PCA to compute principal components. We present a new approach for
supervised SER with a focus on feature importance and model explainability
over the entire framework.

##  III PROPOSED METHOD

###  III-A Motivation and overview

Both the quality and quantity of used features can significantly impact the
performance of emotion recognition. Using too many irrelevant or redundant
features can lead to overfitting and lack of generalization, while using few
or insufficient features can result in underfitting and poor performance.
Therefore, it is important to carefully select the features that are most
useful for emotion recognition, by considering the voice signal
characteristics that are most indicative of emotions.

Our method is based on domain-specific voice features and comprises three main
components. A feature boosting module is firstly used to compute a preliminary
feature set assumed to be useful for emotion recognition, these features are
iteratively refined afterwards via a feedback mechanism. Then, we identify
optimal feature combinations from the resulting feature set and reduce
dimensionality. Secondly, a classification module formulates the SER task as a
supervised classification problem. Classification decisions are then analyzed
by an explainability module, which utilizes Shapley values to evaluate the
importance of features in the classification decision, thus, to provide
insights into the feature boosting module. By incorporating explainability
into the SER process, we aim to enhance the performance of the model with a
better understanding of the contribution of each feature to the final
decision. The obtained explainability results are then used iteratively via a
feedback mechanism to further boost the feature selection process in the first
module. This is achieved through a feedback loop at the end of each iteration.

###  III-B Method

The proposed method is illustrated in Fig. 1. In the feature boosting module,
we compute a preliminary feature set representation including pitch, energy,
and rhythm related characteristics, which we assume being meaningful for SER.
We also calculate related statistical characteristics including the mean,
median, standard deviation, minimum and maximum [18], resulting in a set of
$n$ initial features. To further improve the performance and interpretability
of our technique, we select $m$ optimal distinct combinations of $p$ features
from the dataset according to following steps. We use PCA to reduce data
dimensionality of each combination $i$ and remove noise by transforming it
using an eigenvector matrix $(A_{i})$ and a corresponding eigenvalues vector
($\lambda_{i}$). Each column of the eigenvector matrix represents a principal
component $(PC_{ij})$ capturing specific data information and determining the
dimension $(r_{i})$ of the reduced subset (see eq. 1 and 2).

$A_{i}=\scalebox{0.72}{\mbox{$\displaystyle\begin{bmatrix}a_{i11}&a_{i12}&..&a_{i1r_{i}}\\\
..&..&..&..\\\ ..&..&..&..\\\ a_{ip1}&..&..&a_{ipr_{i}}\\\ \end{bmatrix}$}}$
(1)

$\lambda_{i}=\scalebox{0.72}{\mbox{$\displaystyle\begin{bmatrix}\lambda_{i1}\\\
..\\\ ..\\\ \lambda_{ip}\end{bmatrix}$}}$ (2)

The percentage of variance $(EV_{ij})$ explained by $j^{th}$ principal
component of the $i^{th}$ combination $(PC_{ij})$ can be evaluated using eq.
3:

$EV_{ij}=\frac{\lambda_{ij}}{\sum_{j=1}^{p}\lambda_{ij}}\times 100$ (3)

where $\lambda_{ij}$ represents the eigenvalue and the amount of variance of
the $j^{th}$ principal component of the $i^{th}$ combination $(PC_{ij})$. The
expression of $(PC_{ij})$ is given by eq. 4, where $X_{ik}$ is the $k^{th}$
feature of the $i^{th}$ combination:

$PC_{ij}=(a_{i1j})X_{i1}+...+(a_{ipj})X_{ip}$ (4)

Therefore, we can determine which features contribute the most to each
principal component. This is done in order to identify the best combination of
features representing information in our dataset. We use a threshold $\alpha$
on the sum of the first $c$ explained variances as a criterion to determine
feature combinations that are most informative. The resulting feature set
consists of the principal components of each selected combination. In this
way, we eliminate redundant and less informative features, and use only the
most relevant ones. This above detailed process is refined iteratively through
the feedback loop of our architecture’s explainability module, which we will
describe later in this section.

In the classification module we compare the performance of $M$ candidate
classification models on the resulting features, these models are detailed in
section IV-A. Our choice of classification models aligns with those commonly
used classifiers in the SER literature. The selected machine learning models
have been widely used in various domains for SER and have been known to
achieve good results [5]. The process of comparing the performance of $M$
different classification models on the initial dataset and the resulting
feature set helps us determine how well our approach performs, to consequently
select the appropriate model. Beyond SER, the framework could be applied to a
wide range of classification problems, where other candidate classification
models could be evaluated.

Figure 2: Boosted features importance

Figure 3: Initial features importance

Class-wise feature importance - Extra Trees classifier: The values on the
x-axis are the mean of absolute values of Shapley values, which represent the
average impact on the model’s output magnitude, where a higher value indicates
a more important feature. The features on the y-axis in Fig. 3 are represented
as principal components, which are the result of applying PCA to the optimal
combinations of selected features using the explained variance threshold. In
Fig. 3, the y-axis represents the initial feature importance.

In the explainability module we incorporate explainable artificial
intelligence (XAI) [19] capabilities into the SER system. Thus, we create a
system that is transparent and understandable in terms of prediction and
decision making. To achieve this, we use the Shapley explanation values to
explain the model’s predictions. Shapley values allow to understand the
contribution of each $PC_{ij}$ in the resulting feature set to a model’s
prediction, which enables us to identify which combination’s principal
components are most important for the emotion recognition. We define the
contribution of each $PC_{ij}$ in the resulting feature set denoted
$\phi_{PC_{ij}}$ as:

$\phi_{PC_{ij}}=\sum_{S\subseteq{11,\dots,mJ_{i}}\backslash{ij}}\frac{1}{|S|}\sum_{s\subseteq
S}(-1)^{|s|+1}f_{ij}(z_{s\cup{ij}})-f(z_{s})$ (5)

where $m$ is the number of combinations, $J_{i}$ is the number of principal
components used from each combination $i$ with $J_{i}\geq 1$, $S$ is a subset
of principal components indices, $z$ is the input vector, and $f(z)$ is the
output of the classification model for input $z$. The first summation term
$(f_{ij}(z_{s\cup{ij}}))$ computes the expected output of the model when
$PC_{ij}$ is included in the subset, while the second term $(f(z_{s}))$
computes the expected output of the model when $PC_{ij}$ is discarded from
$S$. As a result, the difference between the two terms represents the
contribution of $PC_{ij}$ to the output, which we use to conduct a feature
importance analysis. This allows to get insight into how the model works and
what factors are most important.

The process involves an iterative feedback loop where we use the
explainability module to determine the most relevant principal components that
capture the essential information for emotion recognition and eliminate less
important ones. We then incorporate the contribution of the initial features
to each of the relevant principal components to identify the most
participating features to the principal components, and thus, to the
classification decision process. We eliminate features that do not contribute
significantly to the model’s performance and iterate the process until
convergence.

##  IV EXPERIMENTS

###  IV-A Dataset and experimental setup

We use the Toronto Emotional Speech Set (TESS) [20] dataset including 2800
audio recordings of two participants expressing 200 target phrases in
different emotional states. The emotions included in the dataset are anger,
disgust, pleasant surprise, fear, sadness, happiness, and neutral, where each
emotion is represented by 400 recordings.

We first set the sampling rate of the audio data to 16 KHz using a mono-
channel format. This ensures that audio signals are properly processed and
analyzed by our system, as most SER algorithms require specific sampling rates
and number of channels for each audio signal. After going through the feature
extraction and selection process, we use stratified random sampling [21] to
divide both original dataset and boosted features dataset into three
homogeneous groups (or strata): training, validation, and testing. We keep 10%
of the data as unseen to be used for testing, 80% for training, and 10% for
validation. This ensures that the distribution of classes is maintained across
all subsets. Then, we use 10-fold cross-validation [22] to train $M=7$ machine
learning models: Extra Trees (ET), Light Gradient Boosting Machine (LGBM),
Random Forest (RF), Quadratic Discriminant Analysis (QDA), Gradient Boosting
Classifier (GBC), Linear Discriminant Analysis (LDA), and Decision Tree (DT),
on both datasets to select the optimal model for each. By using cross-
validation, our performance evaluation should be less sensitive to data
partitioning.

In order to improve the performance of our best-performing machine learning
models, we use the grid search technique, which involves exhaustively
searching through a specified parameter space to find the best combination of
hyperparameters for a given model [23]. In this way, we are able to fine-tune
the model by adjusting its hyperparameters to increase robustness. We thus
find the optimal set of hyperparameters producing the highest performance on
the validation dataset.

We then assess the performance of the final models on the testing set. The
testing performance is an indicator of how well the model would perform on
unseen data without overfitting the training set. Finally, we use the SHAP
approach in our explainability module to evaluate the feature importance in
the predictions of the optimal model. This allows us to understand how the
model is making its predictions and to identify which features are most
important for determining emotions. For performance evaluation, we use
accuracy, recall, precision, and F1-Score metrics.

###  IV-B Model explainability

One key aspect of our approach is the use of feature boosting module to select
the optimal combinations of features that best capture the variance and
information in our dataset. To demonstrate this, we show importance values in
figures 3 and 3. We can observe the contribution of each feature to the
predicted emotion class. The principal components in Fig. 3 are labeled as
$PC_{\\{combination\\_index\\}\\{PC\\_index\\}}$, i.e. $PC_{ij}$ refers to the
$j^{th}$ principal component of the $i^{th}$ feature combination.
Additionally, it can also help in understanding which principal components are
not important and can be removed without relatively affecting the performance
of the model. By examining Fig. 3, we can determine the principal components
that have the highest impact on the output of our model. Through this
analysis, we can also identify the feature combination that correspond to each
principal component and the contribution of each feature to a principal
component. This leads us ultimately to determine the most relevant features
for our SER decision (see Fig. 3). Thus, we can identify the features that
contribute the most to each of the important principal components, this
information is valuable because it allows us to refine our SER system by
eliminating less relevant features and improving the accuracy of the
classification decision in the next iteration.

We set a threshold $\alpha=0.8$ on the cumulative explained variance by the
first two principal components of each of the feature combinations. This means
that we select only the combinations that explain at least 80% of the data
variance. Thus, we obtain the best 19 combinations representing the data. As
an example, we have found a combination of features that are particularly
informative. This feature set captures 84.56% of the total explained variance
in the first two principal components and 98.22% in the first four.

Figure 4: Biplot of TESS optimal feature combination

The length of the arrows represents the relative importance of each feature in
the dataset, and the angle between the arrows represents the correlation
between the features.

The biplot in Fig. 4 displays the data points on a 2D scatter plot, with the
position of each point representing the values of the first two principal
components of the optimal feature combination data. Additionally, the biplot
also shows the directions and the lengths of the arrows representing the
optimal features in the transformed space. Analyzing the biplot allow us to
understand how the optimal features are related to the principal components as
the direction of the arrow indicates the sign of the contribution (positive or
negative), while the length indicates the magnitude of the contribution. This,
informs us on how these features contribute to the overall variance of the
data. Moreover, the cumulative explained variance can be added to the biplot,
which indicates the percentage of the total variance in the optimal feature
combination data explained by each principal component. This is helpful in
determining the optimal number of principal components to retain when
performing PCA to the selected feature combinations.

###  IV-C Results

####  IV-C1 Comparison with SOTA methods

Table I: Compared methods performance on TESS dataset: best results are in
bold font

Compared methods TESS dataset Test Accuracy (%) F1-score (%) Aggarwal et al.
[13] 97.6 97 Praseetha et al. [14] 95.8 NA Choudhary et al. [24] 97.1 96 Huang
et al. [25] 85 NA Iqbal et al. [11] 97 NA Kapoor et al. [26] 97.5 97.4
Krishnan et al. [12] 93.3 NA Dupuis et al. (HLP) [27] 82 NA Our method 98.7
98.7

The results of our proposed method on the TESS dataset are compared with other
state-of-the-art methods, as presented in Table I. The performance metrics of
these state-of-the-art methods are taken from the original papers. We have two
main evaluation approaches for our method. Firstly, we compare our method
against human-level performance (HLP) on the TESS dataset, as evaluated in
[27]. The authors used 56 human annotators to recognize emotions, and we use
their results to evaluate our model’s performance. Secondly, we compare our
method against machine learning-based SER methods. As previously discussed,
SER methods typically involve two main stages: feature extraction and
classification. Many of the compared methods in the literature use MFCC for
feature extraction [11], [14], [25], [24], while some others use spectrograms
[26] combined with PCA [13] or Empirical Mode Decomposition (EMD) [12]. For
classification, some methods employ traditional machine learning techniques
such as SVM [11] or Latent Dirichlet Allocation (LDA) [12], while others use
deep neural networks [13], [25], [24], [26], [14]. By comparing our results to
the HLP, we can see that our model is able to perform emotion recognition
tasks better than humans according to the results in [27]. Our method also
outperformed the compared machine learning methods, achieving the highest
accuracy and F1-score as shown in Table I.

####  IV-C2 Importance of feature boosting and model explainability

In Table II, we can see the performance of various machine learning models
without feature boosting and model explainability on the initially computed
features. It is clear from the table that the ET classifier performs the best,
as it achieves 95.8% in terms of accuracy, recall, precision and F1-score. The
LGBM also performs well with 95% accuracy. RF and GBC also have a very high
accuracy at 94.6% and 94.3%, respectively. This, indicates that the initially
computed features do indeed contain valuable information for the SER task.
Therefor, we can conclude that the ET classifier and LGBM are the best models
for this dataset, and we can use them to achieve high performance.

The confusion matrix in Fig. 5, values between ”( )”, shows the results of the
ET classifier. The matrix indicates that the model performs well overall, as
can be seen by the high rate of correct predictions on the diagonal elements.
For example, 100% of the actual class ”sad” are correctly predicted as ”sad”.
However, there are also some misclassifications, such as when 5.45% of the
actual class ”surprise” is predicted as ”happy” and 8.33% of the actual class
”happy” is predicted as ”surprise”. This means that ”happy” and ”surprise”
shares some acoustic characteristics.

Table II: Compared models on all computed features in (%): Best results are in
bold font

Model Acc. Recall Prec. F1 ET 95.8 95.8 95.8 95.8 LGBM 95 94.9 95 95 RF 94.6
94.6 94.7 94.5 GBC 94.3 94.3 94.3 94.3 LDA 92.9 93 93.3 92.9 DT 92.9 92.1 92.3
92.1 QDA 83.2 83.3 85.7 82

Table III: Compared models on the constructed dataset using feature boosting
and model explainability in (%): Best results are in bold font

Model Acc. Recall Prec. F1 ET 98.7 98.6 98.7 98.7 LGBM 95.9 96 96.3 96 RF 95
95 95 95 LDA 94.5 94.5 94.6 94.5 GBC 93.6 93.6 93.7 93.7 QDA 92.6 92.7 92.7
92.6 DT 90.3 90.3 90.6 90.2

Table III compares the performance of the same machine learning models using
the boosted feature set and the incorporation of model explainability
feedback. The best performing model in terms of accuracy is the ET classifier
with an accuracy of 98.7% and F1-score of 98.7%. The second best performing
model is LGBM with an accuracy and F1-score of 95.9% and 96%, respectively. RF
also has relatively high accuracy of 95%, and good scores for the other
evaluation metrics. Some other models, have lower accuracy and less favorable
scores for the other evaluation metrics. Few of these low-performing models
appear to lose their effectiveness when feature boosting and model
explainability are employed. This may be due to their need for more complex
datasets in order to achieve reasonable output, resulting in increased
computational complexity. This suggests that these models may be less suitable
for SERs than the models that perform well under the same conditions. In
summary, the ET classifier and LGBM are the best performing models on this
dataset with high accuracy and F1-score.

The confusion matrix in Fig. 5, values between ”[ ]”, shows the results of the
ET classifier, which performs well overall with a high number of correct
predictions on the diagonal elements for each emotion. For example, 100% of
the actual class ”neutral” are correctly predicted as ”neural” and 99.09% of
the actual class ”fear” are correctly predicted as ”fear”. However, there are
also some misclassifications, such as 1.87% of ”disgust” being predicted as
”sad” and 2.78% of ”happy” being predicted as ”surprise”. This means that
”happy” shares characteristics with ”surprise” as well as ”disgust” with
”sad”.

Furthermore, the process of hyperparameters tuning can help to optimize the
performance of a model by finding the optimal values for the hyperparameters
that control the model’s complexity and generalization performance, such as
the case of our compared models. We achieve this by using random grid search
technique, which involves training the model using a range of hyperparameter
values and evaluating the performance of each model using cross-validation
set. The best set of hyperparameters can then be selected based on the model’s
performance on the validation set.

To sum up, the use of feature boosting and model explainability to refine and
boost feature selection results in a significant increase and robustness in
the performance of the models, which can be attributed to several factors. One
of the main reasons is that PCA is able to reduce the dimensionality of the
data, which can help to remove noise and redundancy. This induces a more
compact and informative representation of the data that is better suited for
the machine learning models. Additionally, the explainability module drives
the feedback loop by identifying the most important combinations of features
that are driving the classification. This allows us to iteratively boost the
feature selection process, taking into account new insights and understanding
gained through XAI techniques. This iterative process leads to even better
representation of the speech signal via the most important features for the
SER task, resulting in improved model generalization.

Figure 5: Confusion matrices for the Extra Trees classifier. Values between ”(
)”: without feature boosting and model explainability. Values between ”[ ]”:
our framework with feature boosting and model explainability

##  V Discussion

In this work, the objective is to extract the most relevant features for
accurately detecting emotions in speech. We noticed a lack of consensus in the
existing literature regarding a clear set of features that effectively capture
emotional information from speech signals. This knowledge gap highlights the
importance of our research in addressing this issue and providing valuable
insights into feature selection for SER. Through this study, we make
significant contributions to the field by exploring and identifying the
features that play a crucial role in detecting and distinguishing emotional
states in speech.

Our analysis focused on identifying features with high discriminative power
and informativeness for differentiating between various emotional categories
and how can we explain and interpret that. We employ a rigorous feature
selection process to identify the most relevant features. By using an
iterative feature boosting technique supported by the explainabilty module, we
aim to enhance the discriminative power of the selected features and the SER’s
overall performance.

One of the key features that emerges as highly relevant in our study is MFCC,
which capture the spectral characteristics of speech and have been widely used
in speech analysis tasks. MFCCs are known to effectively capture the
distinctive characteristics and spectral variations associated with different
patterns. Another important feature that we find valuable for SER is pitch or
fundamental frequency (F0). Variations in pitch convey important emotional
cues and can help discriminate between different emotional states. Analyzing
pitch-related features such as pitch contour, pitch range, or pitch dynamics
can provide valuable information for emotion classification. Additionally, we
find that energy and intensity measures play a significant role in capturing
emotional intensity and arousal. These features reflect the overall energy
distribution and loudness of speech, which are closely related to emotional
expressiveness. Furthermore, temporal features such as speech rate or duration
demonstrated relevance in capturing the temporal patterns and dynamics of
emotional speech. The rate at which speech is produced and the duration of
specific speech segments can provide important cues for SER.

One limitation of our study is that it was tested solely on the TESS dataset.
While the results obtained from this dataset are promising, it is important to
validate our approach on multiple datasets and in real-world scenarios to
ensure the generalizability of our findings. Validation on diverse datasets
would provide a more comprehensive assessment of the effectiveness of our
feature boosting approach in different contexts and with different speech
samples. It would enable us to evaluate the robustness and reliability of the
selected features across various data sources, potentially uncovering any
dataset-specific biases or limitations. Furthermore, real-world scenarios
often present additional challenges, such as varying recording conditions,
speaker characteristics, and noise levels. These factors can impact the
performance of the SER system and the relevance of the selected features.

However, it is worth noting that our research provides a comprehensive
analysis of the feature selection process and highlights the rationale behind
selecting specific features. By presenting these findings, we contribute to
the development of a standardized feature set for SER, which can serve as a
foundation for future research in the field. This standardized set of features
will enable researchers to focus on these key features when designing and
implementing robust SERs, ultimately enhancing the accuracy and effectiveness
of trustworthy emotion detection in real-world applications.

##  VI Conclusion

In this study, we presented a new approach for supervised SER based on
acoustic features of the voice and their statistical characteristics values.
Our work highlights the importance of feature selection and the role of
explainability in improving the accuracy of SER. Our method involves several
steps including computing speech features, selecting the optimal feature
combinations, boosting the final feature subset, and applying various machine
learning models to evaluate their performance and fine-tune the best
performing model. Additionally, we incorporate XAI into SER to create a system
that is more understandable, to boost feature selection process via the
feedback loop. To the best of our knowledge, this is the first work
incorporating model explainability into an SER framework. Our work provides a
comprehensive SER approach that aims to balance the benefits of advanced
machine learning techniques with the need for transparency and
comprehensibility. Our future work aims to investigate feature boosting within
deep learning frameworks. Moreover, we aim to generalize our feature boosting
approach for other classification problems, addressing the feature relevance
and high dimensionality problems.

## References

  * [1] P. Shaver, J. Schwartz, D. Kirson, and C. O’connor, “Emotion knowledge: further exploration of a prototype approach.,” Journal of personality and social psychology, vol. 52, no. 6, p. 1061, 1987.
  * [2] T. Seehapoch and S. Wongthanavasu, “Speech emotion recognition using support vector machines,” in 2013 5th International Conference on Knowledge and Smart Technology (KST), pp. 86–91, 2013.
  * [3] T. L. Nwe, S. W. Foo, and L. C. De Silva, “Speech emotion recognition using hidden markov models,” Speech communication, vol. 41, no. 4, pp. 603–623, 2003.
  * [4] H. M. Fayek, M. Lech, and L. Cavedon, “Evaluating deep learning architectures for speech emotion recognition,” Neural Networks, vol. 92, pp. 60–68, 2017.
  * [5] M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” Pattern recognition, vol. 44, no. 3, pp. 572–587, 2011.
  * [6] A. Al-Talabani, Automatic speech emotion recognition-feature space dimensionality and classification challenges.  PhD thesis, University of Buckingham, 2015.
  * [7] K. Ashok Kumar and J. M. Iqbal, “Handling high dimensional features by ensemble learning for emotion identification from speech signal,” International Journal of Speech Technology, pp. 1–15, 2022.
  * [8] P. Fewzee and F. Karray, “Dimensionality reduction for emotional speech recognition,” in 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pp. 532–537, IEEE, 2012.
  * [9] A. Nfissi, W. Bouachir, N. Bouguila, and B. Mishara, “Cnn-n-gru: end-to-end speech emotion recognition from raw waveform signal using cnns and gated recurrent unit networks,” Proceedings of the 21st IEEE International Conference on Machine Learning and Applications (ICMLA 2022). IEEE., 2022.
  * [10] A. E. Roth, The Shapley value: essays in honor of Lloyd S. Shapley.  Cambridge University Press, 1988.
  * [11] M. Z. Iqbal, “Mfcc and machine learning based speech emotion recognition over tess and iemocap datasets,” Foundation University Journal of Engineering and Applied Sciences (2020).
  * [12] P. T. Krishnan, A. N. Joseph Raj, and V. Rajangam, “Emotion classification from speech signal based on empirical mode decomposition and non-linear features,” Complex & Intelligent Systems, vol. 7, no. 4, pp. 1919–1934, 2021.
  * [13] A. Aggarwal, A. Srivastava, A. Agarwal, N. Chahal, D. Singh, A. A. Alnuaim, A. Alhadlaq, and H.-N. Lee, “Two-way feature extraction for speech emotion recognition using deep learning,” Sensors, vol. 22, no. 6, p. 2378, 2022.
  * [14] V. Praseetha and S. Vadivel, “Deep learning models for speech emotion recognition,” Journal of Computer Science, vol. 14, no. 11, pp. 1577–1587, 2018.
  * [15] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 3rd International Conference on Learning Representations (ICLR 2015), 1–14. Computational and Biological Learning Society., 2015.
  * [16] P. Song, W. Zheng, Y. Yu, and S. Ou, “Speech emotion recognition based on robust discriminative sparse regression,” IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 2, pp. 343–353, 2020.
  * [17] X. Ke, B. Cao, J. Bai, Q. Yu, and D. Yang, “Speech emotion recognition based on pca and chmm,” in 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), pp. 667–671, 2019.
  * [18] S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, Z. Deng, S. Lee, S. Narayanan, and C. Busso, “An acoustic study of emotions expressed in speech,” in Eighth International Conference on Spoken Language Processing, 2004.
  * [19] W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.-R. Müller, Explainable AI: interpreting, explaining and visualizing deep learning, vol. 11700.  Springer Nature, 2019.
  * [20] K. Dupuis and M. K. Pichora-Fuller, “Toronto emotional speech set (tess),” Scholars Portal Dataverse, 2010.
  * [21] H. Aoyama, “A study of stratified random sampling,” Ann. Inst. Stat. Math, vol. 6, no. 1, pp. 1–36, 1954.
  * [22] P. Refaeilzadeh, L. Tang, and H. Liu, “Cross-validation.,” Encyclopedia of database systems, vol. 5, pp. 532–538, 2009.
  * [23] L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020.
  * [24] R. R. Choudhary, G. Meena, and K. K. Mohbey, “Speech emotion based sentiment recognition using deep neural networks,” in Journal of Physics: Conference Series, vol. 2236, p. 012003, IOP Publishing, 2022.
  * [25] A. Huang and P. Bao, “Human vocal sentiment analysis,” 2019.
  * [26] S. Kapoor and T. Kumar, “Fusing traditionally extracted features with deep learned features from the speech spectrogram for anger and stress detection using convolution neural network,” Multimedia Tools and Applications, pp. 1–22, 2022.
  * [27] K. Dupuis and M. K. Pichora-Fuller, “Recognition of emotional speech for younger and older talkers: Behavioural findings from the toronto emotional speech set,” Canadian Acoustics, vol. 39, no. 3, pp. 182–183, 2011.