normalsize[table]font=normalsize

# A Survey on Physiological Signal Based Emotion Recognition

Zeeshan Ahmad, Graduate Student Member, IEEE, Naimul Khan, Senior Member, IEEE

###### Abstract

Physiological Signals are the most reliable form of signals for emotion
recognition, as they cannot be controlled deliberately by the subject.
Existing review papers on emotion recognition based on physiological signals
surveyed only the regular steps involved in the workflow of emotion
recognition such as preprocessing, feature extraction, and classification.
While these are important steps, such steps are required for any signal
processing application. Emotion recognition poses its own set of challenges
that are very important to address for a robust system. Thus, to bridge the
gap in the existing literature, in this paper, we review the effect of inter-
subject data variance on emotion recognition, important data annotation
techniques for emotion recognition and their comparison, data preprocessing
techniques for each physiological signal, data splitting techniques for
improving the generalization of emotion recognition models and different
multimodal fusion techniques and their comparison. Finally we discuss key
challenges and future directions in this field.

###### Index Terms:

Data annotation, physiological signals, data variability, emotion models,
challenges, review.

##  I Introduction

Emotion is a psychological response to some external stimulus and internal
cognitive processes, supported by a series of physiological activities going
on in human body. Thus, emotion recognition is a promising and challenging
work area which enables us to recognize the emotions of a person for stress
detection and management, risk prevention, mental health, and interpersonal
relations.

††© 2022 IEEE. Personal use of this material is permitted. Permission from
IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional
purposes, creating new collective works, for resale or redistribution to
servers or lists, or reuse of any copyrighted component of this work in other
works.

Emotion recognition is an emerging research area due to its numerous
applications in our daily life. Applications include areas such as developing
models for inspecting driver emotions [1], health care [2] [3], software
engineering [4] and entertainment [5].

Different modalities of data can be used for emotion recognition. They are
commonly divided into behavioral and physiological modalities. Behavioral
modalities includes emotion recognition from facial expressions [6] [7] [8]
[9] [10], from gestures [11] [12] [13] and from speech [14] [15] [16], while
physiological modalities include emotion recognition from physiological
signals such as Electroencephalogram (EEG), Electrocardiogram (ECG), Galvanic
skin response (GSR), Electrodermal Activity (EDA) and so forth [17] [18] [19]
[20] [21].

Behavioral modalities can be effectively controlled by user and the quality of
expressing them may be significantly influenced by personality of the user and
the current environment of the subject [22]. On the other hand, physiological
signals are continuously available and cannot be controlled intentionally or
consciously. The commonly used physiological signals are ECG, EEG and GSR
signals.

Many papers exist in literature where review on emotion recognition using
physiological signals is presented.

In [23], a comprehensive review on physiological signal-based emotion
recognition was presented that includes emotion models, emotion elicitation
methods, the published emotional physiological datasets, features,
classifiers, and the frameworks for emotion recognition based on the
physiological signal. In [24], literature review provides a concise analysis
of physiological signals and instruments used to monitor them, emotion
recognition, emotion models and emotional stimulation approaches. The authors
also discuss selected works on emotional recognition by physiological signals
in wearable devices. In [25], different emotion recognition methods using
physiological signals by machine learning techniques were explained. This
paper also reviewed different stages like data collection, data processing,
feature extraction and classification models for each recognition method. In
[26] recent advancements in emotion recognition research using physiological
signals, including emotion models and stimulation, preprocessing, feature
extraction and classification methodologies were presented. In [27], the
current state-of-the art of emotion recognition was presented. This paper also
presented the main challenges and future opportunities that lie ahead, in
particular for the development of novel machine learning (ML) algorithms in
the context of emotion recognition using physiological signals.

Existing review papers on emotion recognition based on physiological signals
looked for only the regular steps involved in the workflow of emotion
recognition such as preprocessing, feature extraction, and classification but
did not discuss the set of important challenges that are particular to emotion
recognition. None of the aforemtioned works elaborated the problems during
data annotion, the different kinds of data annotion methods for emotion
recognition and advantages/disadvantages of each method. Furthermore, in the
current review papers, the data preprocessing techniques are bundled together
and presented as generic techniques for any physiological signal. In our
opinion, each physiological signal presents its own unique set of challenges
when it comes to preprocessing, and the preprocessing steps should be
discussed separately. Moreover, inter-subject data variability has huge impact
on emotion recognition. The existing reviews neither discuss this effect nor
provide recommendations to reduce inter-subject variability. In addition to
this, the comparison of data splitting techniques, such as subject independent
and subject dependent, is not provided for better emotion recgonition and
generalization of the trained classification models. Also, the comparison and
advantages of multimodal fusion methods are not provided in these review
papers.

Figure 1: Plutchik Wheel of Discrete Emotions [28] Figure 2: 2D Valence-
arousal Model Figure 3: 3D Emotion Model

To adress the above shortcomings of the existing reviews, in this paper, we
are focussed on the missing but essential elements of emotion recognition.
These are

  1. 1.

Important data annotation techniques for emotion recognition and their
comparison.

  2. 2.

Data preproceessing techniques for each physiological signal.

  3. 3.

Effect of Inter-subject data variance on emotion recognition.

  4. 4.

Data splitting techniques for improving the generalization of emotion
recognition models.

  5. 5.

Different multimodal fusion techniques and their comparison.

This paper is organized as follows: in section II, the emotion models are
explained. In section VI, the effect of inter-subject data variance on emotion
recognition is described. In section IV, the data annotation techniques for
emotion recognition are illustrated. Section V demonstrates the data
preprocessing technique for each physiological signal. In section VII, data
splitting techniques for improved generalization of emotion recognition models
are analyzed. Section VIII provides the details of different multimodal fusion
techniques for emotion recognition. Section IX summarizes future challenges of
emotion recognition using physiological signals and finally we conclude the
paper in section X.

##  II Emotion Models

Over the past decades, different emotion models have been proposed by
researchers based on the quantitative analysis and the emergence of new
emotions categories. Based on the recent research, emotional states can be
represented with two models: discrete and multidimensional.

###  II-A Discrete or Categorical Models

Discrete models are the most commonly used models because they contain a list
of distinct emotion classes that are easy to recognize and understand. Ekman
[29] and Plutchik [28] are amongst those scientists that present the concept
of discrete emotion states. Ekman briefed that there are six basic emotions
such as happy, sad, anger, fear, surprise, and disgust, and all the other
emotions are derived from these six basic emotions. Plutchik presented a
famous wheel model to describe eight discrete emotions. These emotions are
joy, trust, fear, surprise, sadness, disgust, anger and anticipation, as shown
in Fig. 1. The model describes the relations among emotion concepts, which are
analogous to the colors on a color wheel. The cone’s vertical dimension
represents intensity, and the circle represents degrees of similarity among
the emotions. The eight sectors are designed to indicate that there are eight
primary emotion dimensions and rest of the motion dimensions are derived from
them.

###  II-B Continuous or Multidimensional Models

All emotional states listed in discrete set of emotions cannot be confined by
a single word and they need a range of intensities for description. For
instance, a person may feel less or more excited OR become less or more afraid
in response to a particular stimulus. Thus, to cover the range of intensities
in emotions, multidimensional models such as 2D and 3D are proposed. Amongst
2D models, the model presented in [30] is the famous model that describes the
emotion along the dimensions of High Arousal (HA) and Low Arousal (LA), and
High Valence (HV) and Low Valence (LV) as shown in Fig. 2. The 2D model
classifies emotions based on two dimensional data consisting of valence and
arousal value. On the other hand, the 3D model deals with valence, arousal,
and dominance. Valence indicates the level of pleasure, Arousal indicates the
level of excitation and Dominance indicates the level of controlling or
dominating emotion. The 3D model is shown in Fig. 3.

Database Modalities Stimulai Subjects Sampling Rate Emotion States AMIGOS [31]
EEG, ECG,EDA Video clips 40 Not Provided Anger, Fear, Sadness, Disgust,
Neutrality, Surprise, Happiness ASCERTAIN [32] EEG, ECG, EDA Movie clips 58
EDA at 100Hz EEG at 32Hz Valence (-3–3), Arousal (0–6) BIO-VID-EMO DB [33]
ECG, EMG,SC Film clips 86 512Hz valence, arousal, amusement, sadness, anger,
disgust, fear DEAP [34] EEG, EDA, EMG, PPG, EOG, RSP Music Videos 32 512Hz
Valence (1–9), Arousal (1–9), Dominance (1–9), Liking (1–9), Familiarity (1–5)
DREAMER [35] EEG, ECG Film clips 23 Not Provided Anger, Fear, Sadness,
Disgust, Calmness, Surprise, Amusement, Happiness, Excitement MAHNOB-HCI [36]
EEG, ECG, EDA, RSP, SKT Video clips 27 256Hz Anger, Anxiety, Fear, Sadness,
Disgust, Neutrality, Surprise, Amusement, Joy MPED [37] EEG, ECG, EDA, RSP
Video clips 23 1000Hz Anger, Fear, Sadness, Disgust. Neutrality, Funny, Joy
SEED [38] EEG Film clips 15 1000Hz Negativity, Neutrality, Positivity

TABLE I: Publicly Available Datasets In Alphabetical Order for Physiological
Signals Based Emotion Recognition

##  III Databases

Several datasets for physiological based emotion recognition are publicly-
available,like AMIGOS [31], ASCERTAIN [32], BIO VID EMO DB [33], DEAP [34],
DREAMER [35], MAHNOB-HCI [36], MPED [37], SEED [38] as shown in Table I. All
datasets, other than SEED, are multimodal and provide more than one modalities
of physiological signals. As can be seen, some of these datasets utilize the
continuous emotion model, while the others use the discrete model. Most of
these datasets are unfortunately limited to a small number of subjects due to
the elaborate process of data collection. Some details of these datasets such
as annotation and preprocessing are discussed in the following sections.

##  IV Data Annotation

Data annotation is amongst the most important steps after data acquisition and
eventually for emotion recognition. However, this subject is not addressed in
detail in exisiting literature. For better discussion on this matter, we
divide the data annotation procedure into discrete and continuous data
annotation.

In discrete annotation, participants are given questionnaire after the
stimulai and they are asked to rate their feelings between some scores ranging
from 1 to 5 or from 0 to 9 and then the final annotations are decided as shown
in Fig 4.

Figure 4: Steps involving discrete annotation

On the other hand in continuous annotations, the participants are required to
continuously annotate the data in real time using some human computer
interface (HCI) mechanism as shown in Fig 5.

Figure 5: Continuous annotation using HCI mechanism [39]

In sections IV-A and IV-B, we provide the review on discrete and continuous
annotation.

###  IV-A Discrete Annotation

In [37], three psychological questionnaires PANAS [40], SAM [41] and DES [42],
were used for data annotation of seven emotion categories such as joy, funny,
anger, sad, disgust, fear and neutrality. Besides questionnaires, T-test was
also conducted to evaluate the effectiveness of the selected emotion
categories.

In [43] and [32], participants were briefed on details of the experiments
including the principal experimental tasks and later on the self-report
questionnaires were filled by the participants according to their experiences
during stimuli. These questionnaires were finally used for data annotation. In
[19], for automatic ECG Based emotion recognition, music is used as stimuli
and in the self-assessment stages, the participants were asked to indicate how
strongly they had experienced the corresponding feeling during stimulus
presentation and then participants gave marks ranging from 1 to 5 for each of
the nine emotion categories.

In [44], the questionnaire, divided into three parts, one for each activity,
in each of which the participants had to select the emotions they felt before,
during, and after the activity, was given to the participants. Based on the
participants marking, 28 emotions are arranged on a 2-dimensional plane with
Valence and Arousal at each axis as shown in Fig 2, according to the model
proposed in [45]. In [46], the data was collected in VR environment. The data
annotation was based on the subjects’ responses to the two questions regarding
level of excitement and pleasantness which were graphed on a 9 square grid, a
discretized version of the arousal-valence model [47]. Each grid shows how
many subjects reported feeling the corresponding level of “pleasantness” and
“excitement” for each VR session. The valence scale was measured as
“Unpleasant”, “Neutral”, or “Pleasant” (from left to right) and the arousal
scale was measured as “Relaxing”, “Neutral”, or “Exiting” (from bottom to
top). The results are very asymmetrical with the majority of the sessions
rated as “Exciting” and “Pleasant”.

###  IV-B Continuous Annotation

In [39], participant viewed a 360∘ videos as stimuli and they rated emotional
states (valence and arousal) continuously using the joystick. It is observed
that collecting continuous annotations can be used to evaluate the performance
of fine-grained emotion recognition algorithms such as weakly supervised
learning or regression. In [48], the authors claim that the lack of continuous
annotations is the reason why they failed to validate their weakly-supervised
algorithm for fine-grained emotion recognition since the continuous labels
allow the algorithms to learn the precise mappings between the dynamic
emotional changes and input signals. Moreover, if only discrete annotations
are available, ML algorithms can overfit because the discrete labels represent
only the most salient or recent emotion rather than the dynamic emotional
changes that may occur within stimuli [49]. A software for continuous emotion
annotation is introduced in [50] and is called EMuJoy ( Emotion measurement
with Music by using a Joystick ). It is better than Schubert’s 2DES software
[51]. It helps subjects to generate self reports for different media in real
time with a computer mouse, joystick, or any other human–computer interface.
Using joystick has an advantage, because joysticks have a return spring to
automatically realign them in the middle of the space. This contrasts with
Schubert’s software, where mouse is used. In [52], stimuli videos were also
continuously annotated along valence and arousal dimensions. Long-short-term-
memory recurrent neural networks (LSTM-RNN) and continuous conditional random
fields (CCRF) were utilized in detecting emotions automatically and
continuously. Furthermore, the effect of lag on continuous emotion detection
is also studied. The database is annotated using a joystick and delays from
250ms up to 4 seconds were in annotations. It is oberved that 250 ms increased
the detection performance whereas longer delays deteriorated it. This shows
the advantage of using joystick over mouse. Authors in [53] analyzed the
effect of lag on continuous emotion detection on SEMAINE database [54] They
found a delay of 2 seconds will improve their emotion detection results.
SEMAINE database is annotated by Feeltrace [55] using a mouse as annotation
interface. Thus the observations in [53] also show that the response time of
joystick is less than mouse. Another software named DARMA is introduced in
[56] for continuous measurement system that synchronizes media playback and
the continuous recording of two-dimensional measurements. These measurements
can be observational or self-reported and are provided in real-time through
the manipulation of a computer joystick.

Touch events are dynamic and can be bidirectional i.e 2D along Valence-arousal
plane. During Continuous annotation, participants are instructed to annotate
their emotion experience by moving the joystick head into one of the four
quadrants. The movement of the joystick head maps the emotions into a 2D
Valence-arousal plane, in which the x axis indicates valence while the y axis
indicates arousal. Since the nature of emotions is time-varying, therefore,
during annotation participants could lose the control over the speed i.e the
movement of a joystick which could lead collection of less precise ground
truth labels. Thus, the training of participants on HCI mechanism and and
study of lag for continuous emotion detection are two important factors for
precise continuous annotation.

###  IV-C Summary of the section

There is no definite or clear-cut advantage of discrete annotation on
continuous annotation and vice versa. However, the choice of annotation method
depends on the application. For example, In Biofeedback systems, participants
learn how to control physical and psychological effects of stress using feed
back signals. Biofeedback is a mind-body technique that involves using visual
or auditory feedback to teach people to recognize the physical signs and
symptoms of stress and anxiety, such as increased heart rate, body
temperature, and muscle tension. Since stress level changes during biofeedback
mechanism, therefore, real time continuous annotation is more important in
this case [57]. The choice of annotation also depends length of the data, the
number of emotion categories, number of participants, feature extraction
method and the choice of classifier.

Figure 6: (a) Raw GSR: rest and motion phases. Signals corresponding to the
movements involving the right hand are delimited by red lines.(b) GSR
decomposition based on EMD, IMF6 and its respective residue [58]. Figure 7:
Transformation of ECG signal into GAF, RP and MTF Images

##  V Data Preprocessing

Physiological signals such as electroencephalogram (EEG), electrocardiogram
(ECG) and galvanic skin response (GSR) are time-series and delicate signals in
the raw form. During acquisition, these physiological signals are contaminated
by numerous factors such as line interference of 50 Hz or 60 Hz,
electromagnetic interference, noise and baseline drifts and different
artifacts due to body movements and different responses of participants to
different stimuli. The bandwidths of the physiological signals are different.
For instance, EEG signals lie in the range of 0.5Hz to 35 Hz, ECG signals are
intensive in the range of 0 to 40 Hz and GSR signals are mainly concentrated
in the range of 0 to 2 Hz. Since the strength of physiological signals is rich
in different frequency ranges, therefore, different physiological signals
required different preprocessing techniques. In this section, we review the
preprocessing techniques used for physiological signals.

###  V-A EEG Preprocessing

In addition to noise, the EEG signals are affected by Electrooculography (EOG)
with frequency below 4 Hz. This artifact is caused by the facial muscle
movement or eye movement [59], [60]. Different methods that have been used for
EEG preprocessing or cleaning include rejection method, linear filtering,
statistical methods such as linear regression and Independent Component
Analysis (ICA).

####  V-A1 Rejection Method

Rejection method involves both manual and automatic rejection of epochs
contaminated with artifacts. Manual rejection needs less computation but it is
more laborous than automatic rejection. In automatic rejection pre-determined
threshold is used to remove the artifact-contaminated trials from the data.
The common disadvantages associated with the rejection method are the sampling
bias [61] and loss of valuable information [62].

####  V-A2 Linear Filtering

Linear filtering is simple and easy to apply and is beneficial mostly when
artifacts located in certain frequency bands do not overlap with the signal of
interest For example, low-pass filtering can be used to remove EMG artifacts
and high-pass filtering can be used to remove EOG artifacts [38]. However,
linear filtering flops when EEG signal and the EMG or EOG artifacts lie in the
same frequency band.

####  V-A3 Linear Regression

Linear regression using least square method has been used for EEG signal
processing for removing EOG based artifacts. In Linear regression using least
square criteria, residual is calculated by subtracting the EOG signal from the
EEG signal and then this residual is optimized to get the cleaned EEG signal
[63]. This method is not suitable for EMG based artifact removal because EMG
data is collected from multiple muscles groups and therefore EMG data has no
single reference.

####  V-A4 Independent Component Analysis

ICA is amongst those methods that do not require reference artifacts for
cleaning the EEG signal and thus making the components independent [64]. The
disadvantage associated with ICA is that it requires prior visual inspection
to identify the artifact part.

###  V-B ECG Preprocessing

ECG signal is usually corrupted by various noise sources and artifacts. These
noises and artifacts deteriorate the signal quality and affect the detection
of QRS.

####  V-B1 Filtering

Filtering is the simplest method of removing noise and artifacts from the ECG
signal and to improve the signal to noise ratio (SNR) of the QRS complex. In
ECG signal, the three critical frequency regions include the very low
frequency band below 0.04 Hz, the low frequency band (0.04 to 0.15 Hz), and
the high frequency band (0.15 to 0.5 Hz) [65]. Thus the commonly used filters
for ECG are high pass filter, low pass filter and band pass filter.

High-pass filters allow only higher frequencies to pass through them. They are
used to suppress low-frequency components the ECG signal. Low frequency
components include motion artifact, respiratory variations, and baseline
wander. In [66], high pass filters with cut-off frequency of 0.5 Hz was
selected to waive off the baseline wander.

Low-pass filters are usually employed to eliminate high-frequency components
from the ECG signal. These compoents include noise, muscle artifacts and
powerline interference [67].

Since bandpass filters remove most of the unwanted components from the ECG
signal, therefore, it is extensively used to preprocess ECG. The structure of
Band-pass filter comprises of both a high-pass and low-pass filter. In [68],
bandpass filter is used to remove muscle noise and baseline wander from ECG.

####  V-B2 Normalization

In ECG, the amplitude of QRS complex amplitude increase from birth to
adolescence and then begins to decrease afterward [69]. Due to this, ECG
signals suffer from inter-subject variability. To overcome this problem,
amplitude normalization is performed. In [70], a method of normalizing ECG
signals to a standard heart rate to lower the false rate detection, was
introduced. The commonly used normalization techniques are min-max
normalization [71], maximum division normalization [72] and Z-score
normalization [73].

###  V-C GSR Preprocessing

There is still no standard methods for GSR preprocessing. In this section, we
are disscussing a few methods to deal with noise and artifacts present in raw
GSR signal.

####  V-C1 Empirical Mode Decomposition

Empirical Mode Decomposition (EMD) is the technique that best addresses the
nonlinear and nonstationary nature of the signal while removing noise and
artifact and was introduced in [74] as a tool to adaptively decompose a signal
into a collection of AM–FM components. The EMD relies on a fully data-driven
mechanism that does not require any a priori known basis like Fourier and
wavelet-based. EMD decomposes the signal into a sum of intrinsic mode
functions (IMFs). An IMF is defined as a function with equal number of extrema
and zero crossings (or at most differed by one) with its envelopes, as defined
by all the local maxima and minima, being symmetric with respect to zero.

After extraction of IMFs from a time series signal the residue tends to become
a monotonic function, such that no more IMFs can be extracted. Finally, after
the iterative process, the input signal is decomposed into a sum of IMF
functions and a residue.

In [58], an algorithm, modified from EMD, called Empirical iterative algorithm
is proposed for denoising the GSR from motion artefacts and quantization
noise. The algorithm does not rely on the shifting process of EMD, and
provides the filtered signal directly as an output of each iteration.

####  V-C2 Kalman Filtering

Low-pass and moving average filters have been used for GSR preprocessing [26],
however, the Kalman filter is the better choice. The Kalman filter is a model-
based filter and a state estimator. It employs a mathematical model of the
process producing the signal to be filtered [75]. In [76], an extended Kalman
filter is used to remove noise and artifact from GSR signal in real time. The
Kalman filter used in this article is comparable to a 3rd order Butterworth
low pass filter with a similar frequency response. However, the kalman filter
is more robust than its counter-part butterworth low pass filter while
suppressing the noise and artifacts.

###  V-D 1D to 2D Conversion

Physiological signals such as EEG, ECG and GSR are 1D signal in raw form. In
[77] and [78], raw ECG signal was converted into 2D form i.e into three
statistical images namely Gramian Angular Field (GAF) images, Recurrence Plot
(RP) images and Markov Transition Field (MTF) images as shown in Fig. 7.
Experimental results show the superiority of 1D to 2D preprocessing. In [79],
ECG and GSR are converted in 2D scalogram for better emotion recognition.
Furthermore, in [80], ECG and GSR signals are converted in 2D RP images for
improved emotion recognition as compared to 1D form signals.

Thus, the transformation from 1D to 2D has been shown to be an important
preprocessing step for physiological signals.

##  VI Inter-Subject Data Variance

In this section, we will review the effect of inter-subject data variance on
emotion recognition. We will also discuuss the reasons of inter-subject data
variance and recommend solutions for the problem.

There exists few papers [43], [32], [81], [82], [83] and [84], which only
reported the problem of inter-subject data variance and its effect on
degradation of emotion recognition accuracy, but they did not shed light on
the possible reasons behind inter-subject data variability.

To derive our point home, we performed experiments on ECG data of two
datasets, WESAD dataset [85] and our own Ryerson Multimedia Research
Laboratory (RML) dataset [86]. The issue of inter-subject data variability was
particularly apparent in the RML dataset, since we controlled the data
collection procss.

To observe the high inter-subject variablity, we utilize leave-one-subject-out
(LOSO) cross-validation (CV) for all our experiments, where the models are
trained with data from all-but-one subjects, and tested on the held out
subject. The average results across all subjects are presented in Tables II,
III, IV and V. In Tables II and IV show the results of experiments conducted
with 1D ECG for both datasets. To see the affect of data preprocessing on
inter-subject data variability, we transform the raw ECG data into
spectrograms. ResNet-18 [87] was trained on these spectrgrams and results are
shown in Tables III and V.

Testing Sub Accuracy Precision Recall F1 Score 2 44.2 47.3 44.2 40.8 3 47.5
47.9 47.5 45.6 4 46.8 55.4 46.8 43.8 5 51.5 51.3 51.5 49 6 46 45.7 46 43.9 7
51.3 47 51.3 46 8 51 51.4 51 50.6 9 42.8 41.5 42.8 41 10 57.6 56.1 57.6 53.4
11 56.1 57.5 56.1 57.4 13 52.1 51.6 52.1 50.6 14 61.6 60 61.6 55.4 15 52.9
52.1 52.9 50.6 16 56.4 52.5 56.4 52 17 53.1 50.7 53.1 50.9 Average 51.4 51.2
51.4 48.7

TABLE II: Accuracy, Precision, Recall and $F_{1}$ Score Using 1D raw WESAD
data with 1D CNN

Testing Sub Accuracy Precision Recall F1 Score 2 75.1 75.3 75.1 75.2 3 73.3
75.6 73.3 73.8 4 73.4 76.7 73.4 73.8 5 73.8 77.4 73.8 74.4 6 65.3 67.7 65.3
65.7 7 71.3 76 71.3 72 8 66.8 70.4 66.8 67.7 9 67.9 72.9 67.9 68.4 10 69.5
71.8 69.5 70 11 75.3 78.4 75.3 75.7 13 68.8 73 68.8 69.3 14 78.1 80.1 78.1
78.4 15 67.8 70.6 67.8 68.7 16 75.8 79 75.8 76.1 17 78.1 81.6 78.1 78.2
Average 72 75.1 72 72.5

TABLE III: Accuracy, Precision, Recall and $F_{1}$ Score Using 2D Spectrograms
of WESAD data with ResNet-18

Testing Sub Accuracy Precision Recall F1 Score 2 58.9 60.9 58.9 58.6 3 65.4
66.4 65.4 65.4 4 59.3 59.4 59.3 58.9 5 64.7 64.2 64.7 63.2 11 45.3 48.8 45.3
38 12 69.6 72.6 69.6 69.1 13 70.6 76.3 70.6 70.3 14 66.7 69 66.7 67.2 16 64.8
67.2 64.8 64.1 Average 62.8 62.5 62.8 61.6

TABLE IV: Accuracy, Precision, Recall and $F_{1}$ Score Using 1D raw RML data
with 1D CNN

Testing Sub Accuracy Precision Recall F1 Score 2 72.1 74.87 72.1 72.25 3 71.3
75.6 71.25 71.17 4 60.7 58.65 60.7 56.06 5 62.5 63.3 62.5 62.54 11 49.3 37.6
49.3 39.35 12 61.11 63.48 61.11 61.25 13 76.1 83.7 76.15 76.22 14 64 66.13 64
63.59 16 70.3 73.28 70.3 70.8 Average 65.34 66.2 65.34 63.7

TABLE V: Accuracy, Precision, Recall and $F_{1}$ Score Using 2D Spectrograms
of RML data with ResNet-18

From Tables II, III, IV and V, we observe the inter-subject data variance
across all the metrics i.e accuracy, precision, recall and $F_{1}$ score. We
also observe that by transforming 1D data to 2D using spectrograms have
increased the accuracy of the affective state but did not reduce the data
variance significantly.

Looking at the WESAD results, we can see that the average accuracy is only
72%. This is significacntly lower than the average accuracy achieved from ECG
in x,y,z. However, x,y,z used 5 or 10-fold cross validation, which means that
a segment of data for each subject was always present in the training set.
This is an unrealistic scenario for practical applications. Therefore, dealing
with inter-subject variability and conducting experiments in a LOSO setting is
very important. We elaborate on this point further in Section VII.

###  VI-A Statistical Analysis

We perform statistical analysis by plotting errorbars on 1D and 2D WESAD data
to notice the variability in accuracy of each individual subject across the
overall mean value of accuracy as shown in Figures 8 and 9. The errorbars
showing the spread or variability in the data. Figures 8 and 9 can also be
used to detect the outlier subjects from the dataset that could be removed
later on for reducing the variability amongst the remaining subjects.

Figure 8: Error-bars showing Inter-subject variability in terms of Accuracy
across the mean value for 1D WESAD data Figure 9: Error-bars showing Inter-
subject variability in terms of Accuracy across the mean value for 2D WESAD
data

###  VI-B Reasons of Inter-Subject Data Variance

Following could be the reason for inter-subject variability.

  * •

Some users feel uncomfortable when wearing a sensor on their bodies and behave
unnaturally and unexpectedly during data collection.

  * •

Body movements during data collection adversely effect the quality of recorded
data. For instance, head and eyeball movement during EEG data collection and
hand movement during EMG data collection are the sources of degradation of
these data.

  * •

An intra-variance subject is also caused due to the placement of electrodes on
different areas of the body. For example, during chest-based ECG collection,
electrodes are placed on different areas of the chest and during collection of
EEG data, the electrodes are placed on all over the scalp i.e from left side
to right side through the center of the head.

  * •

Different users respond differently to the same stimuli. This difference in
their response depends upon their mental strength and physical stability. For
instance, a movie clip can instill fear in some users while other users may
feel excitement while watching the it.

  * •

The length of the collected data is also amongst the major reasons of creating
subject diversity. A long video is more likely to elicit diverse emotional
states [88], while the data with limited samples will create problem during
the training of the model and classification of the affective states.

  * •

Experimental environment and quality of sensors/ equipment are also the
reasons of variance in data. For instance, uncalibrated sensors may create
errors at different stages of data collection.

###  VI-C Reducing Inter-Subject Data Variance

Following actions could be taken to remove the inter-subject variability.

  * •

The collected sample size for each subject should be same. The length of the
data should be set carefully to avoid variance in the data.

  * •

The users or subjects should be trained to remain calm and behave normally
during data collection or wearing a sensor.

  * •

Different channels of the data collecting equipment should be calibrated
against the same standards so that the intra-variance in the data for the
subject can be reduced.

  * •

In [89], statistics-based unsupervised domain adaptation algorithm was
proposed for reducing the inter-subject variance. A new baseline model was
introduced to tackle the classification task, and then two novel domain
adaptation objective functions, Cluster-aligning loss and Cluster-separating
loss were presented to mitigate the negative impacts caused by domain shifts.

We believe more research in the area of domain adaptation could help tackle
inter-subject data variability.

##  VII Data Splitting

There are two major types of data splitting techniques that are being used for
emotion recognition using physiological signals. These two methods of
splitting are called subject independent and subject dependent.

In subject independent case, the model is trained from data of various
subjects and then tested on data of new subjects (that are not included in the
training part). In the subject dependent case, the training and testing part
is randomly distributed among all the subjects.

###  VII-A Subject Independent Splitting

The most commonly used subject independent data splitting method is leave-one-
subject-out (LOSO) crossvalidation. In LOSO, tha model is trained on (k-1)
subjects and leave one subject for testing and this process is repeated for
every subject of the dataset.

In [90] and [91], LOSO cross validation technique is used for DEAP dataset
containing 32 subjects. Thus, the experiment is repeated for every subject and
the final performance is considered as the average of all experiments. In
[92], Leave-one-subject out method is conducted to evaluate the performance,
and the final performance is calculated by the averaging method. In [93], EEG-
based emotion classification was conducted by a leave-one-subject out
validation (LOSOV). One subject among the 20 subjects is used for testing
while the others were used for the training set. The experiments were repeated
for all subjects and the mean and standard deviation of the accuracies were
used to evaluate the model performance.

Figure 10: Conceptual Visualization of 5-fold cross validation.

###  VII-B Subject Dependent Splitting

In subject dependent splitting, various strategies are used. In one of the
strategies, training and testing parts are taken from a single subject with a
particular splitting percentage. In another setting, few trials of same
subject are taken as a training part and rest of them are used as testing.
Furthermore, in another method of subject dependent splitting, testing and
training parts are randomly selected across all the subjects either by
particular percenatge or by k-fold cross validation. For k = 5, the concept of
cross validation is shown in Fig. 10. The final performance is the average of
all the five evaluations.

In [37], physiological signals of 7 trials out of 28 were selected as testing
for each subject and rest of the trials are used for training. This was
repeated for every subject. In [94], models are evaluated using 10-fold cross
validation performed on each subject. This was repeated on all 32 subjects of
DEAP datasets and the final evaluation about accuracies and F1 scores are
based on averaging method. In [95], physiological samples of all subjects of
DEAP dataset are used to evaluate the model, The 10-fold cross validation
technique is used to analyze the emotion recognition performance. All samples
are randomly divided into ten subsets. One subset is regarded as a test set,
and another subset as a validation set. The remaining subsets are regarded as
a training set. The above process is repeated ten times until all subsets are
tested.

###  VII-C Summary of the section

Subject-dependent classification is usually performed to investigate the
individual variability between subjects on emotion recognition from their
physiological signals.

In real application scenarios, Emotional recognition system with subject
independent classification is considered more pratical as the system has to
predict emotions of each subject or patient separately when implemented for
emotion recognition task.

In [84], comparison between subject dependent and subject Independent setting
is provided. it is observed that subject independent splitting show 3% lower
accuracy because of high inter-subject variability. However, the generalized
capability of subject independent model on unseen data is more as these model
learn inter-subject variability.

Thus it is more valuable to develop a good subject-independent emotional
recognition model so that they can generalize well on unseen data or new
patients.

##  VIII Multimodal Fusion

The purpose of multimodal fusion is to obtain complementary information from
different physiological modalities to improve the emotion recognition
performance. Fusing different modalities alleviates the weaknesses of
individual modalities by integrating complementary information from the
modalities.

In [96], new emotional discriminative space is constructed utilizing
Discriminative canonical correlation analysis (DCCA) is used from
physiological signals with the assistance of EEG signals. Finally, machine
learning techniques are utilized to build emotion recognizer. Authors in [97]
trained feedforward neural networks using both the fused and non-fused
signals. Experiments show that the fused method outperforms each individual
signal across all emotions tested. In [98], feature level fusion is performed
between the features from two modalities i.e facial expression images and EEG
signals. Feature map of size $128\times 128$ is constructed by concatenating
the facial images and their corresponding EEG feature maps. Emotion
recognition model is finally trained on these feature maps. Experimental
results show the improved performance of multimodal features over single
modality features. In [99], hybrid fusion of face data, EEG and GSR is
performed. First, features from EEG and GSR modalities are integrated for
estimating the level of arousal. Final fusion was the late fusion of EEG, GSR
and face modalities.

Authors in [100] presented a new emotion recognition framework based on
Decision fusion, where three separate classifiers were trained on ECG, EMG and
skin conductance level(SCL). The majority voting principle was used to
determine a final classification result on the three outputs of the separate
classifiers.

In [101], multidomain features such as features from time domain, frequency
domain and wavelet domain are fused with different combinations to identify
stable features which would best represent the underlying EEG signal
characteristics and performed better classification. Multidomain feature
fusion is performed using concatenation to obtained final feature vector. In
[102], weighted average fusion of physiological signal was done to classify
valence and arousal. Overall performance was quantified as the percentage of
correctly classified emotional states per video. It is also observed that the
classification performance obtained was slightly better for arousal than
valence. Authors in [103] provide the study about combining the EEG data and
audio visual features of the video. Then, PCA algorithm was applied to reduce
the feature dimensions. In [104], feature level fusion of sensor data from EEG
and EDA sensors was performed for human emotion detection. Nine features
(delta, theta, low alpha, high alpha, low beta, high beta, low gamma, high
gamma and galvanic skin response) are selected for training a neural network
model. The classification performance of each modality was also evaluated
separately. It is observed that multimodal emotion recognition is better than
using a single modality. In [105], authors developed two late multi-modal data
fusion methods with deep CNN models to estimate positive and negative affects
(PA and NA) scores and explored the effect of the two late fusion methods and
different combination of the physiological signals on the performance. In the
first late fusion method, a separate CNN is trained for each modality and in
the second late fusion method, the average of the three classes’ (baseline,
stress, and amusement) probabilities across the pretrained CNNs is calculated,
and the emotion class with the highest average probability is selected. In the
case of the PA or NA estimation, the average of the estimated scores across
the pretrained CNNs is provided as the estimated affect score. In [86],
multidomain fusion of ECG signal is performed for multilevel stress
assessment. First, the ECG signal was converted into image and then images are
made multimodal and multidomain by transforming them into frequency and time-
spectrum domain using Discrete Fourier Transform (DFT) and Gabor wavelet
transform (GWT) respectively. Features in different domains are extracted by
Convolutional Neural networks (CNNs) and then decision level fusion is
performed for improving the classification accuracy as shown in Fig. 11.

Figure 11: Multimodal Multidomain Fusion of ECG Signal.

###  VIII-A Summary of the section

Above review shows that the fusion of the physiological modalities exhibits
better performance than the single modality in terms of classification
accuracy for both arousal and valence dimensions. Furthermore, it is observed
from the review that the two major fusion methods practiced for physiological
data features are feature fusion and decision fusion, however, feature fusion
is adopted by the reseracher more than decision level. The greatest advantage
of feature level fusion is that it utilizes the correlation among the
modalities at an early stage. Furthermore, only one classifier is required to
perform a task, making the training process less tedious. The disadvantage
with decision level fusion is the use of more than one classifier. This makes
the task time consuming.

##  IX Future challenges

The work flow for emotion recognition consists of many steps such as data
acquisition, data annotation, data preprocessing, feature extraction and
selection, and recognition. The problems and challenges are in each step of
emotion recognition and are expalined in this section.

###  IX-A Data Acquisition

One of the significant challenge in acquiring physiological signals is to deal
with noise, baseline drifts, different artifacts due to body movements,
different responses of participants to different stimuli and low graded
signal. The data acquiring devices carry noise which corrupts the signal and
induced artifacts that superimposed on the signal. The second challenge is the
setting of stable and noiseless lab and selection of stumuli so that genuine
emotions are induced which are closed to real world feelings. Another
challenge in acquiring a high quality data is the various responses of
participants or subjects to the same stimulai. This causes large inter-subject
variability in the data and is also the reason that most of the avaiable
emotion recognition datasets are of short duration.

To overcome the above challeges, well-designed labs and proper selection of
stimuli are necessary. Furthermore, subjects must be properly trained to avoid
large variance in the data and possibility of gather datasets with long
duration and less inter-subject variability.

###  IX-B Data Annotation

High volume research has been conducted on emotion recognition, even then
there is no uniform standards to annotate data. Discrete and continuous data
annotation techniques are commonly used, however, due to unavailability of
uniform annotation standard, there is no compatibility between the datasets.
Furthermore, all emotions can neither be listed as discrete nor continuous as
different emotions need different range of intensities for description. The
problem with existing datasets is that some of them are annotated with four
emotions, some are with basic six emotions or with eight emotions. Due to this
different emotion labels, datasets are not compatible and emotion recognition
algorithms donot work equally well on these datasets.

To face the above mentioned challenges of data annotation, hybrid data
annotation technique should be introduced where each emotion is labelled
according to its range of energy. However, care must be taken while selecting
new standards of data annotation because hybrid annotation technique may lead
to imbalance data.

###  IX-C Feature Extraction and Fusion

Features for emotion recognition are categorized into time domain features and
transform domain features. Time domain features mostly include statistical
features of first and higher orders. Transform domain features are further
divided into frequency domain and time-frequency domain such as feature
extracted from wavelet transform. Still, there is no set of features that
gurantees to work for all models and situations.

One of the solution of the above problem is to design adaptive intelligent
systems that can automatically select the best features for the classifier
model. Furthermore, another solution is to perform multimodel fusion. In
section VIII, it is explained that commonly practiced fusion techniques for
emotion recognition are feature level and decision level fusion. However, for
feature level fusion, concatenation is mostly used and for decision level
fusion, majority voting is mostly practiced. Thus, a need of more improved and
intelligent based fusion methods is arising because the simple feature level
and decision level fusion cannot counter the non-stationary and nonlinear
nature of features.

###  IX-D Generalization of Models

One of the biggest challenges in emotion recognition is to design the models
that can generalize well on unseen data or new datasets. The main obstacles
are the limited samples in the datasets, non-standard data splitting
techniques and large inter-subject data variability. The commonly used data
splitting techniques are subject dependent and subject independent techniques.
It is explained in section VII, that the models trained on subject independent
settings are more likely to generalize well. However, exisitng subject-
independent recognition models are not intelligent enough to perform well in
realistic and real-time applications.

Since while testing emotion recognition model has to face an unseen subjects,
therefore, one of the solution is to train and validate the model on large
number of subjects so that it can generalize well on testing. The problem of
limited dataset can be solved by carefully applying data augmentation
techniques because that techniques could lead to data imbalance and
overfitting of the model.

###  IX-E Modern Machine Learning Methods

Advanced machine learning tools need to developed for mitigating the
challenges of emotion recognition using physiological signals. For instance,
one of the modern technique which is being used is transfer learning. In
transfer learning, the knowledge learning while solving one task can be
applied for different but related work. For example, the rich features learned
by a machine learning model while doing emotion recognition based on EEG data
from a large dataset could also be applicable for EEG data of other dataset.
This can easily solve the problem of small dataset where less data samples are
available. Transfer learning methods are getting success but still lot of
research work is required to be done for physiological signal based transfer
learning methods for emotion recognition.

##  X Conclusion

In this paper, we provide a review on the physiological signal based emotion
recognition and bridge the gap in the existing literature by mainly focussing
on crucial elements lacking in the existing literature. These crucial elements
include the effect of inter-subject data variance on emotion recognition,
important data annotation techniques for emotion recognition and their
comparison, data preprocessing techniques, data splitting techniques for
improving the generalization of emotion recognition model and different
multimodal fusion techniques and their comparison. We also discuss the future
challenges of this existing research area.

## References

  * [1] S. De Nadai, M. D’Incà, F. Parodi, M. Benza, A. Trotta, E. Zero, L. Zero, and R. Sacile, “Enhancing safety of transport by road by on-line monitoring of driver emotions,” in _2016 11th System of Systems Engineering Conference (SoSE)_. Ieee, 2016, pp. 1–4.
  * [2] E. Ertin, N. Stohs, S. Kumar, A. Raij, M. Al’Absi, and S. Shah, “Autosense: unobtrusively wearable sensor suite for inferring the onset, causality, and consequences of stress in the field,” in _Proceedings of the 9th ACM conference on embedded networked sensor systems_ , 2011, pp. 274–287.
  * [3] A. Kołakowska, “Towards detecting programmers’ stress on the basis of keystroke dynamics,” in _2016 Federated Conference on Computer Science and Information Systems (FedCSIS)_. IEEE, 2016, pp. 1621–1626.
  * [4] A. Kołakowska, A. Landowska, M. Szwoch, W. Szwoch, and M. R. Wróbel, “Emotion recognition and its application in software engineering,” in _2013 6th International Conference on Human System Interactions (HSI)_. IEEE, 2013, pp. 532–539.
  * [5] M. Szwoch and W. Szwoch, “Using different information channels for affect-aware video games-a case study,” in _International Conference on Image Processing and Communications_. Springer, 2018, pp. 104–113.
  * [6] G. Muhammad and M. S. Hossain, “Emotion recognition for cognitive edge computing using deep learning,” _IEEE Internet of Things Journal_ , vol. 8, no. 23, pp. 16 894–16 901, 2021.
  * [7] S. Siddharth, T.-P. Jung, and T. J. Sejnowski, “Impact of affective multimedia content on the electroencephalogram and facial expressions,” _Scientific Reports_ , vol. 9, no. 1, pp. 1–10, 2019.
  * [8] S.-H. Wang, P. Phillips, Z.-C. Dong, and Y.-D. Zhang, “Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm,” _Neurocomputing_ , vol. 272, pp. 668–676, 2018.
  * [9] D. Yang, A. Alsadoon, P. C. Prasad, A. K. Singh, and A. Elchouemi, “An emotion recognition model based on facial recognition in virtual learning environment,” _Procedia Computer Science_ , vol. 125, pp. 2–10, 2018.
  * [10] Y.-D. Zhang, Z.-J. Yang, H.-M. Lu, X.-X. Zhou, P. Phillips, Q.-M. Liu, and S.-H. Wang, “Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,” _IEEE Access_ , vol. 4, pp. 8375–8385, 2016.
  * [11] H. Gunes, C. Shan, S. Chen, and Y. Tian, “Bodily expression for automatic affect recognition,” _Emotion recognition: A pattern analysis approach_ , pp. 343–377, 2015.
  * [12] S. Piana, A. Staglianò, F. Odone, and A. Camurri, “Adaptive body gesture representation for automatic emotion recognition,” _ACM Transactions on Interactive Intelligent Systems (TiiS)_ , vol. 6, no. 1, pp. 1–31, 2016.
  * [13] F. Noroozi, C. A. Corneanu, D. Kamińska, T. Sapiński, S. Escalera, and G. Anbarjafari, “Survey on emotional body gesture recognition,” _IEEE transactions on affective computing_ , vol. 12, no. 2, pp. 505–523, 2018.
  * [14] H. Zheng and Y. Yang, “An improved speech emotion recognition algorithm based on deep belief network,” in _2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS)_. IEEE, 2019, pp. 493–497.
  * [15] S. Latif, R. Rana, S. Khalifa, R. Jurdak, J. Qadir, and B. W. Schuller, “Survey of deep representation learning for speech emotion recognition,” _IEEE Transactions on Affective Computing_ , 2021.
  * [16] M. B. Akçay and K. Oğuz, “Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers,” _Speech Communication_ , vol. 116, pp. 56–76, 2020.
  * [17] W. Li, Z. Zhang, and A. Song, “Physiological-signal-based emotion recognition: An odyssey from methodology to philosophy,” _Measurement_ , vol. 172, p. 108747, 2021.
  * [18] N. S. Suhaimi, J. Mountstephens, and J. Teo, “Eeg-based emotion recognition: a state-of-the-art review of current trends and opportunities,” _Computational intelligence and neuroscience_ , vol. 2020, 2020.
  * [19] Y.-L. Hsu, J.-S. Wang, W.-C. Chiang, and C.-H. Hung, “Automatic ecg-based emotion recognition in music listening,” _IEEE Transactions on Affective Computing_ , vol. 11, no. 1, pp. 85–99, 2017.
  * [20] P. Sarkar and A. Etemad, “Self-supervised learning for ecg-based emotion recognition,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 3217–3221.
  * [21] Q. Zhang, X. Lai, and G. Liu, “Emotion recognition of gsr based on an improved quantum neural network,” in _2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC)_ , vol. 1. IEEE, 2016, pp. 488–492.
  * [22] A. Pentland and T. Heibeck, “Honest signals,” _MIT press_ , 2008.
  * [23] L. Shu, J. Xie, M. Yang, Z. Li, Z. Li, D. Liao, X. Xu, and X. Yang, “A review of emotion recognition using physiological signals,” _Sensors_ , vol. 18, no. 7, p. 2074, 2018.
  * [24] H. Z. Wijasena, R. Ferdiana, and S. Wibirama, “A survey of emotion recognition using physiological signal in wearable devices,” in _2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS)_. IEEE, 2021, pp. 1–6.
  * [25] E. Joy, R. B. Joseph, M. Lakshmi, W. Joseph, and M. Rajeswari, “Recent survey on emotion recognition using physiological signals,” in _2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS)_ , vol. 1. IEEE, 2021, pp. 1858–1863.
  * [26] X. Fan, Y. Yan, X. Wang, H. Yan, Y. Li, L. Xie, and E. Yin, “Emotion recognition measurement based on physiological signals,” in _2020 13th International Symposium on Computational Intelligence and Design (ISCID)_. IEEE, 2020, pp. 81–86.
  * [27] P. J. Bota, C. Wang, A. L. Fred, and H. P. Da Silva, “A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals,” _IEEE Access_ , vol. 7, pp. 140 990–141 020, 2019.
  * [28] R. Plutchik, “The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice,” _American scientist_ , vol. 89, no. 4, pp. 344–350, 2001\.
  * [29] P. Ekman, “An argument for basic emotions,” _Cognition & emotion_, vol. 6, no. 3-4, pp. 169–200, 1992.
  * [30] J. A. Russell, “A circumplex model of affect.” _Journal of personality and social psychology_ , vol. 39, no. 6, p. 1161, 1980.
  * [31] J. A. Miranda-Correa, M. K. Abadi, N. Sebe, and I. Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,” _IEEE Transactions on Affective Computing_ , vol. 12, no. 2, pp. 479–493, 2018.
  * [32] R. Subramanian, J. Wache, M. K. Abadi, R. L. Vieriu, S. Winkler, and N. Sebe, “Ascertain: Emotion and personality recognition using commercial sensors,” _IEEE Transactions on Affective Computing_ , vol. 9, no. 2, pp. 147–160, 2016\.
  * [33] L. Zhang, S. Walter, X. Ma, P. Werner, A. Al-Hamadi, H. C. Traue, and S. Gruss, ““biovid emo db”: A multimodal database for emotion analyses validated by subjective ratings,” in _2016 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2016, pp. 1–6.
  * [34] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals,” _IEEE transactions on affective computing_ , vol. 3, no. 1, pp. 18–31, 2011.
  * [35] S. Katsigiannis and N. Ramzan, “Dreamer: A database for emotion recognition through eeg and ecg signals from wireless low-cost off-the-shelf devices,” _IEEE journal of biomedical and health informatics_ , vol. 22, no. 1, pp. 98–107, 2017.
  * [36] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” _IEEE transactions on affective computing_ , vol. 3, no. 1, pp. 42–55, 2011.
  * [37] T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, “Mped: A multi-modal physiological emotion database for discrete emotion recognition,” _IEEE Access_ , vol. 7, pp. 12 177–12 191, 2019.
  * [38] W.-L. Zheng and B.-L. Lu, “Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks,” _IEEE Transactions on Autonomous Mental Development_ , vol. 7, no. 3, pp. 162–175, 2015\.
  * [39] T. Xue, A. El Ali, T. Zhang, G. Ding, and P. Cesar, “Ceap-360vr: A continuous physiological and behavioral emotion annotation dataset for 360 vr videos,” _IEEE Transactions on Multimedia_ , 2021.
  * [40] D. Watson, L. A. Clark, and A. Tellegen, “Development and validation of brief measures of positive and negative affect: the panas scales.” _Journal of personality and social psychology_ , vol. 54, no. 6, p. 1063, 1988.
  * [41] M. M. Bradley and P. J. Lang, “Measuring emotion: the self-assessment manikin and the semantic differential,” _Journal of behavior therapy and experimental psychiatry_ , vol. 25, no. 1, pp. 49–59, 1994.
  * [42] J. J. Gross and R. W. Levenson, “Emotion elicitation using films,” _Cognition & emotion_, vol. 9, no. 1, pp. 87–108, 1995.
  * [43] K. Yang, C. Wang, Y. Gu, Z. Sarsenbayeva, B. Tag, T. Dingler, G. Wadley, and J. Goncalves, “Behavioral and physiological signals-based deep multimodal approach for mobile emotion recognition,” _IEEE Transactions on Affective Computing_ , 2021.
  * [44] T. Althobaiti, S. Katsigiannis, D. West, and N. Ramzan, “Examining human-horse interaction by means of affect recognition via physiological signals,” _Ieee access_ , vol. 7, pp. 77 857–77 867, 2019.
  * [45] J. A. Russell and A. Mehrabian, “Evidence for a three-factor theory of emotions,” _Journal of research in Personality_ , vol. 11, no. 3, pp. 273–294, 1977.
  * [46] L. B. Hinkle, K. K. Roudposhti, and V. Metsis, “Physiological measurement for emotion recognition in virtual reality,” in _2019 2nd International Conference on Data Intelligence and Security (ICDIS)_. IEEE, 2019, pp. 136–143.
  * [47] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” _Development and psychopathology_ , vol. 17, no. 3, pp. 715–734, 2005.
  * [48] L. Romeo, A. Cavallo, L. Pepa, N. Berthouze, and M. Pontil, “Multiple instance learning for emotion recognition using physiological signals,” _IEEE Transactions on Affective Computing_ , 2019.
  * [49] B. L. Fredrickson and D. Kahneman, “Duration neglect in retrospective evaluations of affective episodes.” _Journal of personality and social psychology_ , vol. 65, no. 1, p. 45, 1993.
  * [50] F. Nagel, R. Kopiez, O. Grewe, and E. Altenmüller, “Emujoy: Software for continuous measurement of perceived emotions in music,” _Behavior Research Methods_ , vol. 39, no. 2, pp. 283–290, 2007.
  * [51] E. Schubert, “Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space,” _Australian Journal of Psychology_ , vol. 51, no. 3, pp. 154–165, 1999.
  * [52] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of eeg signals and facial expressions for continuous emotion detection,” _IEEE Transactions on Affective Computing_ , vol. 7, no. 1, pp. 17–28, 2015.
  * [53] S. Mariooryad and C. Busso, “Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations,” in _2013 Humaine Association Conference on Affective Computing and Intelligent Interaction_. IEEE, 2013, pp. 85–90.
  * [54] G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, “The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent,” _IEEE transactions on affective computing_ , vol. 3, no. 1, pp. 5–17, 2011.
  * [55] R. Cowie, E. Douglas-Cowie, S. Savvidou*, E. McMahon, M. Sawey, and M. Schröder, “’feeltrace’: An instrument for recording perceived emotion in real time,” in _ISCA tutorial and research workshop (ITRW) on speech and emotion_ , 2000.
  * [56] J. M. Girard and A. G. C Wright, “Darma: Software for dual axis rating and media annotation,” _Behavior research methods_ , vol. 50, no. 3, pp. 902–909, 2018.
  * [57] W. Xu, Y. Chen, H. Sundaram, and T. Rikakis, “Multimodal archiving, real-time annotation and information visualization in a biofeedback system for stroke patient rehabilitation,” in _Proceedings of the 3rd ACM workshop on Continuous archival and retrival of personal experences_ , 2006, pp. 3–12.
  * [58] A. Gautam, N. Simões-Capela, G. Schiavone, A. Acharyya, W. De Raedt, and C. Van Hoof, “A data driven empirical iterative algorithm for gsr signal pre-processing,” in _2018 26th European Signal Processing Conference (EUSIPCO)_. IEEE, 2018, pp. 1162–1166.
  * [59] S. Muthukumaraswamy, “High-frequency brain activity and muscle artifacts in meg/eeg: a review and recommendations,” _Frontiers in human neuroscience_ , vol. 7, p. 138, 2013.
  * [60] M. Fatourechi, A. Bashashati, R. K. Ward, and G. E. Birch, “Emg and eog artifacts in brain computer interface systems: A survey,” _Clinical neurophysiology_ , vol. 118, no. 3, pp. 480–494, 2007.
  * [61] G. Gratton, “Dealing with artifacts: The eog contamination of the event-related brain potential,” _Behavior Research Methods, Instruments, & Computers_, vol. 30, no. 1, pp. 44–53, 1998.
  * [62] H. Ramoser, J. Muller-Gerking, and G. Pfurtscheller, “Optimal spatial filtering of single trial eeg during imagined hand movement,” _IEEE transactions on rehabilitation engineering_ , vol. 8, no. 4, pp. 441–446, 2000\.
  * [63] R. J. Croft, J. S. Chandler, R. J. Barry, N. R. Cooper, and A. R. Clarke, “Eog correction: a comparison of four methods,” _Psychophysiology_ , vol. 42, no. 1, pp. 16–24, 2005.
  * [64] A. D. Bigirimana, N. Siddique, and D. Coyle, “A hybrid ica-wavelet transform for automated artefact removal in eeg-based emotion recognition,” in _2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2016, pp. 004 429–004 434.
  * [65] C. Venkatesan, P. Karthigaikumar, A. Paul, S. Satheeskumaran, and R. Kumar, “Ecg signal preprocessing and svm classifier-based abnormality detection in remote healthcare applications,” _IEEE Access_ , vol. 6, pp. 9767–9773, 2018\.
  * [66] R. Alcaraz, F. Sandberg, L. Sörnmo, and J. J. Rieta, “Classification of paroxysmal and persistent atrial fibrillation in ambulatory ecg recordings,” _IEEE Transactions on Biomedical Engineering_ , vol. 58, no. 5, pp. 1441–1449, 2011.
  * [67] K. K. Patro, A. Jaya Prakash, M. Jayamanmadha Rao, and P. Rajesh Kumar, “An efficient optimized feature selection with machine learning approach for ecg biometric recognition,” _IETE Journal of Research_ , pp. 1–12, 2020.
  * [68] R. Cordeiro, D. Gajaria, A. Limaye, T. Adegbija, N. Karimian, and F. Tehranipoor, “Ecg-based authentication using timing-aware domain-specific architecture,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 39, no. 11, pp. 3373–3384, 2020.
  * [69] B. Surawicz and T. Knilans, _Chou’s electrocardiography in clinical practice: adult and pediatric_. Elsevier Health Sciences, 2008.
  * [70] S. Saechia, J. Koseeyaporn, and P. Wardkein, “Human identification system based ecg signal,” in _TENCON 2005-2005 IEEE Region 10 Conference_. IEEE, 2005, pp. 1–4.
  * [71] J.-J. Wei, C.-J. Chang, N.-K. Chou, and G.-J. Jan, “Ecg data compression using truncated singular value decomposition,” _IEEE Transactions on Information Technology in Biomedicine_ , vol. 5, no. 4, pp. 290–299, 2001.
  * [72] M. M. Tawfik, H. Selim, and T. Kamal, “Human identification using time normalized qt signal and the qrs complex of the ecg,” in _2010 7th International Symposium on Communication Systems, Networks & Digital Signal Processing (CSNDSP 2010)_. IEEE, 2010, pp. 755–759.
  * [73] I. Odinaka, P.-H. Lai, A. D. Kaplan, J. A. O’Sullivan, E. J. Sirevaag, S. D. Kristjansson, A. K. Sheffield, and J. W. Rohrbaugh, “Ecg biometrics: A robust short-time frequency analysis,” in _2010 IEEE International Workshop on Information Forensics and Security_. IEEE, 2010, pp. 1–6.
  * [74] N. E. Huang, Z. Shen, S. R. Long, M. C. Wu, H. H. Shih, Q. Zheng, N.-C. Yen, C. C. Tung, and H. H. Liu, “The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis,” _Proceedings of the Royal Society of London. Series A: mathematical, physical and engineering sciences_ , vol. 454, no. 1971, pp. 903–995, 1998.
  * [75] A. J. Haug, _Bayesian estimation and tracking: a practical guide_. John Wiley & Sons, 2012.
  * [76] C. Tronstad, O. M. Staal, S. Sælid, and Ø. G. Martinsen, “Model-based filtering for artifact and noise suppression with state estimation for electrodermal activity measurements in real time,” in _2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2015, pp. 2750–2753.
  * [77] Z. Ahmad, A. Tabassum, L. Guan, and N. Khan, “Ecg heart-beat classification using multimodal image fusion,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 1330–1334.
  * [78] Z. Ahmad, A. Tabassum, L. Guan, and N. M. Khan, “Ecg heartbeat classification using multimodal fusion,” _IEEE Access_ , vol. 9, pp. 100 615–100 626, 2021.
  * [79] A. Rahim, A. Sagheer, K. Nadeem, M. N. Dar, A. Rahim, and U. Akram, “Emotion charting using real-time monitoring of physiological signals,” in _2019 International Conference on Robotics and Automation in Industry (ICRAI)_. IEEE, 2019, pp. 1–5.
  * [80] R. Elalamy, M. Fanourakis, and G. Chanel, “Multi-modal emotion recognition using recurrence plots and transfer learning on physiological signals,” in _2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2021, pp. 1–7.
  * [81] V. Gupta, M. D. Chopda, and R. B. Pachori, “Cross-subject emotion recognition using flexible analytic wavelet transform from eeg signals,” _IEEE Sensors Journal_ , vol. 19, no. 6, pp. 2266–2274, 2018.
  * [82] H. Yao, H. He, S. Wang, and Z. Xie, “Eeg-based emotion recognition using multi-scale window deep forest,” in _2019 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2019, pp. 381–386.
  * [83] D. S. Wickramasuriya, M. K. Tessmer, and R. T. Faghih, “Facial expression-based emotion classification using electrocardiogram and respiration signals,” in _2019 IEEE Healthcare Innovations and Point of Care Technologies,(HI-POCT)_. IEEE, 2019, pp. 9–12.
  * [84] S. Kim, H.-J. Yang, N. A. T. Nguyen, S. K. Prabhakar, and S.-W. Lee, “Wedea: A new eeg-based framework for emotion recognition,” _IEEE Journal of Biomedical and Health Informatics_ , 2021.
  * [85] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven, “Introducing wesad, a multimodal dataset for wearable stress and affect detection,” in _Proceedings of the 20th ACM international conference on multimodal interaction_ , 2018, pp. 400–408.
  * [86] Z. Ahmad and N. M. Khan, “Multi-level stress assessment using multi-domain fusion of ecg signal,” in _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2020, pp. 4518–4521.
  * [87] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [88] L. Santamaria-Granados, M. Munoz-Organero, G. Ramirez-Gonzalez, E. Abdulhay, and N. Arunkumar, “Using deep convolutional neural network for emotion detection on a physiological signals dataset (amigos),” _IEEE Access_ , vol. 7, pp. 57–67, 2018.
  * [89] M. Chen, G. Wang, Z. Ding, J. Li, and H. Yang, “Unsupervised domain adaptation for ecg arrhythmia classification,” in _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2020, pp. 304–307.
  * [90] X. Zhang, J. Liu, J. Shen, S. Li, K. Hou, B. Hu, J. Gao, and T. Zhang, “Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine,” _IEEE transactions on cybernetics_ , vol. 51, no. 9, pp. 4386–4399, 2020.
  * [91] H. Ullah, M. Uzair, A. Mahmood, M. Ullah, S. D. Khan, and F. A. Cheikh, “Internal emotion classification using eeg signal with sparse discriminative ensemble,” _IEEE Access_ , vol. 7, pp. 40 144–40 153, 2019.
  * [92] E.-J. Chang, A. Rahimi, L. Benini, and A.-Y. A. Wu, “Hyperdimensional computing-based multimodality emotion recognition with physiological signals,” in _2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2019, pp. 137–141.
  * [93] C.-J. Yang, N. Fahier, C.-Y. He, W.-C. Li, and W.-C. Fang, “An ai-edge platform with multimodal wearable physiological signals monitoring sensors for affective computing applications,” in _2020 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2020, pp. 1–5.
  * [94] S. Vijayakumar, R. Flynn, and N. Murray, “A comparative study of machine learning techniques for emotion recognition from peripheral physiological signals,” in _2020 31st Irish Signals and Systems Conference (ISSC)_. IEEE, 2020, pp. 1–6.
  * [95] H. Chao and L. Dong, “Emotion recognition using three-dimensional feature and convolutional neural network from multichannel eeg signals,” _IEEE sensors journal_ , vol. 21, no. 2, pp. 2024–2034, 2020.
  * [96] W. Zhao, Z. Zhao, and C. Li, “Discriminative-cca promoted by eeg signals for physiological-based emotion recognition,” in _2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia)_. IEEE, 2018, pp. 1–6.
  * [97] D. Fabiano and S. Canavan, “Emotion recognition using fused physiological signals,” in _2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2019, pp. 42–48.
  * [98] X. Li, X. Zhang, H. Yang, W. Duan, W. Dai, and L. Yin, “An eeg-based multi-modal emotion database with both posed and authentic facial actions for emotion analysis,” in _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_. IEEE, 2020, pp. 336–343.
  * [99] Y. Cimtay, E. Ekmekcioglu, and S. Caglar-Ozhan, “Cross-subject multimodal emotion recognition based on hybrid fusion,” _IEEE Access_ , vol. 8, pp. 168 865–168 878, 2020.
  * [100] J. Xie, X. Xu, and L. Shu, “Wt feature based emotion recognition from multi-channel physiological signals with decision fusion,” in _2018 first asian conference on affective computing and intelligent interaction (ACII Asia)_. IEEE, 2018, pp. 1–6.
  * [101] M. Khateeb, S. M. Anwar, and M. Alnowami, “Multi-domain feature fusion for emotion classification using deap dataset,” _Ieee Access_ , vol. 9, pp. 12 134–12 142, 2021.
  * [102] J. Pinto, A. Fred, and H. P. da Silva, “Biosignal-based multimodal emotion recognition in a valence-arousal affective framework applied to immersive video visualization,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 3577–3583.
  * [103] B. Xing, H. Zhang, K. Zhang, L. Zhang, X. Wu, X. Shi, S. Yu, and S. Zhang, “Exploiting eeg signals and audiovisual feature fusion for video emotion recognition,” _IEEE Access_ , vol. 7, pp. 59 844–59 861, 2019.
  * [104] M. Yasemin, M. A. Sarıkaya, and G. Ince, “Emotional state estimation using sensor fusion of eeg and eda,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 5609–5612.
  * [105] M. D. Hssayeni and B. Ghoraani, “Multi-modal physiological data fusion for affect estimation using deep learning,” _IEEE Access_ , vol. 9, pp. 21 642–21 652, 2021.

|  Zeeshan Ahmad is with department of Electrical and Computer Engineering at
Ryerson University as a post doctarate fellow since 2021. His current research
focus on designing nultimodal fusion frameworks for stress analysis and
emotion recognition. His research interests include physiological signal
processing, deep learning for computer vision, multimodal fusion and medical
imaging.
---|---
|  Naimul Khan is an associate professor of Electrical and Computer
Engineering at Ryerson University, where he co-directs the Ryerson Multimedia
Research Laboratory (RML). His research focuses on creating user-centric
intelligent systems through the combination of novel machine learning and
human-computer interaction mechanisms. He is a recipient of the best paper
award at the IEEE International Symposium on Multimedia, the OCE TalentEdge
Postdoctoral Fellowship, and the Ontario Graduate Scholarship. He is a senior
member of IEEE and a member of ACM.
---|---