# A Comprehensive Survey on Affective Computing; Challenges, Trends,
Applications, and Future Directions

Sitara Afzal  Software Department, Sejong University, Seoul 05006, South Korea
Haseeb Ali Khan  Software Department, Sejong University, Seoul 05006, South
Korea  Imran Ullah Khan  Software Department, Sejong University, Seoul 05006,
South Korea  Md. Jalil Piran  Department of Computer Science and Engineering,
Sejong University, Seoul 05006, South Korea  Corresponding author:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Jong Weon Lee  Software Department,
Sejong University, Seoul 05006, South Korea  Corresponding author:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As the name suggests, affective computing aims to recognize human emotions,
sentiments, and feelings. There is a wide range of fields that study affective
computing, including languages, sociology, psychology, computer science, and
physiology. However, no research has ever been done to determine how machine
learning (ML) and mixed reality (XR) interact together. This paper discusses
the significance of affective computing, as well as its ideas, conceptions,
methods, and outcomes. By using approaches of ML and XR, we survey and discuss
recent methodologies in affective computing. We survey the state-of-the-art
approaches along with current affective data resources. Further, we discuss
various applications where affective computing has a significant impact, which
will aid future scholars in gaining a better understanding of its significance
and practical relevance.

##  I Introduction

Emotions, sentiments, and feelings, along with emotion recognition, constitute
what is referred to as ”affective computing” [1]. In 1997, Prof. Picard
introduced the idea of affective computing, which has helped computers
recognize and communicate their moods, as well as effectively respond to
humans’ moods [2] [3]. In many practical applications, it is desirable to
develop a cognitive, intelligent system that can detect and comprehend
people’s feelings while also providing sensitive and cordial responses [4].
Emotion is a cultural and psychobiological adaptation mechanism that enables
people to adapt dynamically to environmental changes. Emotions give meaning to
our lives, deepen our relationships with others, alert us to our wants and
sentiments, and help us make changes [5].

There are five components to a single sentiment: a cognitive evaluation, a
physical sensation, an intention, a subjective ”feeling,” a motor reaction,
and, in most situations, an interpersonal component. ”Emotion Regulation” (ER)
is a group of mental processes that affect our emotions, as well as the timing
of our emotions [6]. A dynamic process intended to downregulate or upregulate
positive or negative thoughts, it is essential to human mental function. In
order to identify expressed emotions (anger, disgust, fear, happiness, sorrow,
or surprise) and predict their sentiment (positive or negative), a framework
based on frame of reference intermodal attention [7] was developed.

It is called emotion dysregulation when a person is unable to effectively
control or process their emotions, resulting in unintended intensification or
deactivation [3]. Managing emotionally intense experiences is therefore a
component of self-regulation. A specific method of emotion management is not
necessarily beneficial or harmful [8]. In doing so, it avoids making
generalized judgments about which coping mechanisms are more or less adaptive
depending on the situation [9] [10]. In various social media platforms,
affective computing can greatly help in understanding the thoughts expressed
[7]. In this way, emotional computing is seen by many researchers as a way to
advance creation of human-centric AI and human intelligence [11]. There is a
difference between sentiment research and emotion recognition in affective
computing [12]. A diagram showing the different kinds of emotions can be found
in figure 1.

Figure 1: Types of Emotions

To understand and calculate human feelings, psychologists presented two common
theories: the discrete emotion model (or categorical emotion model) [13] and
the dimensional emotion model [14]. In order to identify people’s emotional
state, three main areas of focus for emotion recognition have been
investigated [15]: visual emotion recognition (VER), audio/speech emotion
recognition (AER/SER) and physiological emotion recognition (PER). It has been
shown that VR systems can elicit emotional responses that lead to
psychologically positive changes when experiencing increased presence in a
virtual environment [16]. These widely used datasets have inspired the
development of emotional computing, which is Machine Learning (ML). Study [17]
reports that in everyday human interaction, human emotions are primarily
expressed through facial expressions (55%) voice (38%) and words (7%) Textual,
auditory, and visual signals are generally referred to as physical data.

People’s propensity to openly express their ideas and thoughts on social media
platforms and websites makes it easy to gather a wide range of bodily affect
information. As a result of these data, many academics attempt to identify
minor emotions conveyed either explicitly or implicitly. As a result, [18] and
[19] present an innovative approach to multimodal music emotion analysis in
affective computing using LSTM networks. With the proposed model, a dual-
channel LSTM replicates the human auditory and visual processing pathways,
enabling the emotional information of music and facial expressions to be
effectively processed. As a result, tangible affect recognition may not be
successful, since people may intentionally or unintentionally cover up their
feelings (called social masking) [20].

###  I-A Motivation

Increasingly, scientists in the fields of ergonomics and intelligent systems
are working to improve the effectiveness and adaptability of human-computer
interaction (HCI) in various situations. Computers’ ability to comprehend
human emotions and behavior is a crucial component of their flexibility. The
performance of the state-of-the-art methods and their implications for
recognition have not been fully addressed in previous surveys. The most
important contribution of our review is that it addresses all areas of
emotional computers through a variety of research methods and findings, as
well as discussions and future efforts. Currently, most HCI technologies
cannot recognize human emotion, but the development of advanced HCI systems
depends on automatic computer recognition of human emotion.

###  I-B Contributions

The performance of state-of-the-art methods and their implications for their
recognition ability have not been fully addressed in previous reviews. Our
review offers a variety of research methods and findings, as well as
discussions and future efforts, to address all aspects of emotional computers.
The major contributions of this paper are as follows.

  * •

To the best of our knowledge, this is the only survey of its kind to use ML to
categorize emotional processing into broad categories of emotion
identification.

  * •

By examining how well the various affective modality is utilized to study and
identify affect, we provide a comprehensive classification of the state-of-
the-art emotional computational tools.

  * •

To the best of our knowledge, this is the only survey that classifies
emotional processing using mixed reality machines and even further taxonomies
them based on augmented and virtual reality.

  * •

In order to categorize benchmark databases for emotional computing, audio,
speech, text, and visual modalities are used. Those resources’ salient
features and accessibility are outlined.

  * •

Current approaches and tools are discussed along with their limitations.

  * •

Finally, we outline several open research problems in the field of augmented
reality affective computing and ML.

Table I: List of acronyms Acronym |  Definition
---|---
VER |  Visual Emotion Recognition
SER |  Speech emotion recognition
AER |  Audio emotion recognition
VR |  Virtual Reality
AR |  Augmented Reality
XR |  Mixed Reality
ML |  Machine learning
SVM |  Support vector machine
KNN |  K-Nearest Neighbor
RF |  Random Forest
ANN |  Artificial Neural network
DT |  Decision Tree
ER |  Emotion regulation
HCI |  Human Computer Interface
FER |  Facial emotion Recognition
Figure 2: Organization of this survey paper.

The remainder of the paper is organized as follows. The scope of the paper is
presented in Section II. The existing survey papers are discussed in Section
III. In Section IV, a number of important techniques are compared. The dataset
is covered in detail in Section V. In Section VI, the main challenges of the
paper are summarized and several research directions are identified. Finally,
Section V draws the conclusions. An overview framework is shown in Figure 2 to
improve the general readability of the paper.

##  II Scope of this survey

We will discuss the approach used in this literature review in this section.
Affective computing as it pertains to ML, deep learning, and mixed reality is
the focus of this survey, to the best of our knowledge. As far as we know, no
other study has taken into account affective computing in terms of both ML and
mixed reality. The relevant articles between 2014 and 2021 are considered.
There are three main models in this survey: identification, screening, and
inclusion.

As the first step of this survey, ’Identification’ is used to classify the
data from all the databases. To conduct this survey, we consider some research
questions and keywords. VR, AR, affective computing, ML, supervised learning,
and emotion recognition are explored and acquired from various publishers
icnluding Institute of Electrical and Electronics Engineers (IEEE), Springer,
Hindawi, and Association for Computing Machinery (ACM).

We consider only the relevant articles from all the selected articles during
this step. The duplicate record is removed using EndNote X9 for relevant
articles that are not redundant. To determine whether the study articles’
titles and abstracts were relevant to a developed research topic, the titles
and abstracts were manually reviewed. We evaluated research papers that
contributed a significant amount but were highly relevant.

Data selected following the qualifying stage are subjected to quantitative and
qualitative evaluations at this stage of the research process. Adding records
is the first step in the quality check analysis. During the last stage, known
as qualitative analysis, some of these records are analyzed using meta-
analysis. As a final step, data is extracted from such data that were included
in the survey’s final analysis.

##  III Related Work

The key findings of this study are summarized in Table 2 in comparison with
earlier research in the area of affective computing. In no previous study for
emotion recognition, ML-based and mixed reality approaches have been examined
concurrently. Prior research also ignored the numerous disciplines that
contribute to the field of affective computing, as well as their ideas,
models, and methodologies. This study discusses the classification and
recognition of emotions in terms of their categories and methods for detecting
them. We have also examined the challenges, remedies, and possible trends of
research studies.

Researchers provide a systematic review of emotion models, databases, and
recent advances in [21], which includes both physical and psychological
studies. In addition, they examined the topologies and capabilities of
cutting-edge multimodal emotional analysis and unimodal affect identification.
A mixed reality or ML approach to emotion recognition was not considered by
the authors of [22]. Affective computing is only considered in the context of
interdisciplinary fields like psychology and computer science, along with
their theories, concepts, models, and implications. Additionally, they present
some existing affective databases. In [23], authors surveyed the state-of the
art studies ¡of ????¿. Their main focus was on the types of features in
emotions datasets as they reviewed affective image contents. In affective
computing, they discuss approaches to identifying emotions using features
extraction. The authors of [24] described the state-of-the-art affective
computing technologies for large-scale heterogeneous multimedia data in a
comprehensive manner. Using handcrafted features-based features, they compare
relevant techniques on AC of many multimedia forms, including photos, music,
movies, and multimodal data.

Table II: Summary of existing surveys. Research |  Year |  AR/VR |  ML |  All four categories A/V/T/S |  DB |  Wearables |  Accuracy |  Applications
---|---|---|---|---|---|---|---|---
[21] |  2022 |  $\times$ |  ✓ |  ✓ |  ✓ |  $\times$ |  ✓ |  ✓
[22] |  2021 |  $\times$ |  $\times$ |  ✓ |  $\times$ |  $\times$ |  $\times$ |  ✓
[23] |  2018 |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$ |  $\times$ |  ✓
[24] |  2019 |  $\times$ |  ✓ |  ✓ |  ✓ |  $\times$ |  $\times$ |  $\times$
[25] |  2018 |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$
[26] |  2018 |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$ |  ✓ |  ✓
[27] |  2018 |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$ |  ✓ |  ✓
[28] |  2020 |  $\times$ |  ✓ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |
[29] |  2019 |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$ |  ✓ |  $\times$
[30] |  2022 |  ✓ |  $\times$ |  $\times$ |  $\times$ |  ✓ |  $\times$ |  ✓
Our Survey |  2023 |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓

Additionally, this study examines the different uses, problems, and upcoming
challenges of emotion detection. In order to achieve the study’s objective,
the following research questions were developed:

  * •

(RQ1) What different interconnected domains are a part of emotion detection?

  * •

(RQ2) What are the most common fields wherein affective computing is used?

  * •

(RQ3) Which recent research questions might have an impact on future work in
emotion detection?

  * •

(RQ4) How ML and Mixed Reality recognizes the emotions.

A thorough search on the Microsoft academic research portal was conducted to
obtain a comprehensive understanding of the published literature on affective
computing.

Figure 3: The overall framework of this survey

##  IV Emotion Models

The definition of the emotion or affect is crucial to building an emotional
computing standard. Ekman articulated the fundamental idea of emotions for the
first time in the 1970s. Despite psychologists’ efforts to categorize emotions
in neuroscience, philosophy, and computer science, there is no universally
accepted emotion model. Generic emotion models used in affective computing
include continuous emotion models and multidimensional emotion models
(sometimes called continuous emotion models).

##  V Work Flow of Emotion Recognition

In this section, we discuss the workflow and contributions of all forms of
affective computing, including ML, deep learning, and virtual reality.

###  V-A Textual Based Emotion Recognition

The majority of methods for textual emotion recognition, or TSA, are based on
statistical or knowledge-based approaches [34]. In the former case, an
extensive emotional vocabulary is required to model a thesaurus, while in the
latter case, a large database with emotional labels is required. Online social
media and e-commerce systems are rapidly expanding, allowing users to freely
express their opinions, which generates a significant amount of text data. A
text sentiment classification method was developed to detect subtle sentiments
or emotions expressed directly or implicitly from user-generated material
[31].

The term ”feature engineering” is often used to identify characteristics that
are associated with emotion by using traditional approaches [32]. Through the
use of DL-based models, it is possible to create an end-to-end sentiment
analysis of text data. Statistics and knowledge-based methods are mainly used
in conventional ML-based TSAs [34]. It is common for semantically polarized
terms to be mixed up in various lexicons, including Word-Net, WordNet Affect,
Sentic-Net, and Senti-WordNet [33]. Table 3 presents the articles and their
contribution to text-based emotion detection using deep learning and ML
techniques. Table 3 provides an overview of the main contributions along with
datasets and evaluation parameters.

Table III: An overview of text based emotion recognitions studies  Research |  Contributions |  Dataset |  Evaluation
---|---|---|---
[34] |  Lexicon-based strategy, external evidences system, Context dependent approach |  Review Dataset |  Accuracy
[35] |  Baseline lexicons, Train word-class correlations, |  Weblogs |  Accuracy
[36] |  Concept-level sentiment analysis, logic computing, ML |  VER |  Visual Emotion Recognition
[37] |  Multidisciplinary sentiment classification, Twitter ad hoc conversations of significant conflicts |  Twitter |  Accuracy
[38] |  Asses sentimenatl analysis, context for lexicons methodologies |  Twitter |  Accuracy
[39] |  Sentiment classification using SVMs, combination of phrases and adjective for favorability measures . |  Music review |  Accuracy
[40] |  The opinion-level framework, intra-opinion features and inter-opinion, Bayesian model |  Twitter |  Accuracy
[41] |  Strengthen the impartial, boundary between positive and negative reviews |  Amazon, cinema, movies, |  Accuracy
[42] |  Hybrid strategy, combination of deep learning and a vocabulary method |  Amazon, IMBD and Yelp |  Accuracy
[43] |  Identification of Internet slang and emoticons, k-Nearest Neighbors, Decision Tree, Random Forest, Logistic Regression, Naive Bayes, and Support Vector Machine |  Weibo |  Accuracy
[44] |  New word embedding technique, Dynamically adjust vector representation |  IMDB Yelp 13, 14 and 15 |  Accuracy
[45] |  Novel text processing design (VDCNN), character level, tiny convolutions and pooling procedures |  Amazon |  Accuracy
[46] |  Unique parameterized convolutional neural network, Categorization of sentiments, Incorporate aspect data into CNN |  SemEval 2014 |  Accuracy
[47] |  LM employing contextual BLSTM (cBLSTM), Modified version of bidirectional LSTM (BLSTM) |  IMDB |  Accuracy
[48] |  Joint framework for explicit aspect, Opinion terms co-extraction unifies recursive neural networks |  Laptop |  Accuracy
[49] |  Examination of two distinct types of tweets obtained during the COVID19 pandemic |  Twitter |  Accuracy

###  V-B Audio Emotion Recognition

A process of processing and understanding speech signals is used in audio
emotion recognition (also known as SER) [50]. There are a number of ML- and
DL-based SER systems that have been implemented to enhance analysis [51, 52].
In ML-based SER, the extraction of acoustic features and the selection of
classifiers are the primary focuses. A DL-based SER constructs a CNN
architecture without taking feature engineering and selection into account,
however, in order to forecast the final feeling [53].

In ML-based SER systems, two crucial processes take place: learning
representations of emotional speech and selecting an appropriate
classification for the final emotion prediction. A well-known audio feature
extraction tool called OpenSMILE [54] extracts all the essential components of
speech. For SER systems, HMMs, GMMs, SVMs, RFs, and ANNs are frequently
employed as classifiers. The SER also uses maximum classification and enhanced
traditional easy-to-understand classifiers in addition to these learners. The
following Table 4 shows an overview of the main contributions, along with the
dataset and evaluation parameters.

Table IV: An overview of audio based emotion recognitions studies Research |  Main Contributions |  Dataset |  Evaluation
---|---|---|---
[55] |  Examination of two types of tweets obtained during the COVID19 |  Speaker language |  Accuracy
[56] |  Utilization of prosodic features, Utilization of voice quality parameters, Investigation of the interactions between prosodic and voice quality, Utilization of the bayesian classifier to recognize emotions |  Speaker |  Accuracy
[57] |  Fuse the features, multiple kernels learning-based approach |  Berlin database |  Accuracy
[58] |  Deep Neural Network based identification of emotions |  Speaker |  Accuracy
[59] |  Validate the use of a corpus of semantically impartial text, Transcription of both neutral and emotional (acted) speech |  TTS |  Accuracy
[60] |  Automated speech emotion detection algorithm, Computer model of the human auditory system |  Speaker |  Accuracy
[61] |  Classification on three corpora, namely the RAVDESS, IITKGP-SEHSC, and the Berlin EmoDB using SVM as classifier |  Speaker |  Accuracy
[62] |  Merging of personalized and non-personalized features for speech emotion, Utilze fuzzy C-means clustering algorithm, Employe several trees to identify various emotional states |  CASIA |  Accuracy
[63] |  Extraction of keypoints from spectrogram, Utilizion of CNN, Predictions of seven emotions |  Speaker |  Accuracy
[64] |  Performs auto encoders, Utilzioon of Recurrent neural networks, Classify four fundamental moods, Recognize emotions from spectrogram features |  Speaker |  Accuracy

###  V-C Visual Emotion Recognition

Images or films with face emotional clues are used to execute FER [65]. FER
techniques are summarized in Table 5 as representative examples.

Table V: An overview of visual based emotion recognitions studies Research |  Main Contributions |  Dataset |  Evaluation
---|---|---|---
[66] |  Fully automatic face expression identification, Utilize subsequent frames based on elastic bunch graph, |  CK+ |  Accuracy
[67] |  Fuzzification, Track AAM characteristics |  custom |  Accuracy
[68] |  Linear discriminant analysis to recognize emotions |  JAFFEE MMI SFEW |  Visual Emotion Recognition
[69] |  Auto–encoders, SOM-based classification, Use of deep neural network, Combine geometric characteristics with LBP features |  MMI |  Accuracy
[70] |  Auto–encoders, SOM-based classification, deep neural network, Combine geometric characteristics with LBP features |  CASME II |  Accuracy
[71] |  Subtle deformations in emotion classification |  BU-4DFE |  Accuracy
[72] |  FER employing multi-model 2D and 3D films, convolution network, Encode both static and dynamic information |  BU-4DFE |  Accuracy
[73] |  Recognition of face micro-expressions, weighting scheme as well as the excellent spatiotemporal descriptor HOG3D for action recognition |  HOG3D |  Accuracy
[74] |  TCreation of the neutrality facial images |  BU-4DFE CK+ MMI |  Accuracy
[75] |  The ”identification by generating” system, brand-new hard negative generation (HNG) network and a generalized radial metric learning (RML) network, is the main contribution of this research. |  CK+, MMI |  Accuracy
[76] |  CNN to operate as distinct streams within a bi-stream identity-aware net |  MMI CK+ |  Accuracy
[77] |  Attention-based Salient Expressional Region Descriptor (SERD) and the Multi-Path Variation-Suppressing Network modules |  FER |  Accuracy
[78] |  Uncertainty using a straightforward but effective Self-Cure Network (SCN) |  SemEval 2014 |  Accuracy
[79] |  Dynamic-temporal stream, static-spatial stream, and local-spatial stream module for the TSCNN, Aims to learn and incorporate time, complete facial area, and nearest neighbor signals again for face |  CASME II, SMIC-HS, SAMM |  Accuracy
[80] |  Deep emotion-conditional adaptation net (ECAN) that can learn domain-invariant and discriminative feature representations |  CK+ MMI JAFFE |  Accuracy
[81] |  Transfer knowledge of Emotional Education Mechanism (EEM), Self-taught student network and a competent teacher network (STSN) |  Twitter |  Accuracy
[82] |  Convolutional networks to encode the spatial properties of microexpressions at several expression-states |  CASME-II |  Accuracy
[83] |  Siamese-cascaded metric learning framework that teaches fine-grained distinctions between expressions in video-based tasks |  CK+, Oulu, MMI |  Accuracy

###  V-D Virtual Reality Based Emotion Detection

Emotional representation can be explained by several models (Section II). The
Geneva Emotion Wheel [84], Differential Emotions Scale [85], and Appraisal
Models are some of the models that use specific items to identify emotions
depending on the events as they are framed in dimensional models (valence,
arousal, and dominance). Analyzing emotional reactions utilizing multiple
entities is crucial for grasping complicated emotions and capturing
disparities in emotional responses. In light of this, it’s crucial to examine
how VR is able to evoke different types of emotions. Consequently, this
section examines the potential use of various VR media formats in the
induction of various behaviors.

##  VI Datasets

Our purpose in this section is to describe the datasets we use to detect
emotions in text, audio, and visual images. There are three types of affective
computing databases: textual, speech, audio, and visual. Models for emotional
computing and network architectures are greatly influenced by these databases’
characteristics.

###  VI-A Text Based Dataset

The TSA database contains text data at various granularities, like words,
sentences, and documents. The Multi-domain Sentiment (MDS) [86] database
contains more than 100,000 phrases derived from Amazon.com reviews. These
sentences are divided into two emotion types (positive and negative) and five
sentiment categories. IMDB [87] is another significant resource that is
frequently utilized for categorizing binary sentiment. There are 25,000
reviews of extremely divisive movies available for testing and 25,000 for
training. The Stanford University-annotated semantic lexical database is
called Stanford Sentiment Treebank (SST) [88]. It has 215,154 phrases with
fine-grained emotional descriptors.

###  VI-B Audio based dataset

There are two types of speech databases: spontaneous and non-spontaneous
(simulated and induced). The first non-spontaneous voice collection was
created primarily from professional actors’ performances. Due to their ability
to professionally mimic well-known emotional traits, these performance-based
datasets are considered trustworthy. There are over 500 words in the Berlin
Database of Emotional Speech (Emo-DB) [89] spoken by 10 actors (five men and
five women) in various emotional states, including rage, anxiety, fear,
boredom, and contempt. Artificial emotions, however, are more likely to be
exaggerated than actual feelings. In an effort to close this gap, databases
have been created for spontaneous speech.

###  VI-C Visual base dataset

To create the initial FER datasets, lab volunteers voluntarily expressed their
emotions (in-the-lab). A 1998 publication entitled JAFFE [66] contains 213
photographs of 10 Japanese female models expressing 7 different facial
expressions. In order to develop the Cohn-Kanade+ (CK) [90], subjects were
instructed to make 7 different facial expressions. In order to offer methods
and benchmark findings for tracking features of face, action units (AUs), and
identifying emotions, photos of face expressions were captured and evaluated.
MMI [91], in contrast to CK, contains onset apex offset orders. Multiple views
and multiple poses are supported by many 3D or 4D datasets made for FER. For
the Binghamton University 3D Facial Expression (BU-3DFE) [92] database, 606
facial expression segments were taken from 100 people using six different
facial expressions.

A huge, unrestricted collection called FER2013 [98] contains 35,887 grey
images with a resolution of 48 48 pixels that were autonomously gathered using
the image search API 950,000 of the one million photos in EmotioNet [93] were
generated automatically. 91,793 facial photos from Emotion in the Wild (ExpW)
[94] have each been individually labeled using one of the seven fundamental
facial expressions. The annotating technique eliminated non-face photos. Over
one million facial photos are included in AffectNet [95], of which 4 lac 50k
images have been manually classified as one of eight distinct expressions
(neutral, disdain, the six fundamental emotions, and more). AffectNet also
includes information on the dimensional intensity of valence and arousal. The
Real-world Affective Face Database (RAF-DB)[96] is a collection of 29,672
extremely different facial photos annotated by the public and acquired from
the Internet (seven basic and eleven compound emotion labels). Over 16,000
video snippets that were cut from hundreds of films with different topics make
up Dynamic-Facial-Expressions in the Wild (DFEW) [97].

Figure 4: Sample images of openly available databases. Table VI: Available databases for research purposes Database |  Number of samples |  Emotions
---|---|---
Oulu [98] |  80 |  SBE
FER2013 [99] |  32 |  SBE
SFEW [100] |  32 |  SBE
EmotionNEt [93] |  32 |  Compound
Oulu [98] |  32 |  SBE
ExpW [94] |  32 |  SBE+
AffectNEt [95] |  32 |  SBE+
CASME II [101] |  35s |  SBE +
SMIC [102] |  16 |  3, positive, negatice, surprice
4DFAB [103] |  180 |  SBE
3DFE [92] |  100 |  SBE
MMI [91] |  25 |  SBE
CK+ [90] |  123 |  SBE+
CK [104] |  210 |  SBE+
JAFFE [105] |  10 |  SBE

##  VII Challenges

ML-based techniques have been used in affective computing’s early literature
[15]. The ML pipeline [106]entails the pre-processing of unprocessed signals,
the creation of expertly constructed feature extractors (including selecting
features when practical), and masterfully crafted classifiers. ML-based
approaches for affective analysis are difficult to reuse across similar issues
because to their task-specific and domain-specific feature descriptors,
despite the fact that many types of hand-crafted features have been generated
for diverse modality. The most popular ML-based classifier are the SVM
classifier[107], GMM classifier [59], RF classifier [108], KNN classifier
[109], and ANN classifier [110], with the SVM classifier being employed in the
majority of ML-based affective computing tasks.

Due to their superior feature representation learning capabilities, DL-based
models have recently gained popularity and outperformed ML-based models in the
majority of affective computing applications [111]. CNNs and their derivatives
are made to extract significant and discrete characteristics from static
information (such as face and spectral analysis images) [112]. RNNs and its
variants are made for capturing temporal dynamics for sequence information
(such as physiological signals and movies) [113]. The deep spatial-temporal
feature extraction is a task that can be handled by CNN-LSTM models. By
enhancing data and using cross-domain learning, adversarial learning is
frequently employed to increase the robustness of models [114]. Additionally,
to enhance overall performance, various attention methods and auto encoders
are used with DL-based techniques. The most discriminative traits appear to be
automatically learned by DL-based algorithms, which appears to be an
advantage. When compared to ML-based models, DL-based techniques have not yet
had a significant influence on physiological emotion recognition.

In the field of emotion research, VR is being used more and more, but there
are some limitations. Therefore, standards must be developed and implemented.
Research can be conducted more effectively and restrictions can be reduced by
applying these principles. It is important for researchers to assess the
likelihood of different kinetic settings on the typical population in order to
decrease motion sickness. By filtering participants at an early stage, it
becomes easier to collect data. A questionnaire could be used as the primary
method of contacting the individual.

A few of the questionnaires that can be found in the literature are the Motion
Sickness Susceptibility Questionnaire (MSSQ) and the Virtual Reality Sickness
Questionnaire (VRSQ) [68]. It is possible for the reader to consult.
Participants must have prior exposure to immersive VR environments, as with
the first. Some research findings may be influenced by the novelty bias caused
by using VR for the first time. Further explanations can be found in the
differences between predicted and actual feelings.

People with previous VR experience or training sessions with selected
participants before data collection can lessen the effects of the first-time
VR experience. As a result of understanding the perspective of a VR world and
the operation of hardware, especially when using VR controllers, participants
receive additional benefits. For credible research, it is necessary to confirm
the applicability of using those hardware interfaces integrated with different
facial structures.

##  VIII Open Research Problems

The field of affective computing focuses on developing machines and systems
that can recognize, interpret, and respond to human emotions.

  1. 1.

How to recognize and interpret complex emotional states more accurately and
reliably is one of the open research problems in affective computing.

  2. 2.

The lack of large and diverse labeled datasets to train emotion recognition
models is one of the open research problems in affective computing. For ML
algorithms to recognize and interpret emotions accurately, large amounts of
labeled data are required. Existing emotion recognition datasets, however, are
often limited in size, scope, and diversity, which can limit their performance
and generalizability.

  3. 3.

There is also a lack of consistency in the labeling of emotions. A standard
labeling scheme for emotions does not exist, since emotions are interpreted
differently by different cultures and individuals. It may be difficult to
compare results across studies and develop emotion recognition models that are
effective across diverse populations because of this.

  4. 4.

Another open research problem is how to develop more robust and interpretable
ML models for affective computing. ML models used in affective computing, such
as deep neural networks, are known as ”black box” models, which make
understanding how they work difficult. As a result, emotion recognition models
are difficult to trust and fine-tune for specific applications due to their
lack of interpretability.

  5. 5.

A more personalized and adaptive emotion recognition model is needed that can
take into account individual differences and changes in emotional states over
time. Developing models that adapt to individual users’ emotional patterns
requires learning from their emotional expressions.

  6. 6.

The development of ML approaches for affective computing and the development
of more accurate and effective emotion recognition systems will be made
possible by addressing these research problems.

##  IX Conclusion

This survey examined research works based on current studies on affective
computing and an illustrative taxonomy of affective computing. Various
corresponding measures were used to assess the recognition outcomes obtained
by classification or regression. Emotional computing develops computational
models for DL-based or ML-based affective understanding, as well as benchmark
databases for training. A survey of the commonly used baseline databases for
affective computing was also presented in this article, including textual,
audio, and visual databases. Most affective analysis techniques can be applied
to these publicly available databases. ML-based methods, DL-based approaches,
and mixed reality-based approaches are some of the recent developments in
affective computing. A survey of text sentiment classification, voice emotion
recognition, and visual emotion recognition (FER and EBGR) is presented. Even
though affective computing systems that use either unimodal or multimodal data
have made substantial progress, there are only a few reliable and potent
algorithms to predict emotion and distinguish feelings. Consequently,
affective computing’s important future research directions are summarized as
follows. New and expanded baseline databases, particularly multimodal effect
databases (textual, audio, visual, physiological), would be crucial. Affective
analysis problems such as FER under occlusions and fake emotion expression
need to be solved. Fusion methodologies, particularly those based on
regulation or statistics, need to be improved. In the presence of constrained
or biased databases, zero/few-shot learning and unsupervised learning
techniques, such as self-supervised learning, can improve the stability and
robustness of affective analysis. The use of emotive analysis in robotics is
well known. As discussed in this paper, emotionally intelligent robots are
capable of accurately mimicking and reacting to emotions.

## References

  * [1] K. S. Fleckenstein, “Defining affect in relation to cognition: A response to susan mcleod,” _Journal of Advanced Composition_ , vol. x, pp. 447–453, 1991\.
  * [2] R. W. Picard, “Affective computing,” _Journal of Advanced Composition_ , vol. 1, pp. 447–453, 2000.
  * [3] R. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: Analysis of affective physiological state,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 23, pp. 1175–1191, 2001.
  * [4] J. Park, J.-H. Kim, and Y.-H. Oh, “Feature vector classification based speech emotion recognition for service robots,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 55, pp. 1590–1596, 2009.
  * [5] K. Anderson and P. W. McOwan, “A real-time automated system for the recognition of human facial expressions,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 36, pp. 96–105, 2006\.
  * [6] Z. Yin, M. Zhao, Y. Wang, J. Yang, , and J. Zhang, “Recognition of emotions using multimodal physiological signals and an ensemble deep learning model,” _Computer methods and programs in biomedicine_ , vol. 140, pp. 93–110, 2017\.
  * [7] M. S. Akhtar, D. S. Chauhan, D. Ghosal, S. Poria, A. Ekbal, and P. Bhattacharyya, “Multi-task learning for multi-modal emotion recognition and sentiment analysis,” _arXiv preprint_ , vol. X, p. 1905.05812, 2019.
  * [8] G. Chanel, J. Kronegg, D. Grandjean, , and T. Pun, “Emotion assessment: Arousal evaluation using eeg’s and peripheral physiological signals,” _Multimedia Content Representation, Classification and Security: International Workshop, MRCS_ , vol. 1, pp. 530–537, 2006.
  * [9] L. A. Schmidt and L. J. Trainor, “Frontal brain electrical activity (eeg) distinguishes valence and intensity of musical emotions,” _Cognition and Emotion_ , vol. 15, no. 4, pp. 487–500, 2001.
  * [10] L.-P. Morency, J. Whitehill, and J. Movellan, “Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation,” _In 2008 8th IEEE International Conference on Automatic Face and Gesture Recognition_ , vol. 1, pp. 1–8, 2008.
  * [11] E. Cambria, D. Das, S. Bandyopadhyay, and A. Feraco, “Affective computing and sentiment analysis,” _A practical guide to sentiment analysis_ , vol. 1, pp. 1–10, 2017.
  * [12] M. Munezero, C. S. Montero, E. Sutinen, and J. Pajunen, “Are they different? affect, feeling, emotion, sentiment, and opinion detection in text,” _IEEE transactions on affective computing_ , vol. 5, no. 2, pp. 101–111, 2014\.
  * [13] Ekman, Paul, T. Dalgleish, and M. Power, “Basic emotions,” _Handbook of cognition and emotion_ , vol. 98, no. 45-60,, p. 16, 1999.
  * [14] Mehrabian and Albert., “Basic dimensions for a general psychological theory: Implications for personality, social, environmental, and developmental studies,” 1980.
  * [15] C. A. Corneanu, M. O. Simón, J. F. Cohn, and S. E. Guerrero, “Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 8, pp. 1548–1568, 2016.
  * [16] Z. Yin and J. Zhang, “Task-generic mental fatigue recognition based on neurophysiological signals and dynamical deep extreme learning machine,” _Neurocomputing_ , vol. 283, pp. 266–281, 2018.
  * [17] A. J. Holte and F. R. Ferraro, “True colors: Grayscale setting reduces screen time in college students,” _The Social Science Journal_ , vol. x, pp. 1–17, 2020.
  * [18] W. Chen, “A novel long short-term memory network model for multimodal music emotion analysis in affective computing,” _Journal of Applied Science and Engineering_ , vol. 26, pp. 367–376., 2022.
  * [19] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, and S. Gao, “A systematic review on affective computing: emotion models, databases, and recent advances,” _Information Fusion_ , vol. x, pp. 1–8, 2022.
  * [20] J. Zhang, Z. Yin, P. Chen, and S. Nichele, “Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review,” _Information Fusion_ , vol. 59, pp. 103–126, 2020.
  * [21] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, and W. J. I. F. Zhang, “A systematic review on affective computing: Emotion models, databases, and recent advances,” _Information Fusion_ , vol. x, p. x, 2022.
  * [22] R. Arya, J. Singh, and A. J. C. S. R. Kumar, “A survey of multidisciplinary domains contributing to affective computing,” _Computer Science Review_ , vol. 40, p. 100399, 2021.
  * [23] S. Zhao, G. Ding, Q. Huang, T.-S. Chua, B. Schuller, and K. Keutzer, “Affective image content analysis: A comprehensive survey,” 2018.
  * [24] S. Zhao, S. Wang, Soleymani, Joshi, and Q. Ji, “Affective computing for large-scale heterogeneous multimedia data: A survey,” _ACM Transactions on Multimedia Computing, Communications, and Applications_ , vol. 15(3), pp. 1–32., 2019.
  * [25] B. C. J. s. Ko, “A brief review of facial emotion recognition based on visual information,” _sensors_ , vol. 18, p. 401, 2018.
  * [26] S. Li and W. J. I. t. o. a. c. Deng, “Deep facial expression recognition: A survey,,” _IEEE transactions on affective computing_ , vol. 13(3, pp. 1195–1215, 2020.
  * [27] Merghani, Walied, A. K. Davison, and M. H. Yap, “A review on facial micro-expressions analysis: datasets, features and metrics,” _arXiv preprint_ , vol. x, p. x, 2018.
  * [28] G. R. Alexandre, J. M. Soares, and G. A. P. J. P. R., “Thé, “systematic review of 3d facial expression recognition methods,” _Pattern Recognition,_ , vol. 100, p. 107108, 2020.
  * [29] S. Poria, N. Majumder, R. Mihalcea, and E. J. I. A. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” _IEEE Access_ , vol. 7, pp. 100 943–100 953, 2019.
  * [30] R. Somarathna, T. Bednarz, and G. J. I. T. o. A. C. Mohammadi, “Virtual reality for emotion elicitation–a review,” _IEEE Transactions on Affective Computing_ , vol. x, p. x, 2022.
  * [31] F. Pozzi, E. Fersini, E. Messina, and B. Liu, “Sentiment analysis in social networks,” vol. x, p. x, 2016.
  * [32] Stone, P. J., and E. B. Hunt, “A computer approach to content analysis: studies using the general inquirer system,” _Proceedings of the May_ , vol. x, pp. 241–256, 1963.
  * [33] Suttles, Jared, and N. Ide, “Distant supervision for emotion classification with discrete binary values,” _Computational Linguistics and Intelligent Text Processing: 14th International Conference_ , vol. x, pp. 121–136, 2013.
  * [34] X. Ding, B. Liu, and P. S. Yu, “A holistic lexicon-based approach to opinion mining.” _In Proceedings of the 2008 international conference on web search and data mining_ , vol. x, pp. 231–240, 2008.
  * [35] P. Melville, W. Gryc, and R. D. Lawrenc, “Sentiment analysis of blogs by combining lexical knowledge with text classification,” _In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_ , vol. 1, pp. 1275–1284, 2009.
  * [36] S. Poria, E. Cambria, G. Winterstein, and G.-B. Huang, “Sentic patterns: Dependency-based rules for concept-level sentiment analysis,” _Knowledge-Based Systems_ , vol. 69, pp. 45–63, 2014.
  * [37] I. Blekanov, M. Kukarkin, A. Maksimov, and S. Bodrunova, “Sentiment analysis for ad hoc discussions using multilingual knowledge-based approach.” _Proceedings of the 3rd International Conference on Applications in Information Technology_ , vol. x, pp. 117–121, 2018.
  * [38] E. Aydoğan and M. A. Akcayol, “A comprehensive survey for sentiment analysis tasks using machine learning techniques.” _2016 international symposium on innovations in intelligent systems and applications (INISTA)_ , vol. x, pp. 1–7, 2016.
  * [39] T. Mullen and N. Collier, “Sentiment analysis using support vector machines with diverse information sources,” _Proceedings of the 2004 conference on empirical methods in natural language processing_ , vol. x, pp. 412–418, 2004\.
  * [40] Y. Xia, E. Cambria, A. Hussain, and H. Zhao, “Word polarity disambiguation using bayesian model and opinion-level features,” _Cognitive Computation_ , vol. 7, pp. 369–380, 2015.
  * [41] A. Valdivia, M. V. Luzón, E. Cambria, and F. Herrera, “Consensus vote models for detecting and filtering neutrality in sentiment analysis,” _Information Fusion_ , vol. 44, pp. 126–135, 2018.
  * [42] T. Le, “A hybrid method for text-based sentiment analysis,” _2019 International Conference on Computational Science and Computational Intelligence (CSCI)_ , vol. x, pp. 1392–1397, 2019.
  * [43] D. Li, R. Rzepka, M. Ptaszynski, and K. Araki, “A novel machine learning-based sentiment analysis method for chinese social media considering chinese slang lexicon and emoticons.” _AffCon@ AAAI_ , vol. x, p. 2328, 2019.
  * [44] F. Liu, L. Zheng, and J. Zheng, “Hienn-dwe: A hierarchical neural network with dynamic word embeddings for document level sentiment classification,” _Neurocomputing_ , vol. 403, pp. 21–32, 2020.
  * [45] A. Conneau, H. Schwenk, L. Barrault, and Y. Lecun, “Very deep convolutional networks for text classification,” _rXiv preprint arXiv:1606.01781_ , vol. x, p. x, 2016.
  * [46] B. Huang and K. M. Carley, “Parameterized convolutional neural networks for aspect level sentiment classification,” _arXiv preprint arXiv:1909.06276,_ , vol. x, p. x, 2019.
  * [47] A. Mousa and B. Schuller, “Contextual bidirectional long short-term memory recurrent neural network language models: A generative approach to sentiment analysis,” _Proceedings of the 15tConference of the European Chapter of the Associationfor Computational Linguistics_ , vol. 1, pp. 1023–1032,, 2017\.
  * [48] W. Wang, S. J. Pan, D. Dahlmeier, and X. Xiao, “Recursive neural conditional random fields for aspect-based sentiment analysis,” _arXiv preprint arXiv:1603.06679_ , vol. x, p. x, 2016.
  * [49] K. Chakraborty, S. Bhatia, S. Bhattacharyya, J. Platos, R. Bag, and A. E. Hassanien, “Sentiment analysis of covid-19 tweets by deep learning classifiers—a study to show how popularity is affecting accuracy in social media,” _Applied Soft Computing_ , vol. 97, p. 106754, 2020.
  * [50] S. Narayanan and C. M. Lee, “Toward detecting emotions in spoken dialogs,” _IEEE Trans. Speech Audio Process_ , vol. 13(2), pp. 293–303, 2005.
  * [51] J. Pohjalainen, F. F. Ringeval, Z. Zhang, and B. Schuller, “Spectral and cepstral audio noise reduction techniques in speech emotion recognition,” _Proceedings of the 24th ACM international Conference on Multimedia_ , vol. x, pp. 670–674, 2016.
  * [52] Z. Huang, M. Dong, Q. Mao, and Y. Zhan, “Speech emotion recognition using cnn,” _Proceedings of the 22nd ACM international conference on Multimedia_ , vol. x, pp. 801–804, 2014.
  * [53] H. M. Fayek, M. Lech, and L. Cavedon, “Evaluating deep learning architectures for speech emotion recognition,” _Neural Networks_ , vol. 92, pp. 60–68, 2017.
  * [54] F. Eyben, M. Wöllmer, and B. Schuller, “Openear—introducing the munich open-source emotion and affect recognition toolkit,” _2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE_ , vol. x, pp. 1–6, 2009.
  * [55] C. Busso, S. Lee, and S. Narayanan, “Analysis of emotionally salient aspects of fundamental frequency for emotion detection,” _IEEE transactions on audio, speech, and language processing_ , vol. 17(4), pp. 582–596, 2009.
  * [56] M. Lugger and B. Yang, “The relevance of voice quality features in speaker independent emotion recognition,” _2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07_ , vol. 4, pp. IV–17, 2007\.
  * [57] Y. Jin, P. Song, W. Zheng, and L. Zhao, “A feature selection and feature fusion combination method for speaker-independent speech emotion recognition,” _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , vol. x, pp. 4808–4812, 2014.
  * [58] D. Bitouk, R. Verma, and A. Nenkova, “Class-level spectral features for emotion recognition,” _Speech communication_ , vol. 52(7-8), pp. 613–625, 2010.
  * [59] E. Navas, I. Hernáez, and I. Luengo, “An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional tts,” _IEEE transactions on audio, speech, and language processing_ , vol. 14(4), pp. 1117–1127, 2006.
  * [60] E. Yüncü, H. Hacihabiboglu, and C. Bozsahin, “Automatic speech emotion recognition using auditory models with binary decision tree and svm,” _2014 22nd international conference on pattern recognition. IEEE_ , vol. x, pp. 773–778, 2014.
  * [61] A. Bhavan, P. Chauhan, and R. R. Shah, “Bagged support vector machines for emotion recognition from speech,” _Knowledge-Based Systems_ , vol. 184, p. 104886, 2019.
  * [62] L. Chen, W. Su, Y. Feng, M. Wu, J. She, and K. Hirota, “Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction,” _Information Sciences_ , vol. 509, pp. 150–163, 2020.
  * [63] A. M. Badshah, J. Ahmad, N. Rahim, and S. W. Baik, “Speech emotion recognition from spectrograms with deep convolutional neural network,” _In 2017 international conference on platform technology and service (PlatCon)_ , vol. x, pp. 1–5, 2017.
  * [64] S. Ghosh, E. Laksana, L.-P. Morency, and S. Scherer, “Representation learning for speech emotion recognition,” _In Interspeech_ , vol. x, pp. 3603–3607, 2016.
  * [65] Y.-L. Tian, T. Kanade, and J. F. Colin, “Recognizing action units for facial expression analysis,” _In Multimodal interface for human-machine communication_ , vol. x, pp. 32–66, 2002.
  * [66] D. Ghimire and J. Lee, “Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines,” _Sensors_ , vol. 13(6), pp. 7714–7734, 2013.
  * [67] A. A. Gunawan, “Face expression detection on kinect using active appearance model and fuzzy logic,” _Procedia Computer Science_ , vol. 59, pp. 268–274, 2015.
  * [68] Y. Yan, Z. Zhang, S. Chen, and H. Wang, “Low-resolution facial expression recognition: A filter learning perspective,” _Signal Processing_ , vol. 169, p. 107370, 2020.
  * [69] A. Majumder, L. Behera, and V. K. Subramanian, “Automatic facial expression recognition system using deep network-based data fusion,” _IEEE transactions on cybernetics_ , vol. 48(1), pp. 103–114, 2016.
  * [70] Y. Wang, J. See, R. C.-W. Phan, and Y.-H. Oh, “Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition,” _PloS one_ , vol. 10(5), p. e0124674, 2015.
  * [71] Q. Zhen, D. Huang, H. Drira, B. B. Amor, Y. Wang, and M. Daoudi, “Magnifying subtle facial motions for effective 4d expression recognition,” _IEEE Transactions on Affective Computing_ , vol. 10(4), pp. 524–536, 2017.
  * [72] Y. Yao, D. Huang, X. Y. Y. Wang, and L. Chen, “Texture and geometry scattering representation-based facial expression recognition in 2d+ 3d videos,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , vol. 14(1s), pp. 1–23, 2018.
  * [73] M. Chen, H. T. Ma, J. Li, and H. Wang, “Emotion recognition using fixed length micro-expressions sequence and weighting method,” _In 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR)_ , vol. x, pp. 427–430, 2016.
  * [74] H. Yang, U. Ciftci, and L. Yin, “Facial expression recognition by de-expression residue learning,” _In Proc. of the IEEE conference on computer vision and pattern recognition_ , vol. x, pp. 2168–2177, 2018.
  * [75] X. Liu, B. V. Kumar, P. Jia, and J. You, “Hard negative generation for identity-disentangled facial expression recognition,” _Pattern Recognition_ , vol. 88, pp. 1–12, 2019.
  * [76] Z. Meng, P. Liu, J. Cai, S. Han, and Y. Tong, “Identity-aware convolutional neural network for facial expression recognition,” _In 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017)_ , vol. x, pp. 558–565, 2017.
  * [77] H. H. S. Xie and Y. Wu, “Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition,” _Pattern recognition_ , vol. 92, pp. 177–191, 2019.
  * [78] K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , vol. x, pp. 6897–6906, 2020.
  * [79] B. Song, K. Li, Y. Zong, J. Zhu, W. Zheng, J. Shi, and L. Zhao, “Recognizing spontaneous micro-expression using a three-stream convolutional neural network,” _IEEE Access_ , vol. 7, pp. IEEE Access, vol. 7, pp. 184 537–184 551, 2019.
  * [80] S. Li and W. Deng, “A deeper look at facial expression dataset bias,” _IEEE Transactions on Affective Computing_ , vol. 13, pp. 881–893, 2020.
  * [81] H. Li, N. Wang, X. Ding, X. Yang, and X. Gao, “Adaptively learning facial expression representation via c-f labels and distillation,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2016–2028, 2021.
  * [82] D. H. Kim, W. J. Baddar, and Y. M. Ro, “Micro-expression recognition with expression-state constrained spatio-temporal feature representations,” _Proceedings of the 24th ACM international conference on Multimedia_ , vol. x, pp. 382–386, 2016.
  * [83] D. Liu, X. Ouyang, S. Xu, P. Zhou, K. He, and S. Wen, “Saanet: Siamese action-units attention network for improving dynamic facial expression recognition,” _Neurocomputing_ , vol. 413, pp. 145–157, 2020.
  * [84] Z. Zhao, Y. Zhao, Z. Bao, Z. Z. H. Wang, and C. Li, “Deep spectrum feature representations for speech emotion recognition,” _Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data_ , vol. x, pp. 27–33, 2018.
  * [85] J. Han, Z. Zhang, Z. Ren, F. Ringeval, and B. Schuller, “Towards conditional adversarial training for predicting emotions from speech,” _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , vol. x, pp. 6822–6826, 2018.
  * [86] Y. Ding, N. Robinson, Q. Zeng, D. Chen, A. A. P. Wai, T.-S. Lee, and C. Guan, “Tsception: a deep learning framework for emotion detection using eeg,” _2020 international joint conference on neural networks (IJCNN)_ , vol. x, pp. 1–7, 2020.
  * [87] J. Blitzer, M. Dredze, and F. Pereira, “Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification,” _Proceedings of the 45th annual meeting of the association of computational linguistics_ , vol. x, pp. 440–447, 2007.
  * [88] A. Maas, R. E. Daly, P. T. P. D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , vol. x, pp. 142–150, 2011.
  * [89] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” _Proceedings of the 2013 conference on empirical methods in natural language processing_ , vol. x, pp. 1631–1642, 2013.
  * [90] T. Bänziger and K. R. Scherer, “Introducing the geneva multimodal emotion portrayal (gemep) corpus,” _Blueprint for affective computing: A sourcebook_ , vol. 2010, pp. 271–94, 2010.
  * [91] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for facial expression analysis,” _Proceedings fourth IEEE international conference on automatic face and gesture recognition_ , vol. x, pp. 46–53, 2000.
  * [92] M. Valstar and M. Pantic, “Induced disgust, happiness and surprise: an addition to the mmi facial expression database,” _Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect_ , vol. x, p. 65, 2010.
  * [93] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, and D.-H. Lee, “Challenges in representation learning: A report on three machine learning contests,” _20th International Conference, ICONIP 2013, Daegu, Korea_ , vol. x, pp. 117–124, 2013.
  * [94] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , vol. x, pp. 5562–5570, 2016.
  * [95] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “From facial expression recognition to interpersonal relation prediction,” _International Journal of Computer Vision_ , vol. 126, pp. 550–569, 2018.
  * [96] A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” _IEEE Transactions on Affective Computing_ , vol. 10, pp. 18–31, 2017.
  * [97] S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , vol. x, pp. 2852–2861, 2017.
  * [98] X. Jiang, Y. Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,” _Proceedings of the 28th ACM international conference on multimedia_ , vol. x, pp. 2881–2889, 2020.
  * [99] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato, “A 3d facial expression database for facial behavior research,” _7th international conference on automatic face and gesture recognition (FGR06)_ , vol. x, pp. 211–216, 2006\.
  * [100] G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. PietikäInen, “Facial expression recognition from near-infrared videos,” _Image and vision computing_ , vol. 29, pp. 607–619, 2011.
  * [101] A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” _2011 IEEE international conference on computer vision workshops (ICCV workshops)_ , vol. x, pp. 2106–2112, 2011.
  * [102] W.-J. Yan, X. Li, S.-J. Wang, G. Zhao, Y.-J. Liu, Y.-H. Chen, and X. Fu, “Casme ii: An improved spontaneous micro-expression database and the baseline evaluation,” _PloS one_ , vol. 9, p. 86041, 2014.
  * [103] X. Li, T. Pfister, X. Huang, G. Zhao, and M. Pietikäinen, “A spontaneous micro-expression database: Inducement, collection and baseline,” _2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (fg)_ , vol. x, pp. 1–6, 2013.
  * [104] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” _2010 IEEE computer society conference on computer vision and pattern recognition-workshops_ , vol. x, pp. 94–101, 2010\.
  * [105] S. Cheng, I. Kotsia, M. Pantic, and S. Zafeiriou, “4dfab: A large scale 4d database for facial expression analysis and biometric applications,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , vol. x, pp. 5117–5126, 2018.
  * [106] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial expressions with gabor wavelets,” _Proceedings Third IEEE international conference on automatic face and gesture recognition_ , vol. x, pp. 200–205, 1998.
  * [107] A. Pak and P. Paroubek, “Text representation using dependency tree subgraphs for sentiment analysis,” _16th International Conference, DASFAA 2011, International Workshops: GDB, SIM3, FlashDB, SNSMW, DaMEN, DQIS, Hong Kong, China_ , vol. x, pp. 323–332, 2011.
  * [108] L. Kerkeni, Y. Serrestou, K. Raoof, M. Mbarki, M. A. Mahjoub, and C. Cleder, “Automatic speech emotion recognition using an optimal combination of features based on emd-tkeo,” _Speech Communication_ , vol. 114, pp. 22–35, 2019.
  * [109] R. Santhoshkumar and M. K. Geetha, “Human emotion recognition using body expressive feature,” _Microservices in Big Data Analytics_ , vol. x, pp. 141–149, 2020.
  * [110] S. Z. Bong, M. Murugappan, and S. Yaacob, “Analysis of electrocardiogram (ecg) signals for human emotional stress classification,” _Trends in Intelligent Robotics, Automation, and Manufacturing: First International Conference, IRAM 2012, Kuala Lumpur, Malaysia_ , vol. x, pp. 198–205, 2012.
  * [111] C.-H. Wu and W.-B. Liang, “Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels,” _IEEE Transactions on Affective Computing_ , vol. 2, pp. 10–21, 2010.
  * [112] H. H. Do, P. Prasad, A. Maag, and A. Alsadoon, “Deep learning for aspect-based sentiment analysis: a comparative review,” _Expert systems with applications_ , vol. 118, pp. 272–299, 2019.
  * [113] Y. Fu, X. Wu, X. Li, Z. Pan, and D. Luo, “Emotion recognition based on multi-view body gestures,” _2019 IEEE international conference on image processing (icip)_ , vol. x, pp. 6535–6548, 2020.
  * [114] Z. Shen, J. Cheng, X. Hu, and Q. Dong, “Emotion recognition based on multi-view body gestures,” _2019 IEEE international conference on image processing (icip)_ , vol. x, pp. 3317–3321, 2019.