# HICEM: A High-Coverage Emotion Model for Artificial Emotional Intelligence

Benjamin Wortman and James Z. Wang B. Wortman and J. Z. Wang are with the
College of Information Sciences and Technology, The Pennsylvania State
University, University Park, PA 16802, USA. (e-mails: {bvw5145,
<EMAIL_ADDRESS>Manuscript received June 1, 2022.

###### Abstract

As social robots and other intelligent machines enter the home, artificial
emotional intelligence (AEI) is taking center stage to address users’ desire
for deeper, more meaningful human-machine interaction. To accomplish such
efficacious interaction, the next-generation AEI need comprehensive human
emotion models for training. Unlike theory of emotion, which has been the
historical focus in psychology, emotion models are a descriptive tools. In
practice, the strongest models need robust coverage, which means defining the
smallest core set of emotions from which all others can be derived. To achieve
the desired coverage, we turn to word embeddings from natural language
processing. Using unsupervised clustering techniques, our experiments show
that with as few as 15 discrete emotion categories, we can provide maximum
coverage across six major languages–Arabic, Chinese, English, French, Spanish,
and Russian. In support of our findings, we also examine annotations from two
large-scale emotion recognition datasets to assess the validity of existing
emotion models compared to human perception at scale. Because robust,
comprehensive emotion models are foundational for developing real-world
affective computing applications, this work has broad implications in social
robotics, human-machine interaction, mental healthcare, and computational
psychology.

###### Index Terms:

Modeling human emotion, basic emotions, emotion theory, statistical
clustering, natural language.

##  1 Introduction

As far back as Darwin, researchers have studied the subjectivity and
universality of human emotions [1]. While such research was primarily limited
to academic discussions in university psychology departments, with the rise of
in-home social robots and other intelligent machines (e.g., Alexa, Astro), has
expanded this subject matter into the field of affective computing where
developing an accurate model of human emotion is a stepping stone toward
artificial emotional intelligence (AEI) [2]. Here, emotion modeling is a
descriptive tool used to ensure that systems being developed have sufficient
coverage for a wide range of human-machine or human-robot interactions.

Ideally, a robust model with sufficient coverage means identifying the
smallest core set of independent human emotions from which all other emotions
can be derived. If the emotion model used in an AEI program has too many
components, the AEI may struggle to distinguish among these components. On the
other hand, if the model used is too simple, the AEI may not be able to
understand human emotion to a level necessary for the intended application.
Some researchers used the VAD dimensional model to circumvent this problem
[3], but such an approach is not suitable for many AEI applications.

Ekman

Plutchik

Cowen

GoEmotions

EMOTIC

HICEM-15

Figure 1: Existing emotion models provide incomplete representations across
the entire emotion space. Here we visualize the maximum log cosine similarity
(with a ceiling of 0.5) between the word vectors of 1,720 emotion concepts and
the contents of the model. A higher cosine similarity (yellow) means the
contents of the model have a similar semantic meaning to the given concept.
Let $k$ denote the number of labels or components in an emotion model. We can
see Ekman’s basic emotions [4] ($k$=7) are insufficient to describe the space.
Plutchik’s wheel of emotion [5] ($k$=32) is better, but its labels contain a
significant amount of overlap. Using unsupervised techniques, our proposed
HICEM with 15 components, named HICEM-15, minimizes the overlap between labels
and provides almost the exact same coverage as Plutchik’s with half as many
components. For comparison we also include Cowen’s emotions identified in
video [6] ($k$=27), the annotation categories for the GoEmotions Dataset [7]
($k$=28), and the EMOTIC [8] dataset annotation scheme ($k$=27).

Developing an ideal emotion model for AEI is a complex problem. Existing
models either do not provide enough coverage [4] or include excessive,
overlapping labels to describe the space [5, 6, 7]. We provide a visual
representation of an emotion model’s coverage so the power of different models
can be compared (Fig. 1). These are generated by taking the FastText word
vectors [9] for 1,720 emotion concepts and projecting them down to two
dimensions using Uniform Manifold Approximation and Projection (UMAP) [10].
This dimensionality reduction technique is similar to t-SNE [11], but UMAP has
also been shown to help preserve global relationships. We then generate the
heatmaps using the maximum log cosine similarity between the FastText word
vectors for the model and our emotion-concepts list.111Since the distribution
of the maximum cosine similarity across the entire list is exponential, the
log of this distribution is taken to assist in visualization. This step allows
us to visualize the coverage of each emotion model in relation to our emotion
concepts list. More details will be provided later.

To overcome the limitations in existing models and understand the full range
of human emotion, we turn to natural language processing (NLP). Language has
evolved to be the principal means of human communication and has been shown to
influence our perception of the world [12, 13]. By examining emotion-related
words across cultures, we can identify trends and develop more robust,
universal emotion models for next-generation affective computing applications.
Previous work has taken similar approaches by having groups manually annotate
existing emotions across continuous affective dimensions such as the valence,
arousal, and dominance (VAD) space [14]. However, word embeddings popular in
NLP are now able to encode this information automatically. Understanding this
advancement, we leverage statistical techniques to identify the minimum number
of components with maximum coverage across multiple cultures. We propose a new
emotion model, named the HIgh-Coverage Emotion Model (HICEM), to provide
higher coverage with fewer components compared with existing emotion models
popular in psychology used for affective computing. Using two separate
evaluation metrics, we show HICEM is able to achieve this goal. In support of
this assertion, we also analyze the results from recent large-scale emotion
recognition datasets to assess the validity and coverage of existing discrete
and continuous emotion models.

The main contributions of our work include:

  * •

We provide to the affective computing and AEI communities a new high-coverage
emotion model, named HICEM, which contains more information with fewer labels
than existing emotion models.

  * •

We developed a principled framework to quantify the quality of existing
emotion models across different languages using two new metrics and a list of
1,720 emotion concepts. We evaluated existing models with six major languages
recognized by the UN.

  * •

We developed a new data-driven approach, leveraging the annotations of
existing large-scale emotion recognition datasets, to assess the validity of
existing emotion models in relation to human perception at scale.

The rest of the paper is organized as follows. We cover related work in
emotion modeling in Section 2. Section 3 describes our methods for generating
a high-coverage cross-cultural model of emotion, the HICEM. The methodology
and analysis of existing large-scale emotion recognition datasets are shown in
Section 4. We discuss our results and identify future areas of interest in
Section 5 and conclude in Section 6.

##  2 Related Work

There are three competing schools of thought on emotion: basic emotions,
continuous models, and componential models. In this section, we briefly
discuss these approaches in relation to affective computing and AEI.

###  2.1 Basic Emotion Theory

Basic emotion theory suggests that humans evolved a set of discrete,
independent emotions which when triggered produce a physiological response or
action tendency. From these basic emotions, all other human emotions can be
derived. As shown in Table I, these basic emotions are often used as
categorical labels in affective computing datasets. More specifically, Paul
Ekman’s research into basic universal emotions serves as the foundation for
most annotation schemes currently used [15, 8, 16, 17, 18, 19]. His original
research identified six emotions universally recognizable by their facial
expression [4]. They are fear, anger, joy, sadness, disgust, and surprise.
However, several studies [20, 21, 22] suggest facial expressions alone are
insufficient to differentiate emotions. Since it has been demonstrated that
body language cues are also universal across cultures [23], there may exist a
subset of emotions that are universal for body language while being
indistinguishable in facial expressions alone [24]. Although not shown to be
cross-cultural, analysis by Cowen et al. on perceived emotions from
vocalization [25], facial expressions [26], and perceived emotion from video
[6] suggests not six but more than 24 emotion categories are required to
adequately map the space. However, this list was limited in that the label
space was predetermined by the researchers. In attempting to develop an
emotion model for text classification, Demszky et al. expanded upon Cowen et
al.’s work by using user-submitted labels to augment their emotion model.
These labels were then pruned and refined to generate a more annotator-
friendly list of 27 emotions and a neutral category [7].

TABLE I: Recent Emotion Recognition Datasets

| Dataset
---
| Labeled
---
Samples
| Categorical
---
Emotions
| Continuous
---
Emotions
Year
BoLD [15] | 20k | 26† | VAD | 2020
DFEW [16] | 16k | 7‡ | - | 2020
GoEmotions [7] | 58k | 28‡ | - | 2020
MOSEI [17] | 23k | 6† | Sentiment | 2018
OMG-Emotion [18] | 0.6k | 7‡ | VA | 2018
Aff-Wild [27] | 0.3k | - | VA | 2018
EMOTIC [8] | 34k | 26† | VAD | 2017
EmoReact [19] | 1.1k | 16‡ | V | 2016
|  † Contains a subset of Ekman’s basic emotions ‡ Ekman’s basic + neutral
---
Continuous Emotion Key: (V)alence, (A)rousal, (D)ominance

Although Ekman’s basic emotions are the most commonly used in affective
computing, other models do exist. In taking an evolutionary inspired approach,
Plutchik proposes an alternative to Ekman’s model which consists of eight
primary affective states arranged to form a wheel of emotion [5]. Each of
these affective states has varying degrees of intensity and when combined form
more complex human emotions. Although a useful tool, this model is criticized
as being too simplistic and hasn’t been shown to have a strong empirical
foundation [28]. Compared with Plutchik’s palette theory, Jaak Panksepp took a
biological approach to understanding emotion. His work pioneered the field of
affective neuroscience which works to map specific regions of the brain to
emotional experience [29, 30, 31]. In his original work, he describes seven
affective systems common across mammalian brains which control specific types
of behaviours and generate distinct emotional states [32]. He describes these
structures as the “core-SELF.”222These include SEEKING (expectancy), FEAR
(anxiety), RAGE (anger), LUST (sexual excitement), CARE (nurturing),
PANIC/GRIEF (sadness), and PLAY (social joy). Despite its neurological
underpinnings, Panksepp’s model hasn’t been widely used in the affective
computing community.

###  2.2 Continuous Models

Recognizing the limits of discrete labels for human emotions, some researchers
have worked to define continuous dimensions to measure a person’s affective
state. As shown in Table I, annotations along continuous dimensions are often
used together with basic emotions. In the simplest case, such annotations
simply means labeling a sample based on how positive or negative it is. This
dimension is usually described as the sentiment, pleasure, or valence of the
sample. Expanding beyond one dimension, the Circumplex of Affect by Russel
considers arousal (relaxed vs. aroused) and valence (pleasant vs. unpleasant)
as the two fundamental dimensions which together provide a mapping for the
discrete emotions [33]. There is strong support for the two-dimensional
approach of the Affective Circumplex. These two dimensions appear across a
wide range of studies [34, 35, 36]. Similar to Panksepp’s mapping of discrete
emotions, there has also been a considerable amount of work mapping valence
and arousal to processes in the human brain [37, 38, 39, 40].

For three dimensions, another popular model comes from the researchers
Mehrabian et al. who described the emotion space across pleasure-displeasure,
arousal-nonarousal, and dominance-submissiveness (PAD333Valence and pleasure
are often used interchangeably and sometimes referred to as VAD for valence,
arousal, and dominance.) [14]. This mirrors earlier work by Osgood et al., who
considered the closely related concept of control instead of dominance [41].
Here, control can be thought of in terms of both the feelings of power or
weakness in addition to interpersonal dominance or submission. With regards to
the PAD model, other proposed dimensions include anticipation-expectation,
anxiety-confidence, boredom-fascination, frustration-euphoria, terror-
enchantment, and intensity (how far the person is from a state of pure, cool
rationality) [42, 43, 44].

###  2.3 Componential Models

In contrast with discrete basic emotions and the previously described
continuous models, some work has led to componential models derived from the
appraisal theory of emotion [45, 43, 46, 47]. Under this framework, emotion is
a dynamic process thought to result from a person’s repeated evaluation
(appraisal) of their circumstances [48, 49]. This perspective has an advantage
over both descriptive models leveraging basic emotions and continuous
dimensions because it provides an explanation for why an emotion presents
itself. Since these componential models rely heavily on subjective experience
[50, 43], outside of lab constrained experiments [51, 52], they have not been
widely adopted for use in affective computing.

Figure 2: Our pipeline for generating a cross cultural, high-coverage model of
emotion. Starting with our list of emotion-concepts, we generate their
FastText word vectors. Then we perform dimensionality reduction and clustering
to generate summary words. This process was repeated to create summary words
for six major languages. Finally, these were once again clustered to produce
our final model, the HICEM.

##  3 Cross Cultural Word Embeddings

NLP provides an interesting avenue for research into emotion modeling. By
examining how current annotation schemes relate to other emotion-related
words, we can take a quantitative approach toward identifying gaps in existing
models. In this section, we outline our methods for generating the HIgh
Coverage Emotion Model, or HICEM, from NLP word embeddings.

###  3.1 Generating a List of Emotion-Concepts

As shown in Fig. 2, we first compiled a list of emotion-related concepts from
various models [4, 5, 1], online sources [53, 54, 55], and the Semantic Atlas
of Emotional Concepts [56]. This list is then passed through a pre-trained
Word2Vec model [57], which encodes the semantic meaning of a word into a
300-dimensional vector and allows us to perform operations across these
embeddings to identify relationships between words. A popular example of this
process is

$\overrightarrow{\text{king}}-\overrightarrow{\text{man}}+\overrightarrow{\text{woman}}\approx\overrightarrow{\text{queen}}\;.$

That is, if we take the vector for “king,” subtract the vector for “man,” and
then add the vector for “woman,” the resulting output vector is approximately
equal to “queen.” For each pairwise combination of words in this list, we
append synonyms based on the cosine similarity of the average of their
Word2Vec vectors. For example, given “happy” and “sad” we would be able to
identify “bittersweet” because its vector is approximately the average of the
two. That is,

$\text{average}(\overrightarrow{\text{happy}},\overrightarrow{\text{sad}})\approx\overrightarrow{\text{bittersweet}}\;.$

This expanded set was manually pruned to remove adverbs and words unrelated to
a person’s emotional state. For example, words such as “terrorist” and
“terrorism” are closely related to the pairwise combination “terror” and
“anger,” but these have little to do with emotion so they are removed from the
final expanded list. Likewise, there was a small subset of words typically
used in religious contexts (‘glee’, ‘woe’, ‘joy’, ‘awe’) which had an
extremely high cosine similarity between their word vectors. This had an
adverse effect on clustering so they were removed. Although, alternate forms
used in different contexts remained (‘gleeful’, ‘woeful’, ‘joyful’, ‘awed’).
Because the objective of this study was to identify gaps in existing emotion
models for affective computing tasks, we took an inclusive approach and kept
words that may not qualify classically as emotions but are still concepts and
dispositions that influence a person’s expressions [31, 58]. In total the
final list contained 1,720 emotion-related keywords.444The full emotion-
concept word list is available upon request.

###  3.2 UMAP Reduction

Using the list of 1,720 emotion concepts, a pre-trained FastText model [59] is
used to encode the semantic meaning of the word. FastText is a variant of
Word2Vec that operates at the n-gram level which allows for the use of subword
information to improve the quality of the embeddings [60]. Trained on Common
Crawl and Wikipedia, this model provides a 300-dimensional embedding for each
word. FastText was chosen over more advanced techniques such as BERT [61] due
to its location invariance and its use of the same methodology to generate
vectors for multiple languages [9]. BERT embeddings vary based on the location
of the word within the text. In our testing, BERT produced poor results when
feeding individual words to the model.

PCA

SVD

t-SNE

UMAP

Figure 3: We compare four different dimensionality reduction methods using
sentiment as a heuristic. Unlike t-SNE [11], UMAP [10] is able to separate and
organize positive and negative emotion concepts. Compared with other methods,
when we plot Ekman’s basic emotions [4] in the UMAP embedding space the
separation between “Happiness” and the negative basic emotions is increased.
This result is also shown in Table II where we compare the cosine similarity
between the raw FastText [9] word vectors and UMAP embeddings across synonyms
and antonyms for “Happiness.”

Since word embeddings are generated based on their local context, antonym word
pairs (e.g., Happiness/Sadness) which are commonly used in the same context
may have high cosine similarities relative to their perceived similarity. To
overcome this, we use Uniform Manifold Approximation and Projection (UMAP)
[10]. This dimensionality reduction technique is similar to t-SNE [11] but has
the added benefit of maintaining the global relationships among samples.
Because UMAP relies on a number of neighbors to generate the embedding, as
long as a word has more true synonyms than similar-context antonyms, it will
close the distance between the synonyms while increasing the distance between
antonyms. This outcome is demonstrated in Table II, where UMAP increases the
similarity between synonyms while pushing antonyms away. In the raw vectors,
“Sadness” has the highest cosine similarity with “Happiness” despite the
semantic meaning being quite different. Other dimensionality reduction
techniques like principal component analysis (PCA), singular value
decomposition (SVD), and t-distributed stochastic neighbor embedding (t-SNE)
fail to produce accurate cosine similarities between synonyms and antonyms
after reduction.

In Fig. 3, we visualize the different embeddings with respect to the sentiment
for each word using the Python Natural Language Toolkit (NLTK) [62]. Although
we don’t use any sentiment information when generating the embeddings, all
dimensionality reduction techniques are able to create a separation between
positive and negative emotions. However, t-SNE is only able to achieve this
locally and fails to provide any consistent global separation. Similar to the
result in Table II, when we plot Ekman’s basic emotions, we see much clearer
separation between “Happiness” and Ekman’s negative basic emotions in the UMAP
embeddings than in other techniques.

TABLE II: UMAP’s Effect on Similarity

|  | Cosine Similarity
---|---|---
Emotion Pairs | Synonyms |  | Raw Word
---
Vector
| PCA
---
$d=2$
| SVD
---
$d=2$
| t-SNE
---
$d=2$
| UMAP
---
$d=2$
Happiness, Sadness |  | 0.43 | 0.38 | 0.65 | 0.88 | -0.97
Happiness, Anger |  | 0.16 | 0.02 | 0.39 | 0.75 | -0.92
Happiness, Fear |  | 0.13 | 0.05 | 0.39 | 0.82 | -0.80
Happiness, Contempt |  | 0.06 | 0.61 | 0.70 | 0.82 | -0.78
Happiness, Disgust |  | 0.10 | 0.12 | 0.56 | 0.89 | -0.98
Happiness, Surprise |  | 0.18 | 0.98 | 0.93 | 0.89 | -0.55
Happiness, Joyous | ✓ | 0.23 | -0.37 | -0.87 | 0.37 | 0.45
Happiness, Gladness | ✓ | 0.39 | 0.99 | 0.89 | 0.63 | 0.99
Happiness, Bliss | ✓ | 0.42 | 0.82 | 0.54 | 0.99 | 0.99

$d$: number of dimensions

###  3.3 Hierarchical Clustering

Figure 4: Using the elbow method to identify the number of clusters for each
culture for the cross-cultural model of emotion. $k$ is the number of
clusters.

Agglomerative clustering [63, 64] was used to programmatically select emotion
words with the highest coverage. We chose agglomerative clustering over other
methods because it mirrors the hierarchical nature of basic emotions [65, 32].
To do this, we first determine the optimum number of clusters using the elbow
method [66]. This automatically finds the “elbow” or “knee” which corresponds
to point of maximum curvature on the plot of cluster distortion by clusters
count [67] as shown in Fig. 4. Finally, we summarize the contents of each
group by finding the word embedding closest to the centroid of each cluster.
The results of this step are shown in Fig. 5.

To develop our cross-cultural model, we repeat this process by translating our
list of emotion-related concepts from English into the other official
languages recognized by the United Nations (Arabic, Chinese, French, Spanish,
and Russian) using Google’s Translation
API.555https://cloud.google.com/translate We then once again proceeded with
Facebook’s FastText models which have been trained on Common Crawl and
Wikipedia for each of these languages [9].

Figure 5: Cluster summaries for English for the number of clusters, $k$, increases from seven to 30. As $k$ increases, we are able to generate more complete models of human emotion. Since UMAP maintains global relationships among samples, we can visualize the relationships among different emotions.  TABLE III: Recommended Number of Clusters from the Elbow Method Language |  | # Clusters
---
(Elbow)
English | 14
Arabic | 14
Mandarin Chinese | 15
French | 11
Spanish | 13
Russia | 14
Cross-Cultural | 15

The translated UMAP embeddings for both Chinese and Russian needed adjustments
due to limitations in translation. For Chinese, the embeddings originally
formed two distinct clusters. This separation is caused by the inclusion of
the nominalization particle “的,” which converts nouns or noun phrases into
adjectives (e.g., “快乐” or happiness $\rightarrow$ “快乐的” or happy). An
equivalent example in English would be the use of the suffix “-ness” such as
in “happiness.” This suffix converts the adjective “happy” into a noun. Since
FastText uses subword information to generate its word vectors, the semantic
meaning of this character influences the final word embedding. When UMAP is
run on these embeddings, the inclusion of this character creates enough
separation between the word vectors that all words containing this character
get clustered separately from the rest of the emotion-concepts list. To
resolve this divide, this cluster was excluded from the analysis since the
other was significantly larger and already contained a superset of Ekman’s
basic emotions. In Russian, there was a similar separation caused by a noun-
adjective divide. Here, the noun cluster was excluded because it was roughly a
quarter the size of the adjective cluster.

With the translated word vectors, we once again apply agglomerative clustering
to each language. Since this process combines the two closest centroids in
each iteration, this clustering can handle situations where one-to-one
translations do not exist. As shown in Table III, we start experiencing
diminishing returns in all languages around 15 clusters. This result provides
quantitative evidence that all other emotion-related words can be embedded
into the semantic space of roughly 15 dimensions.

TABLE IV: HICEM Components in Six Major Languages

An interesting finding from this analysis is that these clusters, broadly
capture the basic emotions as described by Ekman (i.e., fear, anger,
happiness/joy, sadness, disgust, and surprise) in each language. Likewise,
there seem to be themes of humor, abnormal behavior, friendliness, anxiety,
and confusion. To ensure our model has coverage across cultures, we then took
the top 50 cluster summary words from each language and once again performed
agglomerative clustering ($k$=15) to generate a list of cross-cultural summary
words. As a final step, we then manually adjusted our emotion model by
replacing rare or outdated language with more common English terms. This
process provides us with the final list of 15 high coverage emotion concepts
in Table IV. We repeat this final agglomerative clustering step for 25
clusters (named HICEM-25) to provide an additional comparison with similar-
sized emotion models.

###  3.4 Assessing Performance

TABLE V: Coverage across Languages and Emotion Models

|  | Average Coverage
---|---|---
Emotion Model | # Components | English | Arabic | Mandarin | French | Spanish | Russian | Total
Ekman | 7 | 0.324 | 0.273 | 0.286 | 0.355 | 0.333 | 0.327 | 0.314
EMOTIC | 27* | 0.400 | 0.374 | 0.332 | 0.409 | 0.397 | 0.437 | 0.390
Cowen | 27 | 0.432 | 0.352 | 0.332 | 0.443 | 0.447 | 0.505 | 0.415
GoEmotions | 28† | 0.430 | 0.362 | 0.332 | 0.449 | 0.448 | 0.479 | 0.414
Plutchik | 32 | 0.461 | 0.365 | 0.341 | 0.457 | 0.475 | 0.503 | 0.428
HICEM-15 | 15 | 0.458 | 0.349 | 0.315 | 0.455 | 0.458 | 0.501 | 0.416
HICEM-25 | 25 | 0.503 | 0.378 | 0.337 | 0.484 | 0.490 | 0.528 | 0.444

*The combined category Doubt/Confusion was split into two labels †27 Emotion labels + Neutral

We can qualitatively assess the quality of this list, by projecting it into
the English embeddings as shown in Fig. 1. Because this list is uniformly
distributed throughout the space, we can assume there is minimal overlap in
its labels. In addition to this, we also define a new metric “coverage” to
provide a quantitative assessment of the quality of each model. This metric is
based on cosine similarity between the words in the model and our emotion list
as follows

$\text{Avg. Coverage}(M)=\frac{1}{n}\sum_{i=1}^{n}\max_{m\epsilon
M}\text{CosSim}(m,w_{i})\;,$

and

$\text{CosSim}(a,b)=\frac{a\cdot b}{\lVert a\rVert\lVert b\rVert}\;.$

Figure 6: In this histogram of coverage by cosine similarity across English
word vectors, we show that our HICEM-15 model is able to provide higher
coverage with fewer labels compared to other models of emotion popular in
psychology [4, 42, 5, 8, 6, 7]. In English, our model achieves an average
similarity of 0.45. This result is equivalent to the similarity between
“happiness” and “calmness,” “merriment,” or “euphoria” ($0.45\pm 0.01$).

Here $M$ is the set of FastText word vectors in an emotion model, and $W-M$ is
the set of vectors for the words from the set of 1,720 emotion concepts (the
set $W$) excluding the components of the emotion model (the set $M$). We
define $n$ as the cardinality of set $W-M$. We chose to proceed with max
cosine similarity since this gives an indication of how strong the top synonym
is for a given emotion model and emotion-concept pair. As seen in Table V and
Fig. 6, our model provides higher coverage with fewer components compared with
previous models.

In addition to analyzing the coverage, we also conducted an experiment to show
how much emotion information is captured in each model. The intuition behind
this experiment is that ideally, we should be able to recover a large number
of emotional states for a sample using only the annotations from a given
model. In contrast with coverage where we only consider the max cosine
similarity, here we leverage the additional annotations across all the
components to make a prediction on the emotion present. To do this, we assume
we have an oracle who, when given an emotion model $M$, is able to generate an
embedding $X$ for a given word vector $w_{i}$ using the cosine similarities
between the FastText word vectors of each $m\in M$ and $w_{i}$. More formally,

$X=\text{CosSim}(w_{i},M)\;.$

Then,

$\hat{w_{i}}=G(X)\;,$

where $G$ is a function we pass $X$ through in order to make a prediction on
the original word vectors $\hat{w_{i}}$. Then once again using cosine
similarity, we compare the original word vector $w_{i}$ with our recovered
prediction $\hat{w_{i}}$ as follows:

$\text{Avg. Recovered
Information}(M)=\frac{1}{n}\sum_{i=1}^{n}\text{CosSim}(w_{i},\hat{w_{i}})\;.$

In practice, if we are to annotate the emotional expression of a character in
a video clip using a categorical emotion model, this would be equivalent to a
human annotator for the clip judging how similar or dissimilar the sample
(i.e., the character’s emotional expression) is to the labels of the given
emotion model and then using those annotations to recover the ground-truth
emotion present in the clip. There is often a level of disagreement among
annotators in real-world settings which adds noise to the oracle’s embedding
$X$. However, the assumption of a perfect oracle provides a theoretical upper
bound for the amount of recoverable information which we can use for
comparison.

Figure 7: In this plot of recovered cosine similarity vs. the number of
components using English word vectors, the only models which outperform a
uniform random sample of words (i.e., the dotted line) are HICEM-15 and
HICEM-25.

In our experiment, we use the Ridge regression [68] for $G$ and a 50/50 train-
test split on the list of 1,720 emotion concepts. We then train on the
similarity embeddings for each model to make a prediction on the original
embedding. To provide a baseline, we use random subsets of words from the
emotion-concept list of varying sizes. As shown in Fig. 7 and Table VI, HICEM
is the only emotion model that performs above a random subset of the same
size. There are several possible explanations for this. First, this model’s
success could be influenced by the presence of redundant or overlapping
labels. Second, because the random subsets were selected using a random
uniform distribution, they are more likely to sample across the entire emotion
concept space, in turn providing more unique information. Lastly, HICEM
benefits from being biased towards the emotion-concept list it is being tested
against. Although other models touch on concepts such as craving [6] and pain
[15, 8], these types of concepts have only recently been included in the
discussion of emotion [58, 69, 70]. Since these were included in the list of
1,720 emotion concepts, HICEM is better able to represent them since it is
derived from this same list.

Figure 8: Categorical emotions plotted against valence and dominance for the
BoLD dataset. Horizontal delineation suggests valence and vertical delineation
suggests dominance as the discriminant factor. In these scatterplots of
categorical emotions according to their valence and dominance ratings, there
appears to be a gap (i.e., blank space) for low-dominance, high-valence
emotions. Looking at the few samples present in this area; these seem to
represent guardian-child type relationships where one subject takes on the
role of the protector and the other takes on the role of the child. We suggest
the lack of samples in this area is due to some bias in the original data
collection and not necessarily due to a limitation of the continuous VAD
dimensions.

It is worth noting that in practice, there is a trade-off between the size of
the emotion model and the amount of information the annotators will give for
any sample. As emotion models become more complex and include more abstract
concepts, the agreement between annotators decreases substantially. This
reduction can be seen in the levels of inter-annotator agreement across
several large-scale datasets that reported this information. For example,
despite several quality-control measures implemented during data collection
and the post-processing done to filter unreliable annotators in the BoLD
dataset [15], the average Fleiss’ Kappa [71] across emotion categories is
$\kappa=0.173$ [15]. Intuitively less complex emotions like “Happiness” have
higher levels of agreement comparable to objective tasks performed at the time
of data collection like determining age or ethnicity. More abstract concepts,
such as “Yearning” and “Sensitivity,” had almost no agreement among
participants. This result mirrors those from the EMOTIC [8] and GoEmotions [7]
datasets. This comparison is important since as the size of the emotion model
increases the types of emotion concepts included are bound to become more
abstract. Not only do additional components have diminishing returns in terms
of the information they provide, but since emotion is subjective, they also
suffer an agreement penalty during the annotation process further reducing
their effectiveness. One method to help mitigate this effect would be to
choose more concrete emotion concepts as the foundation for emotion models.
This method wasn’t considered when generating the random models in Table VI.
Although they seem to outperform existing models in terms of recoverable
information, the abstractness of their labels would severely limit their real-
world effectiveness.

TABLE VI: Recoverable Information across Languages and Emotion Models

|  | Average Recoverable Information
---|---|---
Emotion Model | # Components | English | Arabic | Mandarin | French | Spanish | Russian | Total
Ekman | 7 | 0.370 | 0.263 | 0.336 | 0.328 | 0.325 | 0.339 | 0.327
EMOTIC | 27* | 0.585 | 0.470 | 0.502 | 0.515 | 0.534 | 0.528 | 0.522
Cowen | 27 | 0.577 | 0.414 | 0.497 | 0.512 | 0.545 | 0.529 | 0.512
GoEmotions | 28† | 0.592 | 0.473 | 0.504 | 0.540 | 0.544 | 0.552 | 0.534
Plutchik | 32 | 0.620 | 0.485 | 0.525 | 0.552 | 0.573 | 0.562 | 0.552
Random-7 | 7 | 0.372 | 0.312 | 0.346 | 0.332 | 0.340 | 0.386 | 0.348
Random-15 | 15 | 0.507 | 0.393 | 0.435 | 0.442 | 0.468 | 0.497 | 0.457
Random-25 | 25 | 0.597 | 0.468 | 0.477 | 0.522 | 0.544 | 0.550 | 0.526
Random-30 | 30 | 0.625 | 0.492 | 0.491 | 0.550 | 0.566 | 0.564 | 0.548
HICEM-15 | 15 | 0.520 | 0.412 | 0.426 | 0.464 | 0.473 | 0.494 | 0.464
HICEM-25 | 25 | 0.591 | 0.451 | 0.482 | 0.537 | 0.548 | 0.556 | 0.528

*The combined category Doubt/Confusion was split into two labels †27 Emotion labels + Neutral

##  4 Large-Scale Data Analysis

To gain additional insight into how people consciously perceive emotion, we
also examined the annotations for two large-scale emotion recognition
datasets: EMOTIC [8] and BoLD [15]. These are both in-the-wild datasets
annotated for the same 26 categorical emotions as well as valence, arousal,
and dominance [14]. Although the categorical emotions were not based on any
pre-existing emotion models, there is enough overlap with these emotion models
to assess their validity. In addition to this, since EMOTIC and BoLD are image
and video data, respectively, together they offer some insight into how humans
process these different modalities.

Projections of BoLD Categorical Labels Across Continuous Dimensions

Circumplex Projection

Raw Valence-Arousal Projection

Raw Dominance-Valence Projection

Figure 9: Left: When the BoLD labels are projected onto the unit circle for
valence and arousal, the locations of the emotions broadly line up with the
Affective Circumplex [33]. Center: The relationship between raw Valence-
Arousal dimensions is visualized. These are colored by Dominance. The circle
size is based on the number of samples for that label within the BoLD dataset.
The compression of emotions along the Arousal dimension is in line with
previous experiments [72]. Right: Similar to previous work [73], the
Dominance-Valence projection shows a strong linear relationship between these
two dimensions.

BoLD UMAP Embeddings in the Categorical Space

Figure 10: Heatmaps for each of the 26 categorical emotions in the BoLD
dataset based on their categorical UMAP embeddings. The blue areas represent
where that emotion exists within this embedding. Ekman’s basic emotions form
concrete clusters suggesting they are the building blocks for more complex
emotions such as Annoyance or Pleasure. Similarly, other higher-order emotions
such as Anticipation and Engagement appear to spread out throughout the
embeddings representing the wide range of scenarios in which they present
themselves. Although the axes are meaningless, by projecting the continuous
VAD dimension in this space (last three images) we can roughly correlate
Valence and Dominance to the X-axis and Arousal to the Y-axis. Figure 11:
Categorical emotions plotted against dominance and arousal for the BoLD
dataset. Horizontal delineation suggests dominance and vertical delineation
suggests arousal as the discriminant factor. Figure 12: Categorical emotions
plotted against valence and arousal for the BoLD dataset. Horizontal
delineation suggests valence and vertical delineation suggests arousal as the
discriminant factor. Figure 13: Categorical emotions plotted against dominance
and arousal for the EMOTIC dataset. Horizontal delineation suggests dominance
and vertical delineation suggests arousal as the discriminant factor. Figure
14: Categorical emotions plotted against valence and dominance for the EMOTIC
dataset. Horizontal delineation suggests valence and vertical delineation
suggests dominance as the discriminant factor. Figure 15: Categorical emotions
plotted against valence and arousal for the EMOTIC dataset. Horizontal
delineation suggests valence and vertical delineation suggests dominance as
the discriminant factor.

To visualize how these emotions relate to each other, they were projected into
the VAD space as shown in Fig. 9. In general, the categorical emotions share
similar distributions for both datasets. However, looking at the plots for
valence vs. dominance, there is a distinct cluster in the BoLD dataset for
high-dominance, low-valence emotions such as “Anger,” “Disapproval,”
“Aversion,” and “Annoyance.” Because these are all high arousal emotions, this
difference between datasets may be caused by the additional motion information
video samples have over static photos. Other than this cluster, there appears
to be a strong correlation between dominance and valence. Past work has shown
that these dimensions are not completely orthogonal [73]. The collected
annotations from these two datasets seem to confirm this finding. Even with
the distinct clustering off the mainline dominance/valence plot, this can be
accounted for by their separation in the arousal dimension. The scatter plots
in Fig. 8 show that the only categorical emotion outside of the “Anger”
cluster that really benefits from the inclusion of dominance is confidence as
its heatmap appears uniformly distributed across the valence axis while
decreasing from high dominance to low dominance. Similar to the linear
relationship between valence and dominance, in the raw valence-arousal
projections in Fig. 9, we also observe a collapse along the arousal dimension.
This observation suggests changes in valence are the primary discriminative
factor for differentiating emotions. This “valence focus” mirrors results from
previous studies [72]. Still, when the labels from the BoLD dataset are
projected onto the unit circle based on the mean valence and arousal for each
emotion, the outcome broadly aligns with the affective circumplex [33, 74].

In addition to the simple VAD embeddings (Fig. 11-15), we also once again
produce UMAP embeddings [10] of the space as shown in Fig. 10. We performed
this analysis on both datasets. However, due to the large imbalance in the
EMOTIC dataset as well a lack of data cleaning similar to BoLD’s annotator
reliability analysis, we limit the following analysis strictly to BoLD. First,
it is important to note that although the dimensions are not meaningful, we
can assign some meaning to them by also looking at the continuous VAD
annotations in the same embedding space. In general, valence corresponds to
the horizontal axis with arousal on the vertical. Similar to their correlation
in the 2D VAD embeddings, dominance closely follows valence by generally
increasing from left to right. For each individual categorical emotion, we
then generate a heatmap so we can see where it exists in these embeddings. The
discreteness of these heatmaps provides an idea of how fundamental each of
these emotions is. For example, in Fig. 4 it is clear that “Happiness” and
“Anger” occupy their own distinct clusters away from the main grouping.
Likewise, Ekman’s other basic emotions (Sadness, Fear, Surprise), also have
relatively discrete clusters suggesting they are a fundamental building block
for more complex emotions. On the opposite end of the spectrum, “Anticipation”
and “Engagement” appear almost randomly dispersed throughout the embedding
space. This spread suggests they are more complex in their expression and
present themselves in a wider range of scenarios.

##  5 Discussion

Using word embeddings, we were able to show how existing models of human
emotion popular in psychology either lack sufficient coverage or contain
redundant, overlapping labels. Through agglomerative clustering, we were able
to identify 15 components that minimized overlap and maximized coverage across
cultures. This model, HICEM, was able to provide almost as much coverage with
half as many labels as compared with existing emotion models. Interestingly
enough, the only basic emotion described by Ekman which was not included in
the HICEM was “Disgust.” The combination of “Anger” and “Disgust” in our
analysis is similar to previous work on biologically basic emotions [75].

Another consideration in the construction of comprehensive emotion models is
the inclusion of general wellbeing/pain as well as a neutral affective state
in the model. Although these are not classically thought of as emotions, the
inclusion of these makes sense for HICEM because wellbeing/pain is expressed
through facial expressions and body language similar to other emotions, and
“Neutral” affective states also contain a variety of information useful for
filtering out functional movements (e.g., walking) common in everyday life. In
addition to this, recent work taking inspiration from componential models of
emotion has looked to relate other physiological systems to affective states.
Hunger, thirst, sleepiness, and stress have all been shown to be connected to
emotion [76, 58]. These factors can be added to the components found in HICEM
to provide complete coverage across both physiological processes and emotion.

Although quantitative analysis shows diminishing returns after around 15
clusters, qualitatively the completeness of the model seems to peak between 25
and 30 emotion concepts similar to previous results in earlier studies [6,
25]. Thus, it may be more beneficial to proceed with more than 15 base
components in future dataset creation to ensure completeness. Likewise, if
there are certain areas of the emotion space which are more important for a
given domain, the hierarchical nature of our model easily allows for
additional dimensions while maintaining maximum coverage. An example of this
might be the inclusion of abnormal emotional states for mental health
diagnoses. If we consider the clustering in Fig. 5 for 15 components, instead
of using “bizarre” as a label, we can look at the two clusters which formed it
and split the label into “zany” and “nonsensical.”

Finally, similar to previous work, it is recommended that HICEM is used in
tandem with continuous affective dimensions (i.e., VAD) to provide a holistic
representation of the emotion space. Given the relationship between valence
and arousal in existing large-scale datasets, it is recommended to drop
dominance as a dimension in future data collection to reduce redundancies and
costs associated with the data collection. Instead, alternate dimensions such
as certainty or effort may be included. An interesting extension of HICEM
would be to examine the latent continuous dimensions of emotion and develop a
similar high-coverage model to describe the space.

###  5.1 Limitations

As stated previously, one limitation with using word embeddings has to do with
antonyms being used in similar contexts, potentially having a higher cosine
similarity than their synonyms. Although UMAP helps in increasing the distance
between these antonyms in our analysis, this context issue still influences
cluster purity because there are situations where it is difficult to tease
out. By taking the median vector for each cluster, we minimize this effect
when generating our summary words. Another alternative would be to use word-
level emotion distributions as described by Li et al. [77] for text
classification tasks. In addition to this, the use of Deep Learning (DL)
generated embeddings [61] from free text annotations provides an interesting
alternative. These do not rely on local word context to generate their vectors
and have shown higher performance in NLP tasks compared to classical methods.

In addition to the embeddings, our method also is limited by translation. We
once again minimized the influence of this by averaging across several
cultures; however, better results for each individual language may be achieved
by having native speakers generate localized emotion word lists and then using
those lists to fine-tune the models. Similarly, although we can generate a
consistent set of labels for use across languages, the way people perceive
these different emotion words is unique to their culture. Given that all
languages appear to share the same 15 base components, transfer learning and
fine-tuning can be performed to better fit machine learning models to their
local language context.

###  5.2 Applications

HICEM is primarily a descriptive tool, so its value comes from its ability to
describe large numbers of affective states with relatively few labels. This is
ideal for dataset annotation in modern data-driven AI because it maximizes the
return on investment in terms of the amount of information gathered from each
sample labeled. Although limited to 15 components, HICEM still provides high
coverage over a wide range of human emotions. This advancement means next-
generation affective computing or AEI applications leveraging HICEM will allow
for more natural human-machine or human-robot interactions.

In addition to HICEM’s value as an annotation tool, the process for developing
our emotion model can also be used to build a taxonomy of human emotion
similar to WordNet [78]. Such a tool has potential applications in
psychological research and can also be used for succinctly describing a
patient’s emotional state. Because HICEM is limited to discrete emotions, an
interesting extension of this would be to use NLP word embeddings to find the
equivalent for the continuous emotion dimensions.

##  6 Conclusions

There has been a substantial amount of research into computational techniques
for recognizing human emotion. However, little work has been done examining
the actual emotion models underpinning this research. As our analysis shows,
existing models of emotion are insufficient for real-world applications. In
affective computing, coverage is much more important than the completeness of
the emotion model due to practical limitations in data collection and
annotation. Here, using unsupervised techniques we were able to identify 15
components with minimal overlap and maximum coverage across 1,720 emotion
concepts. This work provides a more efficient model of human emotion which is
a step toward achieving artificial emotional intelligence.

## Acknowledgments

This research was supported by generous gifts from the Amazon Research Awards
program. The project used computational resources from the Extreme Science and
Engineering Discovery Environment (XSEDE), which is supported by National
Science Foundation grant No. ACI-1548562. The authors are grateful to Yu Luo
for developing the BoLD dataset used in this study. The authors thank Tal
Shafir and Reginald B. Adams, Jr. for helpful discussions. The authors
appreciate the support and encouragement from Yelin Kim and Adam Fineberg.

## References

  * [1] C. Darwin, _The Expression of the Emotions in Man and Animals_. University of Chicago press, 2015.
  * [2] M. Krakovsky, “Artificial (emotional) intelligence,” _Comm. ACM_ , vol. 61, no. 4, pp. 18–19, Mar. 2018\. https://dl.acm.org/doi/10.1145/3185521
  * [3] X. Lu, P. Suryanarayan, R. B. Adams Jr, J. Li, M. G. Newman, and J. Z. Wang, “On shape and the computability of emotions,” in _Proc. 20th ACM Int. Conf. on Multimedia_ , 2012, pp. 229–238.
  * [4] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion,” _J. Personality and Soc. Psychol._ , vol. 17, no. 2, p. 124, 1971, publisher: American Psychological Association.
  * [5] R. Plutchik, _The Emotions_. Lanham, Md: University Press of America, 1991.
  * [6] A. S. Cowen and D. Keltner, “Self-report captures 27 distinct categories of emotion bridged by continuous gradients,” _Proc. Natl. Academy Sci._ , vol. 114, no. 38, pp. E7900–E7909, Sep. 2017. http://www.pnas.org/lookup/doi/10.1073/pnas.1702247114
  * [7] D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: A dataset of fine-grained emotions,” in _58th Annl. Mtg. Assc. Comput. Linguistics (ACL)_ , 2020, pp. 4040–4054.
  * [8] R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “EMOTIC: Emotions in context dataset,” in _2017 IEEE Conf.Comput. Vis. Pattern Recognit. Workshops (CVPRW)_. Honolulu, HI, USA: IEEE, Jul. 2017, pp. 2309–2317. http://ieeexplore.ieee.org/document/8015019/
  * [9] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov, “Learning word vectors for 157 languages,” in _Proc. of the Int. Conf. Lang. Resources Evaluation (LREC)_ , 2018.
  * [10] L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,” 2018, arXiv preprint arXiv:1802.03426. https://arxiv.org/abs/1802.03426
  * [11] G. E. Hinton and S. Roweis, “Stochastic neighbor embedding,” in _Adv. in Neural Info. Proc. Syst._ , S. Becker, S. Thrun, and K. Obermayer, Eds., vol. 15. MIT Press, 2003. https://proceedings.neurips.cc/paper/2002/file/6150ccc6069bea6b5716254057a194ef-Paper.pdf
  * [12] L. F. Barrett, K. A. Lindquist, and M. Gendron, “Language as context for the perception of emotion,” _Trends Cog. Sci._ , vol. 11, no. 8, pp. 327–332, Aug. 2007. https://linkinghub.elsevier.com/retrieve/pii/S1364661307001532
  * [13] K. A. Lindquist and M. Gendron, “What’s in a word? Language constructs emotion perception,” _Emotion Rev._ , vol. 5, no. 1, pp. 66–71, Jan. 2013\. http://journals.sagepub.com/doi/10.1177/1754073912451351
  * [14] A. Mehrabian, “Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament,” _Current Psychology_ , vol. 14, no. 4, pp. 261–292, Dec. 1996. http://link.springer.com/10.1007/BF02686918
  * [15] Y. Luo, J. Ye, R. B. Adams, J. Li, M. G. Newman, and J. Z. Wang, “ARBEE: Towards automated recognition of bodily expression of emotion in the wild,” _Int. J. Comput. Vis._ , vol. 128, no. 1, pp. 1–25, Jan. 2020. https://doi.org/10.1007/s11263-019-01215-y
  * [16] X. Jiang, Y. Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “DFEW: A large-scale database for recognizing dynamic facial expressions in the wild,” in _Proc. 28th ACM Int. Conf. on Multimedia_ , 2020, pp. 2881–2889.
  * [17] A. Bagher Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” in _Proc. 56th Annl. Mtg. Assc. Comput. Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 2236–2246. https://aclanthology.org/P18-1208
  * [18] P. Barros, N. Churamani, E. Lakomkin, H. Siqueira, A. Sutherland, and S. Wermter, “The omg-emotion behavior dataset,” in _2018 Int. Joint Conf. Neural Netw. (IJCNN)_. Rio de Janeiro: IEEE, Jul 2018, p. 1–7. https://ieeexplore.ieee.org/document/8489099/
  * [19] B. Nojavanasghari, T. Baltrušaitis, C. E. Hughes, and L.-P. Morency, “EmoReact: A multimodal approach and dataset for recognizing emotional responses in children,” in _Proc. 18th ACM Int. Conf. Multimodal Interaction_ , ser. ICMI. New York, NY, USA: Association for Computing Machinery, Oct. 2016, pp. 137–144. https://doi.org/10.1145/2993148.2993168
  * [20] P. Ekman, “Facial expression and emotion,” _American Psychologist_ , vol. 48, no. 4, pp. 384–392, 1993. http://doi.apa.org/getdoi.cfm?doi=10.1037/0003-066X.48.4.384
  * [21] L. F. Barrett, B. Mesquita, and M. Gendron, “Context in emotion perception,” _Curr. Dir. Psychol. Sci._ , vol. 20, no. 5, pp. 286–290, Oct. 2011. http://journals.sagepub.com/doi/10.1177/0963721411422522
  * [22] T. Le Mau, K. Hoemann, S. H. Lyons, J. M. B. Fugate, E. N. Brown, M. Gendron, and L. F. Barrett, “Professional actors demonstrate variability, not stereotypical expressions, when portraying emotional states in photographs,” _Nature Communications_ , vol. 12, no. 1, p. 5037, Aug. 2021. https://www.nature.com/articles/s41467-021-25352-6
  * [23] C. Parkinson, T. T. Walker, S. Memmi, and T. Wheatley, “Emotions are understood from biological motion across remote cultures,” _Emotion_ , vol. 17, no. 3, pp. 459–477, Apr. 2017.
  * [24] D. T. Cordaro, R. Sun, S. Kamble, N. Hodder, M. Monroy, A. Cowen, Y. Bai, and D. Keltner, “The recognition of 18 facial-bodily expressions across nine cultures,” _Emotion_ , vol. 20, no. 7, pp. 1292–1300, Oct. 2020. http://doi.apa.org/getdoi.cfm?doi=10.1037/emo0000576
  * [25] A. Cowen, D. Sauter, J. L. Tracy, and D. Keltner, “Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression,” _Psychol. Sci. Public Interest_ , vol. 20, no. 1, pp. 69–90, Jul. 2019. http://journals.sagepub.com/doi/10.1177/1529100619850176
  * [26] A. S. Cowen and D. Keltner, “What the face displays: Mapping 28 emotions conveyed by naturalistic expression,” _American Psychologist_ , vol. 75, no. 3, pp. 349–364, Apr. 2020. http://doi.apa.org/getdoi.cfm?doi=10.1037/amp0000488
  * [27] D. Kollias, P. Tzirakis, M. A. Nicolaou, A. Papaioannou, G. Zhao, B. W. Schuller, I. Kotsia, and S. Zafeiriou, “Deep affect prediction in-the-wild: Aff-Wild database and challenge, deep architectures, and beyond,” 2018, arXiv preprint arXiv:1804.10938. http://arxiv.org/abs/1804.10938
  * [28] H. Smith and A. Schneider, “Critiquing models of emotions,” _Soc. Methods Res._ , vol. 37, no. 4, pp. 560–589, May 2009. http://journals.sagepub.com/doi/10.1177/0049124109335790
  * [29] J. E. LeDoux, “Emotion circuits in the brain,” _Annl. Rev. Neuroscience_ , vol. 23, no. 1, pp. 155–184, Mar. 2000. https://www.annualreviews.org/doi/10.1146/annurev.neuro.23.1.155
  * [30] K. Phan, T. Wager, S. F. Taylor, and I. Liberzon, “Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI,” _NeuroImage_ , vol. 16, no. 2, pp. 331–348, Jun. 2002. https://linkinghub.elsevier.com/retrieve/pii/S1053811902910876
  * [31] J. Panksepp and L. Biven, _The Archaeology of Mind: Neuroevolutionary Origins of Human Emotions_ , 1st ed., ser. A Norton professional book. New York: W. W Norton, 2012.
  * [32] J. Panksepp, _Affective Neuroscience: The Foundations of Human and Animal Emotions_. Oxford university press, 2004.
  * [33] J. A. Russell, “A circumplex model of affect,” _J. of Personality and Soc. Psychol._ , vol. 39, no. 6, pp. 1161–1178, 1980. http://content.apa.org/journals/psp/39/6/1161
  * [34] R. P. Abelson and V. Sermat, “Multidimensional scaling of facial expressions,” _J. of Exp. Psychol._ , vol. 63, no. 6, pp. 546–554, 1962. http://content.apa.org/journals/xge/63/6/546
  * [35] J. A. Russell and M. Bullock, “Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults,” _J. of Personality and Soc. Psychol._ , vol. 48, no. 5, pp. 1290–1298, 1985. http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-3514.48.5.1290
  * [36] D. Watson, D. Wiese, J. Vaidya, and A. Tellegen, “The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence,” _J. of Personality and Soc. Psychol._ , pp. 820–838, 1999.
  * [37] H. A. Sackeim, “Hemispheric asymmetry in the expression of positive and negative emotions: neurologic evidence,” _Archives of Neurology_ , vol. 39, no. 4, p. 210, Apr. 1982. http://archneur.jamanetwork.com/article.aspx?doi=10.1001/archneur.1982.00510160016003
  * [38] A. Anderson, K. Christoff, I. Stappen, D. Panitz, D. G. Ghahremani, G. Glover, J. Gabrieli, and N. Sobel, “Dissociated neural representations of intensity and valence in human olfaction,” _Nature Neuroscience_ , vol. 6, no. 2, pp. 196–202, Feb. 2003. http://www.nature.com/articles/nn1001
  * [39] B. E. Jones, “Arousal systems,” _Front. Biosci._ , vol. 8, no. 6, pp. s438–451, 2003\. https://imrpress.com/journal/FBL/8/6/10.2741/1074
  * [40] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” _Dev. and Psychol._ , vol. 17, no. 03, Sep. 2005\. http://www.journals.cambridge.org/abstract_S0954579405050340
  * [41] C. E. Osgood, W. H. May, M. S. Miron, and M. S. Miron, _Cross-cultural Universals of Affective Meaning_. University of Illinois Press, 1975, vol. 1.
  * [42] B. Kort, R. Reilly, and R. Picard, “An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion,” in _Proc IEEE Int. Conf. Adv. Learning Tech._ , 2001, pp. 43–46.
  * [43] J. R. Fontaine, K. R. Scherer, E. B. Roesch, and P. C. Ellsworth, “The world of emotions is not two-dimensional,” _Psychol. Sci._ , vol. 18, no. 12, pp. 1050–1057, Dec. 2007. http://journals.sagepub.com/doi/10.1111/j.1467-9280.2007.02024.x
  * [44] G. McKeown, M. F. Valstar, R. Cowie, and M. Pantic, “The SEMAINE corpus of emotionally coloured character interactions,” in _2010 IEEE Int. Conf. Multimedia and Expo_ , 2010, pp. 1079–1084.
  * [45] K. R. Scherer, A. Schorr, and T. Johnstone, Eds., _Appraisal Processes in Emotion: Theory, Methods, Research_ , ser. Series in affective science. Oxford, New York: Oxford University Press, 2001.
  * [46] D. Grandjean, D. Sander, and K. R. Scherer, “Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization,” _Consciousness Cogn._ , vol. 17, no. 2, pp. 484–495, Jun. 2008. https://linkinghub.elsevier.com/retrieve/pii/S1053810008000457
  * [47] R. P. Marinier, J. E. Laird, and R. L. Lewis, “A computational unification of cognitive behavior and emotion,” _Cog. Syst. Res._ , vol. 10, no. 1, pp. 48–69, Mar. 2009. https://linkinghub.elsevier.com/retrieve/pii/S1389041708000302
  * [48] M. B. Arnold, _Emotion and Personality Psychological Aspects_. New York: Columbia Univ. Pr, 1960, vol. 1.
  * [49] Lazarus, R. S., _Psychological Stress and the Coping Process_. McGraw-Hill, 1966.
  * [50] D. Sander, D. Grandjean, and K. R. Scherer, “A systems approach to appraisal mechanisms in emotion,” _Neural Netw._ , vol. 18, no. 4, pp. 317–352, May 2005. https://linkinghub.elsevier.com/retrieve/pii/S0893608005000341
  * [51] G. Mohammadi and P. Vuilleumier, “A multi-componential approach to emotion recognition and the effect of personality,” _IEEE Trans. Affect. Comput._ , pp. 1–1, 2020.
  * [52] B. Meuleman and D. Rudrauf, “Induction and profiling of strong multi-componential emotions in virtual reality,” _IEEE Trans. Affect. Comput._ , vol. 12, no. 1, pp. 189–202, 2021.
  * [53] Tchiki Davis, “List of emotions: 271 emotion words,” [Online]. Available: https://www.berkeleywellbeing.com/list-of-emotions.html. [Accessed: 29-Jan-2022]. https://www.berkeleywellbeing.com/list-of-emotions.html
  * [54] “380 High Emotion Words,” Jun. 2014, [Online]. Available: http://www.emarketer.com/Article/Social-Networking-Reaches-Nearly-One-Four-Around-World/1009976. [Accessed: 29-Jan-2022]. https://thepersuasionrevolution.com/380-high-emotion-persuasive-words/
  * [55] “List of emotions: a huge list of useful words to describe feelings and emotions,” May 2019, [Online]. Available: https://eslforums.com/list-of-emotions/. [Accessed: 29-Jan-2022]. https://eslforums.com/list-of-emotions/
  * [56] J. R. Averill, _A Semantic Atlas of Emotional Concepts_. American Psycholog. Ass., Journal Suppl. Abstract Service, 1975.
  * [57] T. Mikolov, K. Chen, G. S. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” 2013, arXiv preprint arXiv:1301.3781. http://arxiv.org/abs/1301.3781
  * [58] C. L. Dancy, “A hybrid cognitive architecture with primal affect and physiology,” _IEEE Trans. Affect. Comput._ , vol. 12, no. 2, pp. 318–328, Apr. 2021.
  * [59] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” 2016, arXiv preprint a arXiv:1607.01759.
  * [60] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin, “Advances in pre-training distributed word representations,” in _Proc. Int. Conf. Lang. Rscs. Eval. (LREC)_ , 2018.
  * [61] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. 2019 Conf. NA Chapt. Assc. Comput. Ling.: Human Lang. Tech., Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. https://aclanthology.org/N19-1423
  * [62] S. Bird, E. Klein, and E. Loper, _Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit_. O’Reilly Media, Inc., 2009.
  * [63] O. Maimon and L. Rokach, Eds., _Data Mining and Knowledge Discovery Handbook_. New York: Springer, 2005.
  * [64] A. Dalmia and S. Sia, “Clustering with UMAP: Why and how connectivity matters,” 2021, arXiv preprint arXiv:2108.05525. https://arxiv.org/abs/2108.05525
  * [65] P. Ekman, “An argument for basic emotions,” _Cog. Emo._ , vol. 6, no. 3-4, pp. 169–200, 1992, publisher: Taylor & Francis.
  * [66] R. L. Thorndike, “Who belongs in the family?” _Psychometrika_ , vol. 18, no. 4, pp. 267–276, Dec. 1953. http://link.springer.com/10.1007/BF02289263
  * [67] V. Satopaa, J. Albrecht, D. Irwin, and B. Raghavan, “Finding a ”Kneedle” in a haystack: detecting knee points in system behavior,” in _2011 31st Int. Conf. Distr. Comput. Syst. Workshops_ , Jun. 2011, pp. 166–171.
  * [68] A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” _Technometrics_ , vol. 12, no. 1, pp. 55–67, 1970, publisher: Taylor & Francis.
  * [69] L. Al-Shawaf, “The evolutionary psychology of hunger,” _Appetite_ , vol. 105, pp. 591–595, Oct. 2016. https://linkinghub.elsevier.com/retrieve/pii/S0195666316302501
  * [70] G. Gilam, J. J. Gross, T. D. Wager, F. J. Keefe, and S. C. Mackey, “What is the relationship between pain and emotion? Bridging constructs and communities,” _Neuron_ , vol. 107, no. 1, pp. 17–21, Jul. 2020\. https://linkinghub.elsevier.com/retrieve/pii/S0896627320303937
  * [71] K. L. Gwet, _Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters_. Gaithersburg, Md: Advances Analytics, LLC, 2014.
  * [72] L. A. Feldman, “Valence focus and arousal focus: Individual differences in the structure of affective experience,” _J. of Personality and Soc. Psychol._ , vol. 69, no. 1, pp. 153–166, 1995. http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-3514.69.1.153
  * [73] A. B. Warriner, V. Kuperman, and M. Brysbaert, “Norms of valence, arousal, and dominance for 13,915 English lemmas,” _Behav. Res. Methods_ , vol. 45, no. 4, pp. 1191–1207, Dec. 2013. https://doi.org/10.3758/s13428-012-0314-x
  * [74] L. F. Barrett and J. A. Russell, “The structure of current affect,” _Curr. Dir. in Psychol. Sci._ , vol. 8, pp. 10 – 14, 1999.
  * [75] R. Jack, O. Garrod, and P. Schyns, “Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time,” _Curr. Bio._ , vol. 24, no. 2, pp. 187–192, Jan. 2014. https://linkinghub.elsevier.com/retrieve/pii/S0960982213015194
  * [76] S. Danziger, J. Levav, and L. Avnaim-Pesso, “Extraneous factors in judicial decisions,” _in Proc. Proc. Natl. Academy Sci._ , vol. 108, no. 17, pp. 6889–6892, Apr. 2011. https://pnas.org/doi/full/10.1073/pnas.1018033108
  * [77] Z. Li, H. Xie, G. Cheng, and Q. Li, “Word-level emotion distribution with two schemas for short text emotion classification,” _Knowledge-Based Syst._ , vol. 227, p. 107163, Sep. 2021. https://www.sciencedirect.com/science/article/pii/S0950705121004263
  * [78] G. A. Miller, “WordNet: A lexical database for English,” _Comm. ACM_ , vol. 38, no. 11, pp. 39–41, Nov. 1995. https://dl.acm.org/doi/10.1145/219717.219748

|  Benjamin Wortman received the bachelor’s degree in data sciences from The
Pennsylvania State University, University Park, in 2020. He is expected to
receive the MS degree in informatics from Penn State in 2022. He is also a PhD
candidate in the Informatics program of the College of Information Sciences
and Technology at Penn State. His research interests include affective
computing, computer vision, and machine learning.
---|---
|  James Z. Wang is a Distinguished Professor of the Data Sciences and
Artificial Intelligence section and the Human-Computer Interaction section of
the College of Information Sciences and Technology at The Pennsylvania State
University. He received the bachelor’s degree in mathematics summa cum laude
from the University of Minnesota (1994), and the MS degree in mathematics
(1997), the MS degree in computer science (1997), and the PhD degree in
medical information sciences (2000), all from Stanford University. His
research interests include affective computing, image analysis, image
modeling, image retrieval, and their applications. He was a visiting professor
at the Robotics Institute at Carnegie Mellon University (2007-2008), a lead
special section guest editor of the IEEE Transactions on Pattern Analysis and
Machine Intelligence (2008), and a program manager at the Office of the
Director of the National Science Foundation (2011-2012). He is on the
editorial board of the IEEE BITS – The Information Theory Magazine’s special
issue on Information Processing in Arts and Humanities (2022). He was a
recipient of a National Science Foundation Career award (2004) and Amazon
Research Awards (2018-2022).
---|---