Implicit Design Choices and Their Impact on Emotion Recognition Model
Development and EvaluationMimansa JaiswalDoctor of Philosophy Computer Science
and Engineering2023 Professor Emily Mower Provost, Chair
Professor Nikola Bancovic
Professor Benjamin Fish
Douwe Kiela, Contextual AI
Professor V.G. Vinod Vydiswaran

Mimansa Jaiswal

<EMAIL_ADDRESS>
ORCID iD: 0009-0001-8290-7743

Dedicated to my parents and my late granddad, B.P. Bhagat, and my late
grandmom, Smt. Radha Devi for their boundless love and support

ACKNOWLEDGEMENTS

I would like to start by expressing my profound gratitude towards my PhD
advisor, Emily (Emily Mower Provost). She has been my go-to person for any
research brainstorming and has shown me tremendous patience, support, and
guidance throughout my PhD journey. Without her persistence and suggestions,
completing this PhD would not have been possible. I am also incredibly
grateful for my thesis committee members: Vinod (VG Vinod Vydiswaran), Nikola
(Nikola Banovic), Benjamin (Benjamin Fish), and Douwe (Douwe Kiela). Their
valuable insights during my thesis proposal helped shape the final version of
the thesis.

For putting up with me during my PhD journey, my immense gratitude goes
towards my family, especially my parents, grandparents, and Bert. My parents
have been a rock for me over the past six years. Though I could not visit them
often or talk to them much, they were always there when I needed someone. They
seem to have aged fifteen years in the six years of my PhD, stressed about me,
but their support never wavered. My mom (Archana Kumari) received her own PhD
in 2021, and my dad (Abhay Kumar) became the Vice-Chancellor of a new IIIT—two
accomplishments that were their lifelong dreams and inspired me immensely. I
unfortunately lost two of my grandparents during the PhD program, and I will
never forget their blessings and excitement for me embarking on my higher
education journey. During early 2020, in the midst of COVID, I adopted a cat
named Bert—yes, named after the language model. Without him, I would not have
maintained my sanity during the dark, lonely nights and tiring, long work
days. His purring loudly into my ear calmed me down on the worst of nights.

I was lucky enough to secure three internships and have amazing research
mentors for all of them. Ahmad (Ahmad Beirami) taught me how to approach
Conversational AI, how to create effective presentations, and how to write
research proposals. Adina (Adina Williams) taught me how to work with
linguistics mixed in with NLP, and how subjectivity can infiltrate seemingly
objective parts (like NLI) of NLP. Ana (Ana Marasović) was the first person I
worked with on really large language models (foundational models), and she
taught me how to approach evaluation and benchmarking for generative models—a
major part of my current research path.

I want to thank my lab members, starting with Zak (Zakaria Aldeneh). Zak
exemplifies what all senior PhD mentors should be, helping me with code,
brainstorming, and working with me on papers. He has been an amazing research
collaborator. I also want to thank Minxue (Minxue Sandy Niu) for being the
junior research collaborator anyone would be proud of. She has not only been
an amazing collaborator but was also always willing to discuss interesting
research problems. I want to thank Matt (Matthew Perez) for being the
batchmate who has always been there to help, to vent, to advise, and to
collaborate, serving as my go-to person for any speech-based research
questions. Finally, I want to thank Amrit (Amrit Romana) for being an amazing
lab member; her observant questions helped me immensely during lab
presentations.

I also want to thank my friends, without whom this journey would not have been
possible. I will start with Abhinav (Abhinav Jangda), who has been my support
system throughout my PhD journey, starting from the application process.
Diksha (Diksha Dhawan) was the best PhD roommate one could ask for during the
first four years of my PhD. She shared laughter and tears with me, cooked with
me, and supported me through all the highs and lows. Without her, I could not
have survived my PhD. She taught me the value of being proud of my interests
in both my personal and professional life, and how friends can sometimes be
family, which is the best gift anyone could have given me. Eesh (Sudheesh
Srivastava), for all the conversations at the intersection of machine
learning, physics, and philosophy, has taught me about areas and theories that
I would have otherwise not encountered in any way or format. Conversations
with him have always left me rejuvenated, happy, and feeling peppier—a
testament to how amazing a best friend he is. Sagarika (Sagarika Srishti), for
all her support, both in India and when she came to the US. Her move to the US
during my PhD was a major personal highlight. Ariba (Ariba Javed), thanks for
all the discussions, talks, and emotional conversations, and for always being
up for anything interesting, including a pottery class. Shobhit (Shobhit
Narain) has been an amazing companion, helping me with job applications and
always being the sarcastic, serious, yet most helpful guy I have had the
pleasure of calling a friend. And finally, Sai (Sairam Tabibu) helped me fill
out the PhD application for UMich on the exact deadline, without which, I
would not be here at all.

This is probably an unconventional paragraph in acknowledgments, but these
were unconventional times during COVID. For the two years of lockdown, I
turned to Among Us when I felt lonely or lost in my research. I am really
thankful for the streamers whose broadcasts provided some semblance of social
interaction. For almost three years, I watched them stream at least 8 hours a
day while I worked, to simulate a social environment. And when my research
progress stalled, I turned to anonymous Discord communities, playing Among Us
and golf for hours, which helped alleviate feelings of depression and sadness,
providing a much-needed uplift.

My PhD journey wasn’t easy, and a lot happened over the six years, but I made
it through. The credit for that goes to all the people mentioned here, to whom
I am forever indebted.

TABLE OF CONTENTS

toc

LIST OF FIGURES

lof

LIST OF TABLES

lot

ABSTRACT

Emotion recognition is a complex task due to the inherent subjectivity in both
the perception and production of emotions. The subjectivity of emotions poses
significant challenges in developing accurate and robust computational models.
This thesis examines critical facets of emotion recognition, beginning with
the collection of diverse datasets that account for psychological factors in
emotion production. To address these complexities, the thesis makes several
key contributions.

To handle the challenge of non-representative training data, this work
collects the Multimodal Stressed Emotion dataset, which introduces controlled
stressors during data collection to better represent real-world influences on
emotion production. To address issues with label subjectivity, this research
comprehensively analyzes how data augmentation techniques and annotation
schemes impact emotion perception and annotator labels. It further handles
natural confounding variables and variations by employing adversarial networks
to isolate key factors like stress from learned emotion representations during
model training. For tackling concerns about leakage of sensitive demographic
variables, this work leverages adversarial learning to strip sensitive
demographic information from multimodal encodings. Additionally, it proposes
optimized sociological evaluation metrics aligned with cost-effective, real-
world needs for model testing.

The findings from this research provide valuable insights into the nuances of
emotion labeling, modeling techniques, and interpretation frameworks for
robust emotion recognition. The novel datasets collected help encapsulate the
environmental and personal variability prevalent in real-world emotion
expression. The data augmentation and annotation studies improve label
consistency by accounting for subjectivity in emotion perception. The
stressor-controlled models enhance adaptability and generalizability across
diverse contexts and datasets. The bimodal adversarial networks aid in
generating representations that avoid leakage of sensitive user information.
Finally, the optimized sociological evaluation metrics reduce reliance on
extensive expensive human annotations for model assessment.

This research advances robust, practical emotion recognition through
multifaceted studies of challenges in datasets, labels, modeling, demographic
and membership variable encoding in representations, and evaluation. The
groundwork has been laid for cost-effective, generalizable emotion recognition
models that are less likely to encode sensitive demographic information.

##  Chapter I Introduction

In human communication, perceiving and responding to others’ emotions in
interpersonal conversations play a crucial role [76]. To create systems that
can aid in human-centered interpersonal situations, it is necessary for these
systems to possess the capability to recognize emotions effectively [220].
Robust Emotion Recognition (ER) models can be beneficial in various
situations, such as crisis text lines or passive mental health monitoring
[160]. However, these ML models often lack robustness when faced with unseen
data situations, making deploying them in high-risk situations or healthcare a
challenging task [241].

Recongizing emotion is a challenging task because it is subjective in both
perception and production [178]. The labels used to train emotion recognition
models are perceptually subjective [28]. The same emotion can be perceived
differently by different people, depending on their cultural background,
personal experiences, and other factors [147]. Additionally, there is
production subjectivity. The same emotion can be expressed differently by
different people, depending on their individual personality, cultural
background, physiological and other factors [13]. The subjectivity of emotion
recognition makes it difficult to develop accurate and robust models that
account for these numerous variations [221].

In addition to the challenges posed by subjectivity, there are challenges that
relate to the information that is learned in addition to and beyond the
expression of emotion itself. The manner in which emotions are expressed are
correlated with a person’s demographic and identifying features. Hence,
systems trained to recognize emotion can often learn implicit associations
between an individual’s demographic factors and emotion [208]. When used as a
component in larger systems, these implicit associations can lead to either
the leakage of demographic information, or can bias the larger system’s output
based on demographic information, even when not explicitly trained to do so.

Training any robust machine learning model necessitates having access to large
amounts of diverse and labelled data. Training models for emotion recognition
faces the challenge of not having access to large quantities of diverse data.
Scraping data over the internet, as is done for other areas, leads to a
dataset that is often demographically biased, and, often exaggerated for
entertainment purposes. On the other hand, data collected in laboratory
environments is intentionally cleaner and often exaggerated in case of
scripted sessions. Therefore, both of these data collection methods do not
encapsulate possible environmental and personal factors, which leads to models
often being trained on either highly skewed or non-representative data. The
resulting models are either fragile or biased, and ultimately unable to handle
real-world variability.

In this dissertation, critical facets of emotion recognition are thoroughly
explored, beginning with the collection of datasets, which take into account
psychological factors in producing emotions. This is followed closely by
examining the influence that alterations in data augmentation processes have
on emotion labels, while also challenging and interrogating the validity of
previously established labels. Alterations in labeling techniques and the
resulting effects on annotator-assigned labels are also scrutinized.
Simultaneously, the research develops robust models specifically trained to
disregard certain physiological emotion production factors. Integral to the
research is the creation of bimodal models that generate representations
aiming to tackle the reduction of leakage of sensitive demographic variables.
The concluding portion of the study involves an in-depth evaluation of the
robustness and impartiality of these models, carried out in a human-centric
manner, ensuring an emphasis on minimal costs for data annotation. From this
extensive research, valuable insights are gained into the complexities of
emotion recognition, which pave the way for more nuanced and robust labeling,
modeling, and interpretation techniques. It also lays the groundwork for
future efforts in the development of robust and cost-effective emotion
recognition models.

###  1.1 Emotion Theories and the Impact on Emotion Recognition Model
Development

To better understand the subjectivity inherent in emotion recognition and its
correlation with the research gaps and challenges, we must first explore the
contrasts between emotion production and emotion perception theories. These
theories elucidate the distinct factors related to the subjectivity of
emotions in both production and recognition processes and offer valuable
insights for developing robust and unbiased emotion recognition models.

####  1.1.1 Emotion Production and Emotion Perception

Emotion production refers to experiencing and generating emotional responses,
encompassing several factors, including cognitive appraisal, physiological
response, behavior and expression, and subjective experience. These components
work together to create the unique process of producing emotions within each
person.

Emotion perception, conversely, focuses on recognizing and interpreting
others’ emotional signals, influenced by factors such as emotional cues,
context and environment, past experiences and learning, and individual
differences. This process involves making sense of others’ emotions based on
various internal and external factors.

####  1.1.2 Emotion Theories, Research Challenges, and Implications for
Emotion Recognition

Various theories of emotion provide insights into the challenges faced in
developing computational models for emotion recognition in speech or text.
Below, we discuss the relevance and implications of some prominent theories in
the context of speech or text-based (bimodal) emotion recognition.

  * •

James-Lange Theory and Cannon-Bard Theory [204]: Both theories emphasize
physiological responses’ importance in emotion. In speech or text-based
recognition, it is vital to consider correlations between observable features
(e.g., vocal tonality, speech patterns) and underlying physiological
responses. Accounting for these correlations can help capture emotions, even
though the relationship might be subjective due to personal and cultural
differences.

  * •

Schachter-Singer Two-Factor Theory [204]: This theory stresses the importance
of both physiological arousal and cognitive appraisal for experiencing
emotions. In speech or text-based emotion recognition, cognitive appraisal
aspects such as semantic content, contextual factors, and discourse patterns
can be extracted. However, the subjectivity of cognitive appraisal processes
presents challenges given personal experiences’ impact on interpretation.

  * •

Lazarus Cognitive-Mediational Theory [204]: Centered around the role of
cognitive appraisal, this theory highlights the need for emotion recognition
systems to account for individuals’ interpretations of situations through cues
that may suggest appraisal (e.g., word choice, phrase structure,
conversational context). Advanced models might need to factor in users’
personal and demographic features to better understand cognitive appraisal
processes. This approach introduces more subjectivity and potential privacy
concerns, as individual perspectives and experiences can vary significantly.

Integrating insights from these theories can aid unraveling the complexities
and subjective nature of emotions expressed through language, as speech or
text-based emotion recognition relies primarily on linguistic patterns, tone,
and content analysis.

####  1.1.3 Addressing Challenges Through Thesis Contributions

The thesis contributions align with and address the subjectivity challenges in
emotion production and perception, thus tackling the complexities involved in
developing robust and unbiased emotion recognition models.

  * •

Collecting datasets that account for psychological factors in emotion
production: By considering psychological factors influencing unique emotional
experiences, more diverse datasets are created, allowing models to account for
subjectivity in emotion production and generalize across emotions.

  * •

Examining the influence of data augmentation processes on emotion perception
labels: This contribution seeks to understand data augmentation’s impact on
ground truth labels, creating better representations of emotions in the
datasets, accounting for subjectivity in emotion perception.

  * •

Analyzing labeling setups’ impact on annotators’ emotion perception labels:
This investigates how labeling setups influence emotion perception, aiming to
improve label consistency and reduce inter-annotator disagreement, thus better
representing subjectivity in emotion perception.

  * •

Training robust models by explicitly disregarding emotion production factors:
This minimizes the impact of subjective elements associated with emotion
production, enabling models to focus on core emotional cues.

  * •

Developing bimodal models for generation of emotion representations that are
debiased and reduce encoding of demographic and membership information: This
creates models that consider multiple emotional cues while disregarding
sensitive features, addressing subjectivity challenges in both emotion
production and perception.

  * •

Evaluating models in a human-centric manner: Designing evaluation methods
aligned with real-world expectations and without incurring significant
annotation costs ensures the models effectively tackle subjectivity challenges
in a practical way.

By focusing on these contributions, the thesis emphasizes the connection
between emotion production and perception’s subjectivity and its influence on
model development, advancing the creation of more robust and unbiased emotion
recognition models.

###  1.2 Emotion Recognition

Emotion recognition models are customarily trained using laboratory-collected
data encompassing video, audio, and corresponding text. These algorithms
strive to capture the speaker’s underlying emotional state either autonomously
or as part of a larger pipeline, such as response generation. Supervised
learning techniques predominantly train these models. Obtaining ground truth
labels for the dataset samples is crucial for successfully training a
supervised learning model. The emotion theories presented earlier are
intrinsically linked with the complexity of emotion recognition. Understanding
the interplay between these theories and model development is essential.

####  1.2.1 Emotion Labels

Emotion labels typically fall into two categories: categorical and
dimensional. Categorical variables aim to discretely categorize emotion
attributes, such as excitement, happiness, anger, or sadness. These labels’
limitations align with the James-Lange and Cannon-Bard theories—emotions are
subjective, making it difficult to define universal emotions across cultures.
This subjectivity is intensified by both personal physiological responses to
stimuli and cultural context.

Dimensional emotional labels describe emotions across two dimensions, valence
(sad to happy) and arousal (calm to excited). The dimensional approach is more
consistent with the James-Lange and Cannon-Bard theories, addressing the
physiological components of emotions, as well as the cognitive components
emphasized by the Schachter-Singer Two-Factor Theory and Lazarus Cognitive-
Mediational Theory. However, these dimensional labels also face the challenge
of cultural and personal influences on the perception and expression of
emotions.

####  1.2.2 Emotion Features

Three primary modalities are used in combination to train emotion recognition
models: text, audio, and video. This thesis focuses predominantly on audio and
its corresponding text as the feature set for these models.

Mel-filterbanks (MFBs) are often used as inputs to neural network models in
speech. MFBs can capture correlations between vocal tonality, speech patterns,
and underlying physiological responses. Nevertheless, factors like pitch,
volume, or other nuances of speech may be affected by cultural and linguistic
contexts. Furthermore, personal characteristics can influence these features,
further complicating emotion recognition in cross-cultural or highly diverse
settings.

Language features, which provide contextualized representations for words,
capture the cognitive appraisal aspects (semantic content, contextual factors,
and discourse patterns). The Lazarus Cognitive-Mediational Theory further
highlights the need for models that account for user demographics. More
advanced models may need to balance the understanding of individual emotions
with ethical considerations.

####  1.2.3 Emotion Recognition Models

Audio-based emotion recognition models initially relied on Hidden Markov
Models (HMMs) or Gaussian Mixture Models (GMMs) and later shifted focus to
LSTMs and RNNs. These models aim to capture the dynamic and time-varying
nature of speech, reflecting the James-Lange Theory and Cannon-Bard Theory’s
emphasis on physiological responses. However, these models must also account
for the inherent cultural and linguistic differences in the way emotions are
expressed through speech.

Language-based models, like recent advances in transformer architectures,
address long and indirect contextual information challenges, in line with the
Schachter-Singer Two-Factor Theory’s cognitive appraisal aspects. These models
strive to understand the nuances of language, cultural expressions, and
individual semantic and contextual differences in recognizing emotions.

Multi-modal models exploit relevant information from text, audio, or video to
form powerful emotion recognition models. Informed by the emotion theories,
these models take into account the subjectivity of emotions by leveraging
different modalities to discern the nuances of emotion expression. By
combining these modes, models can better account for the emotional complexity
that arises from intercultural and personal differences in perception,
expression, and context.

###  1.3 Challenges in Emotion Recognition

The variable and subjective nature of emotions make it challenging to train
models that can accurately identify emotion in any given scenario. Addressing
three major challenges is necessary for any emotion recognition model deployed
in a real-world setting: (a) Non-representative training data, (b) Subjective
labels, (c) Unintentional encoding and leakage of sensitive information.
Previous work has looked at varying ways to counter these challenges, talked
about in detail in Chapter III, Section 2.1.

####  1.3.1 Non-representative data

Emotion production in real-world settings is influenced by various factors,
including data collection settings, demographics, and personal factors.
Addressing these confounding factors aligns with the implications of the
earlier-discussed emotion theories. Researchers can tackle this challenge by
developing more robust models, incorporating real-world variability through
dataset augmentation or mitigating confounding factors.

####  1.3.2 Label Subjectivity

As highlighted in the emotion theories, emotions are inherently subjective and
deeply influenced by personal experiences, culture, and context. This
subjectivity leads to difficulty in pinpointing an objective and universal
ground truth for training emotion recognition models. Researchers should
account for label subjectivity by using diverse and representative datasets,
annotations from multiple sources, and considering multiple emotion theories
during the model design process.

####  1.3.3 Unintentional encoding and leakage of sensitive information

Variability can lead unintentional encoding and leakage of sensitive
information concerns, specifically in human centered tasks, such as emotion
recognition models, as the associative nature of the task and sensitive
demographic variables may inadvertently lead to encoding personal information.

###  1.4 Proposed Methods

A robust and effective emotion recognition system must successfully navigate a
range of challenges, including addressing subjectivity in emotion production
and perception, handling natural variations and confounding variables,
reducing encoded sensitive information, and providing relevant evaluation
metrics. Here, we present a series of proposed methods aligned with the
outlined contributions to address these challenges.

####  1.4.1 Dataset Collection for Emotion Recognition

Tackling the challenge of subjectivity in emotion production, it’s essential
that we consider the issues in widely used emotion recognition datasets that
arise due to design choices, methodology of data collection, and inherent
subjectivity. Emotion datasets traditionally aim for minimal variation to
ensure generalizability. However, this can result in non-robust models that
struggle with unexpected variability. We propose the construction and
validation of a new dataset called Multimodal Stressed Emotion (MuSE), which
introduces a controlled situational confounder (stress) to better account for
subjectivity. In addition, we discuss the use of domain adversarial networks
to achieve more stable and reliable cross-corpus generalization while avoiding
undesired characteristics in encodings.

####  1.4.2 Data Augmentation with Noise in Emotion Datasets

Addressing the challenge of subjectivity in emotion perception, we examine
data augmentation with noise in emotion datasets, focusing on the Interactive
Emotional Dyadic Motion Capture (IEMOCAP) dataset, which features dyadic
interactions with text, video, and audio modalities. Introducing realistic
noisy samples through environmental and synthetic noise, we evaluate how
ground truth and predicted labels change due to noise sources. We discuss the
effects of commonly used noisy augmentation techniques on human emotion
perception, potential inaccuracies in model robustness testing, and provide
recommendations for noise-based augmentation and model deployment.

####  1.4.3 Annotations of Emotion Datasets

To further address subjectivity in emotion perception, we investigate how
design choices in the annotation collection process impact the performance of
trained models. Focusing on contextual biasing, we examine how annotators
perceive emotions differently in the presence or absence of context. Commonly-
used emotion datasets often involve annotators who have knowledge of previous
sentences, but models are frequently evaluated on individual utterances. We
explore the implications of this discrepancy on model evaluation, and its
potential for generating errors.

####  1.4.4 Methods for Handling Natural Variations and Confounding Variables

As mentioned earlier, we collect a dataset of differences in similar emotion
production under varying levels of stress. Emotion recognition models may
spuriously correlate these stress-based factors to perceived emotion labels,
which could limit generalization to other datasets. Consequently, we
hypothesize that controlling for stress variations can improve the models’
generalizability. To achieve this, we employ adversarial networks to
decorrelate stress modulations from emotion representations, examining the
impact of stress on both acoustic and lexical emotion predictions. By
isolating stress-related factors from emotion representations, we aim to
enhance the model’s ability to generalize across different stress conditions.
Furthermore, we analyze the transferability of these refined emotion
recognition models across various domains, assessing their adaptability to
evolving contexts and scenarios. Ultimately, our approach aims to improve
emotion recognition model robustness by addressing the inherent variability of
emotional expression due to stress and ensuring greater applicability across
multiple domains.

####  1.4.5 Approaches for Tackling Sensitive Information Leakage in Trained
Emotion Recognition Models

Emotions are inherently related to demographic factors such as gender, age,
and race. Consequently, emotion recognition models often learn these latent
variables even if they are not explicitly trained to do so. This learning
behavior poses a risk to user privacy, as the models inadvertently capture
sensitive demographic information. Storing representations instead of raw data
does not fully mitigate this issue, as latent variables can still compromise
user privacy. To address this challenge, we present approaches for mitigating
the learning of certain demographic factors in emotion recognition embeddings.
Furthermore, we tackle the issue of user-level membership identification by
employing an adversarial network that strips this information from the final
encoding, reduced leakage of sensitive information from generated
representations.

####  1.4.6 Methods for Model Evaluation and Perception

Large language models face limitations in subjective tasks like emotion
recognition due to inadequate annotation diversity and data coverage.
Acquiring comprehensive annotations and evaluations is often costly and time-
consuming. To address these challenges, we propose cost-effective sociological
metrics for emotion generalization and reduced demographic vairable leakage.
These metrics reduce reliance on expensive human-based feedback while still
capturing the nuances of human emotions. By evaluating model performance and
demographic variables encoded in generated representations, the proposed
metrics improve cross-corpus results and allow for the development of
accurate, relevant emotion recognition models in a more economic manner.

###  1.5 Contributions

This dissertation proposes several investigations and novel solutions to
address various concerns related to real-world emotion recognition model
deployment.

The contributions of the works in this dissertation can be summarized as
follows:

  * •

Chapter IV:

    * –

Introduction of Multimodal Stressed Emotion (MuSE) dataset.

    * –

Detailed data collection protocol.

    * –

Potential uses and emotion content annotations.

    * –

Performance measuring baselines for emotion and stress classification.

  * •

Chapter V:

    * –

Speech emotion recognition’s impact under influence of various factors such as
noise.

    * –

Investigation of noise-altered annotation labels and their aftermath.

    * –

Consequences on evaluation of ML models considering noise.

    * –

Specific recommendations for noise augmentations in emotion recognition
datasets.

  * •

Chapter VI:

    * –

Crowdsourced experiments to study the subjectivity in emotion expression and
perception.

    * –

Contextual and randomized annotation schemes of the MuSE dataset.

    * –

Comparative analysis revealing contextual scheme’s closeness to speaker’s
self-reported labels.

  * •

Chapter VII:

    * –

Examination of emotion expressions under stress variations.

    * –

Utilization of adversarial networks to separate stress modulations from
emotion representations.

    * –

Exploration of stress’s impact on acoustic and lexical emotional predictions.

    * –

Evidence of improved generalizability with stress control during model
training.

  * •

Chapter VIII:

    * –

Highlighting the unintentional leak of sensitive demographic information in
multimodal representations.

    * –

Use of adversarial learning paradigm to improve sensitive information
reduction metric.

    * –

Maintenance of primary task performance, despite improvements to privacy.

  * •

Chapter IX:

    * –

New template formulation to derive human-centered, optimizable and cost-
effective metrics.

    * –

Correlation establishment between emotion recognition performance, biased
representations and derived metrics.

    * –

Employment of metrics for training an emotion recognition model with increased
generalizability and decreased bias.

    * –

Finding of positive correlation between proposed metrics and user preference.

###  1.6 Outline of the dissertation

Initiating with Chapter III, it delves into a comprehensive review of
pertinent literature spanning from emotion recognition and privacy
preservation to adversarial networks, model interpretability, and
crowdsourcing designs. Moving forward, Chapter II provides an introduction to
the common datasets, and features employed throughout this research.
Subsequent chapters, from Chapter IV to IX, engage in a thorough exploration
and discussion of the research work undertaken, characterized in the
Contributions section. Lastly, Chapter X serves as a conclusive summary
encapsulating the primary contributions made, elaborating on the proposed
future works.

##  Chapter II Related Work: Modeling Emotions

Emotion recognition is a complex, multifaceted field drawing on various
research areas. This chapter explores the various methods and considerations
in this field, from the use of crowdsourcing to the importance of context, and
from handling confounding factors to the impact of noise on machine learning
models. We explore the ethical considerations of unintentional encoding of
sensisitive variables in data collection and neural networks, the role of
interpretability in model trustworthiness, and the importance of automating
human in the loop feedback. We also delve into the challenge of
generalizability in emotion recognition.

###  2.1 Concerns with Emotion Recognition Datasets

Some aspects of the above mentioned datasets limit their applicability,
including: a lack of naturalness, unbalanced emotion content, unmeasured
confounding variables, small size, small number of speakers, and presence of
background noise. These datasets are also limited in the number of modalities
they use, usually relying on visual and acoustic/lexical information.

####  2.1.1 Recorded Modalities

As shown in Table 2.1, the most common modalities are video, acoustics, and
text. In addition to these modalities, we chose to record two more modalities:
thermal and physiological. Previous research has shown that thermal recordings
perform well as non-invasive measurement of physiological markers like,
cardiac pulse and skin temperature [173, 172, 80]. They have been shown to be
correlated to stress symptoms, among other physiological measures. We used the
physiological modality to measure stress responses [234, 210] to psychological
stressors. This modality has been previously noted in literature for measuring
stress [96], usually measured in polygraph tests. We perform baseline
experiments to show that the modalities collected in the dataset are indeed
informative for identifying stress and emotion.

####  2.1.2 Lack of Naturalness

A common data collection paradigm for emotion is to ask actors to portray
particular emotions. These are usually either short snippets of information
[36], a single sentence in a situation [38], or obtained from sitcoms and
rehearsed broadcasts [47]. A common problem with this approach is that the
resulting emotion display is not natural [113]. These are more exaggerated
versions of singular emotion expression rather than the general, and messier,
emotion expressions that are common in the real world [12, 21, 72]. Further,
expressions in the real world are influenced by both conversation setting and
psychological setting. While some datasets have also collected spontaneous
data [36, 38], these utterances, though emotionally situated, are often
neutral in content when annotated. The usual way to get natural emotional data
is to either collect data using specific triggers that have been known to
elicit a certain kind of response or to completely rely on in-the wild data,
which however often leads to unbalanced emotional content in the dataset
[183].

####  2.1.3 Unbalanced Emotion Content

In-the-wild datasets are becoming more popular [47, 118, 138]. The usual
limitation to this methodology is that, firstly, for most people, many
conversations are neutral in emotion expression. This leads to a considerable
class imbalance [183]. To counter this issue, MSP-Podcast [143] deals with
unbalanced content by pre-selecting segments that are more likely to have
emotional content. Secondly, data collected in particular settings, e.g.,
therapy [162], or patients with clinical issues [130] comprise mostly of
negative emotions because of the recruitment method used in the collection
protocol.

####  2.1.4 Presence of Interactional Variables

The common way of inducing emotions involves either improvisation prompts or
scripted scenarios. Emotion has been shown to vary with a lot of factors that
are different from the intended induction [198, 240, 156]. These factors in
general can be classified into: (a) recording environment confounders and (b)
collection confounders. Recording environment-based variables hamper the
models’ ability to to learn the emotion accurately. These can be environment
noise [16], placement of sensors or just ambient temperature [31].

Table 2.1: Summary of some of the existing emotion corpora. Lexical modality
is mentioned for manually transcribed datasets. A - Audio, L - Lexical, T-
Thermal, V- Visual, P - Physiological.
| Corpus | Size | Speakers | Rec. Type | Language | Modality | Annotation Type
---|---|---|---|---|---|---|---
1. | IEMOCAP | 12h26m | 10 | improv/acted | English | A, V, L | Ordinal, Categorical
2. | MSP-Improv | 9h35m | 12 | improv/acted | English | A, V | Ordinal
3. | VAM | 12h | 47 | spontaneous | German | A, V | Ordinal
4. | SEMAINE | 6h21m | 20 | spontaneous | English | A, V | Ordinal, Categorical
5. | RECOLA | 2h50m | 46 | spontaneous | French | A, V, P | Ordinal
6. | FAU-AIBO | 9h12m | 51 | spontaneous | German | A, L | Categorical
7. | TUM AVIC | 0h23m | 21 | spontaneous | English | A, V, L | Categorical
8. | Emotion Lines | 30k samples | - | spont/scripted | English | A, L | Categorical
9. | OMG-Emotion | 2.4k samples | - | spontaenous | English | A, V, L | Ordinal
10. | MSP-Podcast | 27h42m | 151 | spontaenous | English | A | Ordinal, Categorical
11. | MuSE | 10h | 28 | spontaneous | English | A, V, L, T, P | Ordinal (Random, Context)

####  2.1.5 Demographics in Dataset Collection Recruitment

The data collection variations influence both the data generation and data
annotation stages. The most common confounders are gender, i.e., ensuring an
adequate mix of male vs female, and culture, i.e., having a representative
sample to train a more general classifier. Another confounding factor includes
personality traits [242], which influence how a person both produces [242] and
perceives [158] emotion. Another confounder that can occur at the collection
stage is the familiarity between the participants, like RECOLA [183], which
led to most of the samples being mainly positive due to the colloquial
interaction between the participants. They also do not account for the
psychological state of the participant. Psychological factors such as stress
[132], anxiety [229] and fatigue [26] have been shown previously to have
significant impact on the display of emotion. But the relation between these
psychological factors and the performance of models trained to classify
emotions in these situations has not been studied.

###  2.2 Crowdsourcing and Context in Emotion Recognition

Crowdsourcing has emerged as a highly efficient approach for gathering
dependable emotion labels, as extensively investigated by Burmania et al.
[33]. In addition to this, previous studies have concentrated on enhancing the
dependability of annotations by employing quality-control methods. For
instance, Soleymani et al. [201] have proposed the utilization of
qualification tests to weed out spammers from the crowd, thereby ensuring the
quality of collected data. Furthermore, Burmania et al. [35] have explored the
use of gold-standard samples to continuously monitor the reliability and
fatigue levels of annotators.

The interpretation of emotions is heavily influenced by the context in which
they are expressed. Various factors such as tone, choice of words, and facial
expressions can significantly impact how individuals perceive and understand
emotions [129]. It is noteworthy that this contextual information is
implicitly incorporated in the labeling schemes of commonly used emotion
datasets like IEMOCAP [36] and MSP-Improv [38]. However, a notable disparity
often exists between the information available to human annotators and that
accessible to emotion classification systems. This discrepancy arises because
emotion recognition systems are typically trained on individual utterances
[10, 3, 157, 190].

###  2.3 Handling Confounding Factors

####  2.3.1 Singularly Labeled or Unlabeled Factors

To address confounding factors that are either labeled singularly or cannot be
labeled, researchers have devised specific methods. For instance, Ben-David et
al. [23] conducted a study wherein they showed that a sentiment classifier,
trained to predict the sentiment expressed in reviews, could also implicitly
learn to predict the category of the products being reviewed. This finding
highlights the potential of classifiers to capture additional information
beyond their primary task. In a similar vein, Shinohara [196] employed an
adversarial approach to train noise-robust networks for automatic speech
recognition. By leveraging this technique, Shinohara aimed to enhance the
network’s ability to handle noisy and distorted speech signals.

####  2.3.2 Explicitly Labeled Factors

In addition to addressing confounding factors that are singularly or
unlabeled, researchers have also developed methods to handle confounding
factors that are explicitly labeled during the data collection process. One
such approach involves the use of adversarial multi-task learning, which aims
to mitigate variances caused by speaker identity [153]. By incorporating this
technique, researchers can reduce the influence of speaker-specific
characteristics on the emotion recognition system, thereby enhancing its
generalizability. Furthermore, a similar approach has been employed to prevent
networks from learning publication source characteristics, which could
introduce biases in the classification process [149]

###  2.4 Noise and Approaches to Dealing with it in Machine Learning Models

The impact of noise on machine learning models has been the subject of
extensive research, which can be broadly classified into three main
directions: robustness in automatic speech recognition, noise-based
adversarial example generation, and performance improvement through model
augmentation with noise.

One area of focus is the robustness of models in automatic speech recognition
(ASR) when exposed to noisy environments. Researchers have explored various
techniques to enhance the performance of ASR systems in the presence of noise.
This includes the development of noise-robust feature extraction methods, such
as mel-frequency cepstral coefficients (MFCCs) and perceptual linear
prediction (PLP) features [135]. These techniques aim to minimize the impact
of noise on the accuracy of speech recognition systems, enabling them to
effectively operate in real-world, noisy conditions.

Another line of research involves the generation of noise-based adversarial
examples, which are intentionally crafted to deceive machine learning models.
Adversarial attacks exploit vulnerabilities in models by adding imperceptible
noise to input samples, causing the models to misclassify or produce incorrect
outputs. Carlini and Wagner [42] and Gong et al. [89] have proposed
methodologies for generating adversarial audio examples that can fool ASR
systems. These techniques highlight the importance of understanding and
addressing the susceptibility of machine learning models to adversarial noise.

Furthermore, researchers have explored the potential benefits of incorporating
noise during the training and augmentation process of machine learning models.
By augmenting the training data with various types of noise, models can become
more robust and adaptable to real-world conditions. For instance, Sohn et al.
[200] and Wallace et al. [224] have investigated the effectiveness of noise
augmentation techniques in improving the performance of models across
different tasks. These methods aim to enhance model generalization and reduce
overfitting, ultimately leading to better model performance in noise-affected
scenarios.

While evaluating model robustness to noise or adversarial attacks, researchers
commonly introduce noise into the dataset and assess the model’s performance
[5]. However, when it comes to emotion recognition, introducing noise while
ensuring that the perception of emotions remains intact can be highly
challenging. It is crucial to strike a balance between adding noise for
robustness evaluation purposes and preserving the original emotional content.
This ensures that the introduced noise does not distort or alter the true
emotional expression, enabling accurate and reliable emotion recognition
systems.

###  2.5 Unintentional Sensisitve Variable Encoding, and Ethical
Considerations in Data Collection and Neural Networks

The preservation of privacy in data collection has been a key area of focus in
early research. Various methods such as rule-based systems and the
introduction of background noise have been explored in order to achieve this
goal [88, 69]. However, more recent studies have shifted their attention
towards privacy preservation in the context of neural networks. In particular,
researchers have primarily concentrated on ensuring that the input data used
in these networks are not memorized and cannot be retrieved even when the
model is deployed [41, 2].

Another crucial consideration in the field of privacy preservation is fair
algorithmic representation. The objective here is to develop networks that are
invariant to specific attributes, often related to demographic information, in
order to ensure fairness [29, 59, 61]. Although certain methods have
demonstrated promise in achieving fairness, they may still inadvertently lead
to privacy violations [108].

###  2.6 The Role of Interpretability in Model Trustworthiness

The aspect of interpretability plays a crucial role in establishing
trustworthiness of models. Studies have indicated that individuals are more
inclined to trust the decisions made by a model if its explanations align with
their own decision-making processes [203, 73, 195]. In addition,
interpretability methods can be employed by model designers to evaluate and
debug a trained model [68]. These methods provide insights into the inner
workings of the model and facilitate a better understanding of its decision-
making process.

###  2.7 Automating Human in the Loop Feedback

In order to automate human in the loop feedback, several approaches have been
proposed. One such approach involves the utilization of a teacher-student
feedback model, where feedback from human teachers is used to improve the
performance of the model [179]. Another avenue of research focuses on
enhancing active learning techniques, which aim to select the most informative
data points for annotation by human experts, thereby reducing the overall
labeling effort required [115].

These methods often incorporate a combination of fine-tuning and prompt-based
learning techniques, which further enhance the model’s ability to learn from
human feedback and adapt its performance accordingly [217]. By fine-tuning the
model based on the feedback received and utilizing prompts as guiding cues,
these approaches enable the model to continually improve its performance,
making it more effective in addressing the specific task or problem at hand.

###  2.8 Generalizability in Emotion Recognition

Achieving generalizability in emotion recognition poses a significant
challenge for researchers. To address this challenge, various methods have
been explored in order to obtain models that can generalize well across
different datasets and scenarios. One approach is the use of combined and
cross-dataset training, where multiple datasets are combined during the
training process to create a more comprehensive and diverse training set. This
helps the model learn a wider range of emotion patterns and improves its
ability to generalize to unseen data [137].

Another technique that has been investigated is transfer learning, which
involves leveraging knowledge acquired from pre-trained models on a related
task and applying it to the emotion recognition task. By transferring the
learned representations and weights from a pre-trained model, the model can
benefit from the general knowledge and feature extraction capabilities it has
acquired, leading to improved generalizability in emotion recognition [137].

Furthermore, researchers have also explored the concept of generalizability
from the perspective of noisy signals. Emotion recognition often deals with
noisy data, such as speech with background noise or facial expressions with
occlusions. By developing models that are robust to such noise and can
effectively extract emotion-related information from imperfect signals, the
generalizability of the models can be enhanced [93].

###  2.9 Conclusion

The field of emotion recognition is complex, with many factors and
considerations influencing the development and deployment of effective models.
This chapter has explored some of the key areas in this field, highlighting
the importance of crowdsourcing, context, handling confounding factors,
dealing with noise, and ensuring that the representations don’t inadverdently
encode sensitive demographic or membership information. The role of
interpretability in model trustworthiness and the challenge of automating
human in the loop feedback were also discussed. Although progress has been
made in many of these areas, the challenge of generalizability in emotion
recognition remains, and future research will need to continue to address this
issue.

##  Chapter III Datasets and Pre-processing

This thesis focusses on emotion recognition as a task. For this purpose, we
use a standard set of datasets and features as described in this chapter. This
allows us to perform experiments with a set of known and commonly used
datasets, keeping them uniform across experimental variables.

###  3.1 Datasets Used In Thesis

In the past years, there have been multiple emotional databases collected and
curated to develop better emotion recognition systems. Table 2.1 shows the
major corpora that are used for emotion recognition.

####  3.1.1 IEMOCAP

The IEMOCAP dataset was created to explore the relationship between emotion,
gestures, and speech. Pairs of actors, one male and one female (five males and
five females in total), were recorded over five sessions. Each session
consisted of a pair performing either a series of given scripts or
improvisational scenarios. The data were segmented by speaker turn, resulting
in a total of 10,039 utterances (5,255 scripted turns and 4,784 improvised
turns).

####  3.1.2 MSP-Improv

The MSP-Improv dataset was collected to capture naturalistic emotions from
improvised scenarios. It partially controlled for lexical content by including
target sentences with fixed lexical content that are embedded in different
emotional scenarios. The data were divided into 652 target sentences, 4,381
improvised turns (the remainder of the improvised scenario, excluding the
target sentence), 2,785 natural interactions (interactions between the actors
in between recordings of the scenarios), and 620 read sentences for a total of
8,438 utterances.

####  3.1.3 MSP-Podcast

The MSP-Podcast dataset was collected to build a naturlisitic emotionally
balanced speech corpus by retrieving emotional speech from existing podcast
recordings. This was done using machine learning algorithms, which along with
a cost-effective annotation process using crowdsourcing, led to a vast and
balanced dataset. We use a pre-split part of the dataset which has been
identified for gender of the speakers which comprises of 13,555 utterances.
The dataset as a whole contains audio recordings.

####  3.1.4 MuSE

The MuSE dataset consists of recordings of 28 University of Michigan college
students, 9 female and 19 male, in two sessions: one in which they were
exposed to an external stressor (final exams period at University of Michigan)
and one during which the stressor was removed (after finals have concluded).
Each recording is roughly 45-minutes. We expose each subject to a series of
emotional stimuli, short-videos and emotionally evocative monologue questions.
These stimuli are different across each session to avoid the effect of
repetition, but capture the same emotion dimensions. At the start of each
session, we record a short segment of the user in their natural stance without
any stimuli, to establish a baseline. We record their behavior using four main
recording modalities: 1) video camera, both close-up on the face and wide-
angle to capture the upper body, 2) thermal camera, close-up on the face, 3)
lapel microphone, 4) physiological measurements, in which we choose to measure
heart rate, breathing rate, skin conductance and skin temperature (Figure
4.1). The data include self-report annotations for emotion and stress
(Perceived Stress Scale, PSS) [56, 57], as well as emotion annotations
obtained from Amazon Mechanical Turk (AMT). To understand the influence of
personality on the interaction of stress and emotion, we obtain Big-5
personality scores [87], which was filled by 18 of the participants, due to
the participation being voluntary.

###  3.2 Data Pre-Processing

We use these features consistently across the thesis to have a standardized
set of inputs, aiming to avoid variability that comes from different labelling
or pre-processing schemas. Our preprocessing corresponds to converting Likert
scale emotion annotations to classes based on quartiles. The feature
processing has 2 components, acoustic and lexical, for training, testing or
fine-tuning speech-only, text-only or bimodal models.

####  3.2.1 Emotion Labels

Each utterance in the MuSE dataset was labeled for activation and valence on a
nine-point Likert scale by eight crowd-sourced annotators [105], who observed
the data in random order across subjects. We average the annotations to obtain
a mean score for each utterance, and then bin the mean score into one of three
classes, defined as, {“low”: [min, 4.5], “mid”: (4.5, 5.5], “high”: (5.5,
max]}. The resulting distribution for activation is: {“high”: $24.58\%$,
“mid”: $40.97\%$ and “low”: $34.45\%$} and for valence is {“high”: $29.16\%$,
“mid”: $40.44\%$ and “low”: $30.40\%$}. Utterances in IEMOCAP and MSP-Improv
were annotated for valence and activation on a five-point Likert scale. The
annotated activation and valence values were averaged for an utterance and
binned as: {“low”: [1, 2.75], “mid”: (2.75, 3.25], “high”: (3.25, max]}

####  3.2.2 Stress Labels

Utterances in the the MuSE dataset include stress annotations, in addition to
the activation and valence annotations. The stress annotations for each
session were self-reported by the participants using the Perceived Stress
Scale (PSS) [58]. We perform a paired t-test for subject wise PSS scores, and
find that the scores are significantly different for both sets (16.11 vs
18.53) at $p<0.05$. This especially true for question three (3.15 vs 3.72),
and hence, we double the weightage of the score for this question while
obtaining the final sum. We bin the original nine-point adjusted stress scores
into three classes, {“low”: (min, mean$-2$], “mid”: (mean$-2$, mean$+2$],
“high”: (mean$+2$, max]}. We assign the same stress label to all utterances
from the same session. The distribution of our data for stress is “high”:
$40.33\%$, “mid”: $25.78\%$ and “low”: $38.89\%$

Improvisation Labels. Utterances in the IEMOCAP dataset were recorded in
either a scripted scenario or an improvised one. We label each utterance with
a binary value {“scripted”, “improvised”} to reflect this information.

###  3.3 Lexical and Acoustic Feature Extraction

####  3.3.1 Acoustic

We use Mel Filterbank (MFB) features, which are frequently used in speech
processing applications, including speech recognition, and emotion recognition
[116, 126]. We extract the 40-dimensional MFB features using a 25-millisecond
Hamming window with a step-size of 10-milliseconds. As a result, each
utterance is represented as a sequence of 40-dimensional feature vectors. We
$z$-normalize the acoustic features by session for each speaker.

####  3.3.2 Lexical

We have human transcribed data available for MuSE and IEMOCAP. We use the
word2vec representation based on these transcriptions, which has shown success
in sentiment and emotion analysis tasks [121]. We represent each word in the
text input as a 300-dimensional vector using a pre-trained word2vec model
[155], replacing out-of-vocab words with the $\langle unk\rangle$ token.
Besides, we also incorporate BERT embeddings for enhanced contextual
understanding. These embeddings, generated from the pre-trained BERT model,
provide deep, bidirectional representations by understanding the text context
from both directions. Each utterance is eventually represented as a sequence
of 768-dimensional feature vectors. We use just acoustic inputs for MSP-Improv
because human transcriptions are not available.

##  Chapter IV Emotion Recognition Dataset: MuSE

###  4.1 Motivation and Contributions

Endowing automated agents with the ability to provide support, entertainment
and interaction with human beings requires sensing of the users’ affective
state. These affective states are impacted by a combination of emotion
inducers, current psychological state, and various contextual factors.
Although emotion classification in both singular and dyadic settings is an
established area, the effects of these additional factors on the production
and perception of emotion is understudied. This chapter presents a dataset,
Multimodal Stressed Emotion (MuSE), to study the multimodal interplay between
the presence of stress and expressions of affect. We describe the data
collection protocol, the possible areas of use, and the annotations for the
emotional content of the recordings. The chapter also presents several
baselines to measure the performance of multimodal features for emotion and
stress classification.

###  4.2 Introduction

Virtual agents have become more integrated into our daily lives than ever
before [144]. For example, Woebot is a chatbot developed to provide cognitive
behavioral therapy to a user [74]. For this chatbot agent to be effective, it
needs to respond differently when the user is stressed and upset versus when
the user is calm and upset, which is a common strategy in counselor training
[213]. While virtual agents have made successful strides in understanding the
task-based intent of the user, social human-computer interaction can still
benefit from further research [54]. Successful integration of virtual agents
into real-life social interaction requires machines to be emotionally
intelligent [27, 238].

But humans are complex in nature, and emotion is not expressed in isolation
[90]. Instead, it is affected by various external factors. These external
factors lead to interleaved user states, which are a culmination of
situational behavior, experienced emotions, psychological or physiological
state, and personality traits. One of the external factors that affects
psychological state is stress. Stress can affect everyday behavior and
emotion, and in severe states, is associated with delusions, depression and
anxiety due to impact on emotion regulation mechanisms [122, 193, 216, 225].
Virtual agents can respond in accordance to users’ emotions only if the
machine learning systems can recognize these complex user states and correctly
perceive users’ emotional intent. We introduce a dataset designed to elicit
spontaneous emotional responses in the presence or absence of stress to
observe and sample complex user states.

There has been a rich history of visual [235, 111], speech [143], linguistic
[207], and multimodal emotion datasets [38, 36, 183]. Vision datasets have
focused both on facial movements [111] and body movement [131]. Speech
datasets have been recorded to capture both stress and emotion separately but
do not account for their inter-dependence [185, 97, 127, 243]. Stress datasets
often include physiological data [234, 210].

Existing datasets are limited because they are designed to elicit emotional
behavior, while neither monitoring external psychological state factors nor
minimizing their impact by relying on randomization. However, emotions
produced by humans in the real world are complex. Further, our natural
expressions are often influenced by multiple factors (e.g., happiness and
stress) and do not occur in isolation, as typically assumed under laboratory
conditions. The primary goal of this work is to collect a multimodal
stress+emotion dataset – Multimodal Stressed Emotion (MuSE) – to promote the
design of algorithms that can recognize complex user states.

For MuSE, The extracted features for each modality, and the anonymized dataset
(other than video) will be released publicly along with all the corresponding
data and labels. We present baseline results for recognizing both emotion and
stress in the chapter, in order to validate that the presence of these
variables can be computationally extracted from the dataset, hence enabling
further research.

###  4.3 MuSE Dataset

####  4.3.1 Experimental Protocol

Figure 4.1: Experimental Protocol For Recording

We collect a dataset that we refer to as Multimodal Stressed Emotion (MuSE) to
facilitate the learning of the interplay between stress and emotion. The
protocol for data collection is shown in Figure 4.1. There were two sections
in each recording: monologues and watching emotionally evocative videos. We
measure the stress level at the beginning and end of each recording. The
monologue questions and videos were specifically chosen to cover all
categories of emotions. At the start of each recording, we also recorded a
short one-minute clip without any additional stimuli to register the baseline
state of the subject.

Previous research has elicited situational stress such as public speaking
[123, 85, 8], mental arithmetic tasks [139] or use Stroop Word Test [215].
However, these types of stress are often momentary and fade rapidly in two
minutes [139]. We alleviate this concern by recording both during and after
final exams (we anticipate that these periods of time are associated with high
stress and low stress, respectively) in April 2018. We measure stress using
Perceived Stress Scale [57] for each participant. We measure their self-
perception of the emotion using Self-Assessment Manikins (SAM) [30]. The
recordings and the survey measures were coordinated using
Qualtrics111umich.qualtrics.com  enabling us to ensure minimal intervention
and limit the effect of the presence of another person on the emotion
production.

Each monologue section comprised of five questions broken into sections meant
to elicit a particular emotion (Table 4.1). These questions were shown to
elicit thoughtful and emotional responses in their data pool to generate
interpersonal closeness [11]. We include an icebreaker and ending question to
ensure cool off periods between change in recording section, i.e., from
neutral to monologues, and from monologues to videos, hence decreasing the
amount of carry-over emotion from the previous monologue to the next. Each
subject was presented with a different set of questions over the two
recordings to avoid repetition effect. We also shuffle the order of the other
three questions to account for order effects [133]. Each subject was asked to
speak for a minimum of two minutes. After their response to each question, the
subjects marked themselves on two emotion dimensions: activation and valence
on a Likert Scale of one to nine using self-assessment manikins [30].

For the second part of the recording, the subjects were asked to watch videos
in each of the four quadrants i.e., the combination of {low, high} $\times$
{activation, valence} of emotion. These clips were selected from the corpus
[140, 20], which tested for the emotion elicited from the people when watching
these clips (Table 4.2). The subjects were monitored for their reaction to the
clips. After viewing a clip, subjects are asked to speak for thirty seconds
about how the video made them feel. After their response, they marked a
emotion category, e.g., angry, sad, etc. for the same clip. When switching
videos, the subjects were asked to view a one-minute neutral clip to set their
physiological and thermal measures back to the baseline [189].

The 28 participants were also asked to fill out an online survey used for
personality measures on the big-five scale [87], participation being
voluntary. This scale has been validated to measure five different dimensions
named OCEAN (openness, conscientiousness, extraversion, agreeableness, and
neuroticism) using fifty questions and has been found to correlate with
passion [60], ambition [19], and emotion mechanisms [181]. We received
responses for this survey from 18 participants. These labels can be used in
further work to evaluate how these personality measures interact with the
affects of stress in emotion production, as previously studied in [242].

Table 4.1: Emotion elicitation questions. Icebreaker
---
1. Given the choice of anyone in the world, whom would you want as a dinner guest? 2. Would you like to be famous? In what way?
Positive
1. For what in your life do you feel most grateful? 2. What is the greatest accomplishment of your life?
Negative
1. If you could change anything about the way you were raised, what would it be? 2. Share an embarrassing moment in your life.
Intensity
1. If you were to die this evening with no opportunity to communicate with anyone, what would you most regret not having told someone? 2. Your house, containing everything you own, catches fire. After saving your loved ones and pets, you have time to safely make a final dash to save any one item. What would it be? Why?
Ending
1. If you were able to live to the age of 90 and retain either the mind or body of a 30-year old for the last 60 years of your life, which would you choose? 2. If you could wake up tomorrow having gained one quality or ability, what would it be?
Table 4.2: Emotion elicitation clips. Movie | Description
---|---
Low Valence, Low Activation (Sad)
City of Angels | Maggie dies in Seth’s arms
Dangerous Minds | Students find that one of their classmates has died
Low Valence, High Activation (Anger)
Sleepers | Sexual abuse of children
Schindler’s List: | Killing of Jews during WWII
High Valence, Low Activation (Contentment)
Wall-E | Two robots dance and fall in love
Love Actually | Surprise orchestra at the wedding
High Valence, High Activation (Amusement)
Benny and Joone | Actor plays the fool in a coffee shop
Something About Mary | Ben Stiller fights with a dog
Neutral
A display of zig-zag lines across the screen
Screen-saver pattern of changing colors

####  4.3.2 Equipment Setup

The modalities considered in our setup are: thermal recordings of the
subject’s face, audio recordings of the subject, color video recording of the
subject’s face, a wide-angle color video recording the subject from the waist
up and physiological sensors measuring skin conductance, breathing rate, heart
rate and skin temperature. For these modalities we have set up the following
equipment:

  1. 1.

FLIR Thermovision A40 thermal camera for recording the close-up thermal
recording of the subject’s face. This camera provides a 640x512 image in the
thermal infrared spectrum.

  2. 2.

Raspberry Pi with camera module V2 with wide-angle lens used for the waist up
shot of the subject. We have chosen Raspberry Pi’s due to its low price and
support for Linux OS, which integrates easily into a generic setup.

  3. 3.

Raspberry Pi with camera module V2 used to record the subject from the waist
up.

  4. 4.

TASCAM DR-100 mk II used to record audio. We chose this product for its high
fidelity. It can record 24-bit audio at 48kHz.

  5. 5.

ProComp∞-8 channel biofeedback and neurofeedback system v6.0 used to measure
blood volume pulse (BVP sensor), skin conductance (SC sensor), skin
temperature (T sensor), and abdominal respiration (BR sensor)

Figure 4.2: Close-up view of the thermal and video recording equipment.

The equipment operator started and marked the synchronization point between
video and audio recordings using a clapper. Subsequent time stamps are
recorded by the qualtrics survey using subject click timings.

####  4.3.3 Post-processing

Splitting of the Recordings. Each modality is split into neutral recordings of
one-minute, five questions and four video recordings with associated
monologues, resulting in fourteen recordings for emotional content, thus 28
recordings per subject. In total we have 784 distinct recordings over five
modalities, 28 subjects and two stress states, for a total of 3920 recording
events. Temperatures are clamped to between $0^{o}$C and $50^{o}$C. This helps
reduce the size of the thermal recording files after being zipped.

Utterance Construction. The five monologues extracted above were divided into
utterances. However, since the monologues are a form of spontaneous speech,
there are no clear sentence boundaries marking end of utterance. We manually
created utterances by identifying prosodic or linguistic boundaries in
spontaneous speech as defined by [125]. The boundaries used for this work are:
(a) clear ending like a full stop or exclamation, (b) a change in context
after filler words or completely revising the sentence to change meaning, or
(c) a very long pause in thought. This method has been previously shown to be
effective in creating utterances that mostly maintain a single level of
emotion [118].

The dataset contains 2,648 utterances with a mean duration of 12.44 $\pm$ 6.72
seconds (Table 4.3). The mean length of stressed utterances ($11.73\pm 5.77$
seconds) is significantly different (using two-sample t-test) from that of the
non-stressed utterances ($13.30\pm 6.73$ seconds). We remove utterances that
are shorter than $3$-seconds and longer than $35$-seconds and end up retaining
$97.2\%$ of our dataset. This allows us to to avoid short segments that may
not have enough information to capture emotion, and longer segments that can
have variable emotion, as mentioned in [118]. Because our dataset is comprised
of spontaneous utterances, the mean length of utterance is larger than those
in a scripted dataset [38] due to more corrections and speech overflow.

Stress State Verification. We perform a paired t-test for subject wise PSS
scores, and find that the mean scores are significantly different for both
sets (16.11 vs 18.53) at $p<0.05$. This implied that our hypothesis of exams
eliciting persistently more stress than normal is often true. In our dataset,
we also provide levels of stress which are binned into three categories based
on weighted average (using questions for which the t-test score was
significant).

###  4.4 Emotional Annotation

####  4.4.1 Crowdsourcing

Crowdsourcing has previously been shown to be an effective and inexpensive
method for obtaining multiple annotations per segment [99, 34]. We posted our
experiments as Human Intelligence Tasks (HITs) on Amazon Mechanical Turk and
used selection and training mechanisms to ensure quality [106]. HITs were
defined as sets of utterances in a monologue. The workers were presented with
a single utterance and were asked to annotate the activation and valence
values of that utterance using Self-Assessment Manikins [30]. Unlike the
strategy adopted in [47], the workers could not go back and revise the
previous estimate of the emotion. We did this to ensure similarity to how a
human listening into the conversation might shift their perception of emotion
in real time. These HITs were presented in either the contextual or the random
presentation condition defined below.

In the contextual experiment, we posted each HIT as a collection of ordered
utterances from each section of a subject’s recording. Because each section’s
question was designed to elicit an emotion, to randomize the carry-over effect
in perception, we posted the HITs in a random order over the sections from all
the subjects in our recording. For example, a worker might see the first HIT
as Utterance 1…N from Section 3 of Subject 4’s stressed recording and see the
second HIT as Utterance 1…M from Section 5 of Subject 10’s non-stressed
recording where N, M are the number of utterances in those sections
respectively. This ensures that the annotator adapts to the topic and
fluctuations in speaking patterns over the monologue being annotated.

In the randomized presentation, each HIT is an utterance from any section, by
any speaker, in random order. So, a worker might see the first HIT as
Utterance 11 from Section 2 of Subject 1’s stressed recording monologue and
see the second HIT as Utterance 1 from Section 5 of Subject 10’s non-stressed
monologue recording. We use this method of randomization to ensure lack of
adaptation to both speaker specific style and the contextual information. The
per-utterance and the contextual labels can be used to train different machine
learning models that are apt for either singular one-off instances or for
holding multiple turn natural conversation, respectively.

Table 4.3: Data summary (R:random, C:context, F:female, M:male). Monologue
Subset
---
Mean no. of utterances/monologue |  $9.69\pm 2.55$
Mean duration of utterances |  $12.44\pm 6.72$ seconds
Total no. of utterances |  2,648
Selected no. of utterances |  2,574
Gender distribution |  19 (M) and 9 (F)
Total annotated speech duration |  $\sim 10$ hours
Crowdsourced Data
Num of workers |  160 (R) and 72 (C)
Blocked workers |  8
Mean activation |  3.62$\pm$0.91 (R)
3.69$\pm$0.81 (C)
Mean valence |  5.26$\pm$0.95 (R)
5.37$\pm$1.00 (C)
|
---|---
Figure 4.3: Distribution of the activation and valence ratings in random
labeling scheme (on left) and contextual labeling scheme (on right).

####  4.4.2 Emotion Content Analysis

We show the distribution of the annotations received in both the random and
contextual setting in Table 4.3 and Figure 4.3. The labels obtained for our
dataset form a distribution that mostly covers negative and neutral levels of
activation, and all but extremities for valence. This can also be seen in the
data summary in Table 4.3. We performed a paired t-test between the labels
obtained from random vs contextual presentation and found that these labels
are significantly different (using paired t-test at $p<0.05$ for both
activation and valence for utterances in the non-stressed situation). Although
the obtained labels are significantly different for valence in the stressed
category using the same method as above, the same does not hold true for the
activation annotations in this category.

Figure 4.4: An overview of the instructions provided to the annotators for
annotating an utterance.

Figure 4.5: Annotation scale used by MTurk workers to annotate the emotional
content of the corpus. They annotate valence and activation for each
utterance.

###  4.5 Experiments

In this section, we describe our baseline experiments for predicting emotion
and stress in the recorded modalities. We have a more granular marked
annotation of emotion, i.e., over each utterance, as compared to stress over
the complete monologue. Hence, we extract features for each modality over
continuous one second frame intervals for predicting stress, and over the
complete utterance for emotion. Audio and lexical features are still extracted
over a complete utterance for stress due to higher interval of variation over
time.

####  4.5.1 Evaluation of Emotion Recognition

We use the following set of features for our baseline models:

  1. 1.

Acoustic Features. We extract acoustic features using OpenSmile [71] with the
eGeMAPS configuration [70]. The eGeMAPS feature set consists of $88$
utterance-level statistics over the low-level descriptors of frequency,
energy, spectral, and cepstral parameters. We perform speaker-level
$z$-normalization on all features.

  2. 2.

Lexical Features. We extract lexical features using Linguistic Inquiry and
Word Count (LIWC) [174]. These features have been shown to be indicative of
stress, emotion, veracity and satisfaction [86, 161, 164]. We normalize all
the frequency counts by the total number of words in the sentence accounting
for the variations due to utterance length.

  3. 3.

Thermal Features. For each subject a set of four regions were selected in the
thermal image: the forehead area, the eyes, the nose and the upper lip as
previously used in [172, 80, 6]. These regions were tracked for the whole
recording and a 150-bin histogram of temperatures was extracted from the four
regions per frame, i.e., 30 frames a second for thermal recordings. We further
reduced the histograms to the first four measures of central tendency, e.g.
Mean, Standard Deviation, Skewness and Kurtosis. We combined these features
over the utterance using first delta measures (min, max, mean, SD) of all the
sixteen extracted measures per frame, resulting in 48 measures in total.

  4. 4.

Close-up Video Features. We use OpenFace [15] to extract the subject’s facial
action units. The AUs used in OpenFace for this purpose are AU1, AU2, AU4,
AU5, AU6, AU7, AU9, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU25, AU26, AU28
and AU25 comprising of eyebrows, eyes and mouth. These features have been
previously shown to be indicative of emotion [227, 64] and have been shown to
be useful for predicting deception [110]. We summarize all frames into a
feature using summary statistics (maximum, minimum, mean, variance, quantiles)
across the frames and across delta between the frames resulting in a total of
144 dimensions.

Network Setup. We train and evaluate multiple unimodal Deep Neural Networks
(DNN) models for predicting valence and activation using Keras [91]. [106]
have shown that a match between the context provided to the classifier and the
annotator leads to better classification performance. Because we are
performing single utterance classification, for all further experiments, we
use the annotations obtained in a random manner as mentioned above. In all
cases, we predict the continuous annotation using regression.

We also use an ensemble of these four networks (audio, lexical, visual and
thermal) to measure multimodal performance. For each network setup, we follow
a five-fold subject independent evaluation scheme and report the average RMSE
across the folds. For each test-fold, we use the previous fold for hyper-
parameter selection and early stopping. The hyper-parameters include: number
of layers $\\{2,3,4\\}$ and layer width $\\{64,128,256\\}$. We use ReLU
activation and train the networks with MSE loss using the Adam optimizer.

We train our networks for a maximum of 50 epochs and monitor the validation
loss after each epoch. We perform early stopping if the loss doesn’t decrease
for 15 consecutive epochs. We save the weights that achieved the lowest
validation performance during training. We train each network five times with
different seeds and average the predictions to account for variations due to
random initialization.

Table 4.4: RMSE for emotion classification models using multiple modalities.
Significance established at $p<0.05$.
| Activation | Valence
---|---|---
Unimodal Models |  |
Acoustic (A) | 1.004∗ | 1.122
Lexical (L) | 1.343 | 0.980
Close Video (V) | 1.111 | 0.879∗∗
Thermal (T) | 2.012 | 1.565
Ensemble |  |
A+L | 0.987 | 0.981
A+V | 0.970 | 0.899
L+V | 0.981 | 0.901
A+L+V | 0.972 | 0.856∗
A+L+V+T (All) | 0.961∗ | 0.868

Results. We show our results in Table 4.4. We find that between acoustic and
lexical modalities, the acoustic modality carries more information about
activation and the lexical for valence. This is in line with previous research
[232, 39]. We also note that the visual modality significantly outperforms
both the speech and lexical modalities for valence prediction.

When we merge these networks using late voting on each modality (decision
fusion), we find that the combination of all modalities performs the best for
predicting activation. But for predicting valence, the best performance is
shown by the combination of acoustic, lexical, visual and thermal modalities.
We believe this is true because previous work has shown that thermal features
are mostly indicative of intensity and discomfort [94] and hence improves
performance on activation prediction, while the visual expressions are most
informative about valence [186].

####  4.5.2 Evaluation of Presence of Stress

We use the following set of features for our baseline models. Given that
stress vs non-stressed state is classified for the complete section (monologue
or neutral recording), we extract visual features differently to use the the
sequential information over the whole segment, i.e., a monologue. We also use
physiological features for our network, since we found that even though they
are highly variable over shorter segments (utterances), they are informative
for recognizing physiological state on a whole section.

  1. 1.

Acoustic, Lexical, and Thermal Features. We use the same features as extracted
for predicting emotion.

  2. 2.

Wide-angle Video Features. We extract the subject’s pose using OpenPose [40,
199, 228] at 25 frames per second. For each frame, we extract 14 three-
dimensional points representing anchor points for the upper body. For
classification of each 3D point is interpolated over one second using a
$5^{th}$ order spline [167, 102]. The parameters of the splines are then used
as features for classification.

  3. 3.

Close-up Video Features. We use OpenFace to extract the subject’s action units
[15]. The features are extracted for every frame. In each frame, features
include the gaze direction vectors, gaze angles, 2D eye region landmarks, head
locations, rotation angles of the head, landmark locations, and facial action
units. Landmarks locations offset by the nose location. We window the data
into segments of one-second windows with 0.5 second overlap and calculate
summary statistics (maximum, minimum, mean, variance). We retain the top 300
features based on the F values between the training features and corresponding
labels (stressed vs non-stressed).

  4. 4.

Physiological Features. While the physiological features varied greatly per
second to be informative for emotion, they are informative for recognizing
presence or absence of stress. We consider the raw measurements for heart
rate, breathing rate, skin conductance and skin temperature and compute the
first four measures of central tendency, e.g. mean, standard deviation,
skewness, and kurtosis.

Table 4.5: Baseline results for classifying stressed and non-stressed
situations per time unit, unless specified otherwise. A - Accuracy, P -
Precision, R - Recall.
Recording Parts | $A$ | $P$ | $R$ | $F_{1}$
---|---|---|---|---
| Thermal
Neutral | 0.61 | 0.67 | 0.62 | 0.64
Questions | 0.50 | 0.64 | 0.52 | 0.57
| Wide-angle Video
Neutral | 0.66 | 0.41 | 0.96 | 0.58
Questions | 0.69 | 0.45 | 0.82 | 0.58
| Close-up Video
Neutral | 0.61 | 0.78 | 0.33 | 0.46
Questions | 0.65 | 0.65 | 0.69 | 0.67
| Physiological
Neutral | 0.66 | 0.47 | 0.89 | 0.64
Questions | 0.70 | 0.55 | 0.88 | 0.67
| Audio - Per utterance
Questions | 0.67 | 0.70 | 0.69 | 0.69
| Text - Per utterance
Questions | 0.60 | 0.74 | 0.61 | 0.67
| Late Fusion - Voting
Questions | 0.60 | 0.74 | 0.61 | 0.67

Network. We train a DNN to perform binary classification, i.e., to recognize
stressed vs. non-stressed situation using ReLU as activation, with softmax as
the classification method.The final layer uses a soft-max activation. We train
six different networks for thermal, wide-angle video, close-up video,
physiological, audio, and lexical modalities. Each network is trained in a
subject-independent manner. We train network to recognize stress vs non-stress
situation in both neutral recording,i.e., when the subject isn’t speaking at
the beginning of the recording, and during emotional monologue questions. To
do so, we decide the final prediction by a majority vote over one-second
predictions for the complete section of the recording. For the lexical and
acoustic modality, we train the network for the question monologues, and
decide the final prediction based on a majority vote over prediction for each
utterance.

Results. We report our results for prediction of stress vs non-stress
situation using various modalities in Table 4.5. We see that the captured
modalities are indeed informative for recognizing stress vs non-stressed
situations. We find that for recognizing this distinction when the subjects
are speaking, audio and physiological features perform the best. This is in
agreement with previous related work [131, 234, 96]. Interestingly, we also
find that the thermal and physiological modality is apt at recognizing
differences in stress, even in the neutral recording, i.e., when the subject
is not speaking. This advantage of thermal modality has been previously
documented by researchers [7, 173, 172, 80]. We find that answering emotional
monologue questions interferes with the recorded thermal modality, leading to
a poorer performance at stress recognition.

###  4.6 Conclusions and Future Work

In this chapter, we introduced a dataset that aims to capture the interplay
between psychological factors such as stress and emotion. While various other
datasets have explored the relationship between gender or personality measures
and emotion production and perception, the relationship between psychological
factors and emotion is understudied from a data collection point of view, and
hence an automated modeling perspective.

We verified that the presence of emotion and stress can be detected in our
dataset. Our baseline results for emotion classification using DNNs with
acoustic, linguistic and visual features on our dataset are similar to
reported results on other datasets such as IEMOCAP [36] and MSP-Improv [38].

##  Chapter V Best Practices for Noise Based Augmentation in Emotion Datasets

###  5.1 Motivation and Contributions

Speech emotion recognition is an important component of any human centered
system. But speech characteristics produced and perceived by a person can be
influenced by a multitude of reasons, both desirable such as emotion, and
undesirable such as noise. To train robust emotion recognition models, we need
a large, yet realistic data distribution, but emotion datasets are often small
and hence are augmented with noise. Often noise augmentation makes one
important assumption, that the prediction label should remain the same in
presence or absence of noise, which is true for automatic speech recognition
but not necessarily true for perception based tasks. In this chapter we make
three novel contributions. We validate through crowdsourcing that the presence
of noise does change the annotation label and hence may alter the original
ground truth label. We then show how disregarding this knowledge and assuming
consistency in ground truth labels propagates to downstream evaluation of ML
models, both for performance evaluation and robustness testing. We end the
chapter with a set of recommendations for noise augmentations in speech
emotion recognition datasets.

###  5.2 Introduction

Speech emotion recognition is increasingly included as a component in many
real-world human-centered machine learning models. Modulations in speech can
be produced for a multitude of reasons, both desirable and undesirable. In our
case desirable modulations encode information that we want our model to learn
and be informed by, such as speaker characteristics or emotion. Undesirable
modulations encode information that are extrinsic factors change with the
environment, such as noise. In order to handle these modulations, we need
large datasets that capture the range of possible speech variations and their
relationship to emotion expression. But, such datasets are generally not
available for emotion tasks. To bridge this gap, researchers have proposed
various methods to generate larger datasets. One of the most common is noise
augmentation. The baseline assumption of noise augmentation is that the labels
of the emotion examples do not change once noise has been added [169]. While
this assumption can be confidently made for tasks such as automatic speech
recognition (ASR), the same cannot be said for perception-based tasks, such as
emotion recognition.

In this chapter, we question the assumption that the annotation label remains
the same in the presence of noise. We first create a noise augmented dataset
and conduct a perception study to label the emotion of these augmented
samples, focused on the type of noise in samples whose perception has changed
or remained the same given the agumentation. We use the results from this
study to classify the complete set of augmentation noises into two categories,
perception-altering (i.e., noises that may change the perception of emotion)
and perception-retaining (i.e., noises that do not change the perception of
emotion). We propose that the perception-altering noises should not be used in
supervised learning or evaluation frameworks because we cannot confidently
maintain that the original annotation holds for a given sample. We evaluate
the effects of disregarding emotion perception changes by examining how the
performance of emotion recognition models and analyses of their robustness
change in unpredictable manners when we include samples that alter human
perception in the training of these models. Lastly, we provide a set of
recommendations for noise based augmentation of speech emotion recognition
datasets based on our results.

Researchers have considered the impact of noise on emotion perception and
thereby the annotation of emotions. [X] looked at how pink and white noises in
varying intensities change the perception of emotion. Another set of research
has concentrated on training and validating noise robust models with the
assumption that intent label prediction remains consistent in the presence of
noise. For example, [X] have looked at training student teacher models that
aim to ignore the effect of noise introduced to the model. On the other hand
[X] have proposed copy pasting various emotion segments together along with
neutral noise to balance the classes in an emotion dataset, thus improving
performance.

In this chapter, we claim that the standard assumption about perception and
hence, label retention of emotion in the presence of noise may not hold true
in a multiple noise categories. To understand which noises impact emotion
perception, we use a common emotion dataset, IEMOCAP and introduce various
kinds of noises to it, at varying signal to noise ratio (SNR) levels as well
as at different positions in the sample. We then perform a crowdsourcing
experiment that asks workers to annotate their perception of emotion for both
the clean and the corresponding noise-augmented sample. This enables us to
divide noise augmentation options into groups characterized by their potential
to either influence or not influence human perception.

The results of the crowdsourcing experiments inform a series of empircal
analyses focused on model performance and model robustness. We first present
an empirical evaluation of the effects of including perception-altering noises
in training. It will allow us to observe how the inclusion of perception-
altering noises creates an impression of performance improvement. We will
discuss how this improvement is a myth, this new model will have learned to
predict labels that are not truly associated with a given sample due to the
perceptual effects of these noises. We consider both a general recurrent
neural network (RNN) model and an end-to-end model for this purpose. We
evaluate conditions in which novel augmentation noises are either introduced
during training (matched) or seen for the first time during testing
(mismatched). The second empirical evaluation analyzes whether the gap in
performance between the matched and mismatched conditions can be bridged using
noise robust modeling techniques. The third and final evaluation is focused on
the robustness of the model. It will allow us to observe how the inclusion of
these perception altering noises ultimately leads to a model that is more
susceptible to attack compared to a model that does not include these noises.
We train an attack model for robustness testing. It considers a pool of noises
and picks the best noise with a minimal SNR degradation that is able to change
a model’s prediction. We consider a condition in which the attack model has
black-box access to the trained model. The attack has a fixed number of
allowed queries to the trained model, but not the internal gradients or
structure (i.e., the attack model can only provide input and can only access
the trained model’s prediction). We test and monitor the difference in the
observed robustness of these aforementioned models.

We find that the crowdsourced labels do change in the presence of some kinds
of noise. We then verify that the models perform worse on noisy samples when
trained only on clean datasets. But, we show that this decrease in performance
is different when using the complete set of noises for augmenting the test set
vs. when only using the perception-retaining noises for augmentation. We show
similar patterns for noise-robust models, specifically showing how there is an
increased drop in performance for the end-to-end noise-robust model when
excluding performance-altering noises during augmentation. We then discuss how
our conventional metrics, those that look only at model performance, may be
incorrectly asserting improvements as the model is learning to predict an
emotion measure that is not in line with human perception. Troublingly, we
find that the attack model is generally more effective when it has access to
the set of all noises as compared to when excluding perception-altering noises
for allowed augmentations. We also specifically find that given just a pool of
carefully crafted reverberation modulations, the attack model can be
successful in almost 65% of the cases with minimal degradation in SNR and in
less than ten queries to the trained model. We end the chapter with a general
set of recommendations for noise augmentations in speech emotion recognition
datasets.

###  5.3 Research Questions

In this chapter, we investigate five research questions:

Purpose 1: Premise Validation through Crowdsourcing
RQ1:  Does the presence of noise affect emotion perception as evaluated by
_human raters_? Is this effect dependent on the utterance length, loudness,
the type of the added noise, and the original emotion?
Reason: Noise has been known to have masking effect on humans in specific
situations. Hence, humans can often understand verbalized content even in
presence of noise. Our goal is to understand whether the same masking effect
extends to paralinguistic cues such as emotion, and to what extent. Our
continuing claim from hereon remains that only noises that do not change human
perception should be used for the training and evaluation of machine learning
models. Not doing so, can lead to gains or drops in performance measurement
that may not actually extend to real world settings. We call these changes
”unverified” because we cannot, with certainity, be sure that the model should
have predicted the original label (i.e., the label of the sample before noise
was added) because the human did not neccessarily label the noisy instance
with that same label.

Purpose 2: Noise Impact Quantification
RQ2:  Can we verify previous findings that the presence of noise affects the
performance of _emotion recognition models_? Does this effect vary based on
the type of the added noise?
Reason: We have known that presence of noise in data shifts the data
distribution [50]. This shift often leads to poor performance by machine
learning models. We aim to quantify the amount of performance drop based on
the type of noise in these systems, both, for any kind of noise, and then,
specifically for noises that do not change human perception (perception-
retaining).

Purpose 3: Denoising and Augmentation Benefits Evaluation
RQ3:  Does dataset augmentation (Q3a) and/or sample denoising (Q3b) help
improve the robustness of emotion recognition models to unseen noise?
Reason: We test whether the commonly-used methods for improving the
performance of these models under distribution shifts is helpful. We focus on
two main methods, augmentation and denoising. We specifically look at how
performance changes when we augment with noises that include those that are
perception-altering vs. when we exclude such noises.

Purpose 4: Model Robustness Testing Conditions
RQ4:  How does the robustness of a model to attacks compare when we are using
test samples that with are augmented with perception-retaining noise vs.
samples that are augmented with all types of noise, regardless of their effect
on perception?
Reason:  Another major metric for any deployable machine learning algorithm is
its performance on ”unseen situations” or handling incoming data shifts (i.e.,
robustness testing). We test robustness using a noise augmentation algorithm
that aims to forcefully and efficiently change a model’s output by augmenting
test samples with noise. We look at how often this algorithm is unsuccessful
in being able to ”fool” a model with its augmented samples. We look at the
changes in frequency with which a model is successfully able to defend itself
when the attack algorithm uses a set that includes all types of noises vs.
when it only uses perception-retaining noises.

Purpose 5: Recommendations
RQ5:  What are the recommended practices for speech emotion dataset
augmentation and model deployment?
Reason: We then provide a set of recommendations based on our empirical
studies for deploying emotion recognition models in real world situations.

###  5.4 Noise

We investigate the effects of two types of noise, environmental and signal
distortion. Environmental noises are additive, while signal distortion noise
involves other types of signal manipulation.

####  5.4.1 Environmental Noise

We define environmental noises (ENV) as additive background noise, obtained
from the ESC-50 dataset[176]111https://github.com/karoldvl/ESC-50. ESC-50 is
generally used for noise contamination and environmental sound classification
[231]. These environmental sounds are representative of many types of noise
seen in real world deployments, especially in the context of virtual and smart
home conversational agents. We use the following categories:

  * •

Natural soundscapes (Nat), e.g., rain, wind.

  * •

Human, non-speech sounds (Hum), e.g., sneezing, coughing, laughing or crying
in the background etc.

  * •

Interior/domestic sounds (Int), e.g., door creaks, clock ticks etc.

We manipulate three factors when adding the noise sources:

  * •

Position: The position of the introduction of sound that: (i) starts and then
fades out in loudness or (ii) occurs during the entirety of the duration of
the utterance. In the second case, this complete additive background would
represent a consistent noise source in real world (e.g., fan rotation).

  * •

Quality Degradation: The decrease in the signal to noise ratio (SNR) caused by
the addition of the additive background noise at levels of 20dB, 10dB and 0dB.
This is used only when noise is added to the entirety of the utterance.

####  5.4.2 Signal Distortion

We define signal distortion noise as modulations that aren’t additive in the
background. These kinds of noise in the audio signal can occur from
linguistic/paralinguistic factors, room environment, internet lags, or the
physical locomotion of the speaker.

We use the nine following categories:

  * •

SpeedUtt: The utterance is sped up by either 1.25$\times$ or 0.75$\times$.

  * •

SpeedSeg: A random segment within an utterance is sped up by 1.25$\times$. The
package pyAudio 222https://people.csail.mit.edu/hubert/pyaudio/ that we used
to speed up a segment did not permit slowing a segment down. Thus, the
0.75$\times$ was not used here.

  * •

Fade: The loudness of the utterance is faded by 2% every second, which
emulates the scenario of a user moving away from the speaker. The loudness is
increased for fade in, and decreased for fade out.

  * •

Filler: Non-verbal short fillers such as ‘uh’, ‘umm’ (from the same speaker)
are inserted in the middle of a sentence. The insertion is either just the
filler or succeeded and preceded by a long pause 333Fillers are obtained by
parsing audio files for a given speaker and finding occurrences of any of the
options from the above mentioned set. We will release the extracted fillers
per speaker for IEMOCAP.

  * •

DropWord: A randomly selected set of non-essential words belonging to the set:
{a, the, an, so, like, and} are dropped from an utterance using word-aligned
boundaries and stiching the audio segments together.

  * •

DropLetters: Following the same approach as drop word, letters are dropped in
accordance with various linguistic styles chosen from the set: {/h/+vowel,
vowel+/nd/+consonant(next word), consonant+/t/+consonant(next word),
vowel+/r/+consonant, /ihng/}. This is supported by research that has studied
phonological deletion or dropping of letters in the native US-English dialect
[1, 237].

  * •

Laugh/Cry: “Sob” and “short-laughter” sounds are added to the utterance. They
are obtained from AudioSet [82].

  * •

Pitch: The pitch is changed by $\pm$ 3 half octaves using the pyAudio library.

  * •

Rev: Room reverberation is added to the utterance using py-audio-effects
(pysndfx444https://github.com/carlthome/python-audio-effects ). We vary
metrics such as reverberation ratio or room size to vary the type and
intensity of reverberation added.

####  5.4.3 Sampling and Noise-Perturbations

We randomly select 900 samples from the IEMOCAP dataset, which is far larger
than the ones used for previous perception studies [170, 191]. We select 100
samples from each activation and valence pair bin, i.e., 100 samples from the
bin with activation: _low_ , valence: _low_ ; 100 samples from the bin with
activation: _low_ , and valence: _mid_ , and so on. This ensures that the
chosen 900 samples cover the range of emotions expressed. We impose another
constraint on these 100 samples from each bin, 30 of them are shorter than the
first quartile or greater than fourth quartile of utterance length in seconds
to cover both extremities of the spectrum, and the remaining 70 belong in the
middle. We also ensure that the selected samples had a 50-50 even split
amongst gender. We introduce noise to the 900 samples (Section 3.1). Each
sample is modulated in ten ways: four randomly chosen types of environmental
noise and six randomly chosen signal distortion noise modulations, giving us a
total of 9,000 noisy samples555We will release the script to create these
files..

###  5.5 User study

We first analyze the effects of noise on human perception by relabeling the
noise-enhanced data using the Amazon Mechanical Turk (AMT) platform. We use
insights from this experiment to guide the machine learning analyses that
follow.

####  5.5.1 Crowdsourcing Setup

We recruited 147 workers using Amazon Mechanical Turk who self-identify as
being from the United States and as native English speakers, to reduce the
impact of cultural variability. We ensured that each worker had $>98$%
approval rating and more than 500 approved Human Intelligence Tasks (HITs). We
ensured that all workers understood the meaning of activation and valence
using a qualification task that asked workers to rank emotion content similar
to [105]. The qualification task has two parts: (i) we explain the difference
between valence and activation and how to identify those, and, (ii) we ask
them to identify which of the two samples has a higher/lower valence and a
higher/lower activation, to ensure that they have understood the concept of
activation and valence annotations. All HIT workers were paid a minimum wage
($\$9.45/$hr), pro-rated to the minute. Each HIT was annotated by three
workers.

For our main task, we created pairs that contained one original and one
modulated sample. We then asked each worker to annotate whether or not they
perceived the pair to have the same emotion. If they said yes for both
activation and valence, the noisy sample was labeled same and they could
directly move to the next HIT. If they said no, the noisy sample was labeled
different. In this case, they were asked to assess the activation and valence
of the noisy sample using Self Assessment Manikins [30] on a scale of [1, 5]
(similar to the original IEMOCAP annotation).

We also include three kinds of attention checks:

  1. 1.

We show two samples that have not been modified and ask them to decide if the
emotion represented was different. If the person says yes, then the experiment
ends.

  2. 2.

We observe the time spent on the task. If the time spent on the task is less
than the combined length of both samples, then the user’s qualification to
annotate the HITs is rescinded and their responses are discarded.

  3. 3.

We show two samples, one which has a gold standard label, and another, which
has been contaminated with significant noise (performance degradation
$>$30dB), such that the resulting sample is incomprehensible. If people do not
mark this set of samples as being different, the experiment ends.

The failure rate based on the above criteria was 8%. We ensured the quality of
the annotations by paying bonuses based on time spent, not just number of
HITs, and by disqualifying annotators if they annotated any sample (including
those outside of the attention checks) more quickly than the combined length
of the audio samples.

We then created two sets of labels for each noise-augmented clip. The _first
type_ of label compared a noise-augmented clip to its original. The noise-
augmented clip was labeled the _same_ if the modified and original clip were
perceived to have the same valence or activation, otherwise it was labeled
_different_. We created this label by taking the majority vote over all
evaluations. The _second type_ of label included valence and activation. A
noise-augmented clip was given the average valence and activation over all
evaluations.

The inter-annotator agreement was measured using Cohen’s kappa.
Conventionally, when estimating Cohen’s kappa, annotators are not considered
as individuals, instead reducing annotators to the generic 1, 2, and 3. The
challenge is that this often leads to artificially inflated inter-annotator
agreement because individual characteristics and behavior of a particular
worker are not taken under consideration [95]. We take a different approach,
creating a table for the calculation of the statistic that considers
annotators as individuals with separate entries for each clip, following the
approach of [95]. If an annotator didn’t evaluate a given clip, the cell has a
null (missing data) value. We found that the Cohen’s kappa was 79% for
activation and 76% for valence 666The sample name, code to create the paired
noisy examples, and the resulting annotations will be made available for
further research.

Randomly sample 1 noise variation from each category mentioned in Section
5.4.;

$numAttempts=0$;

for  _each noise in selected random noises:_ do

Add noise to the sample such that the decrease in SNR is 1.;

Get the classifier output with this new sample variation.;

$numAttempts+=1$;

if  _$numAttempts >k$_ then

return _Exit Code = Failure_

end if

if  _classifier output changes_ then

return _Exit Code = Success_

end if

end for

for  _each noise in selected random noises:_ do

Add noise to the sample such that the decrease in SNR is 5.;

Get the classifier output with this new sample variation.;

$numAttempts+=1$;

if  _$numAttempts >k$_ then

return _Exit Code = Failure_

end if

if  _classifier output changes_ then

return _Exit Code = Success_

end if

if  _classifier output changes_ then

while  _classifier output does not change_ do

Iterate over all SNR decreases from 2-5;

Get classifier output for the modified sample;

$numAttempts+=1$;

if  _$numAttempts >k$_ then

return _Exit Code = Failure_

end if

if  _classifier output changes_ then

return _Exit Code = Success_

end if

end while

end if

end for

for  _each noise in selected random noises:_ do

Add noise to the sample such that the decrease in SNR is 10.;

Get the classifier output with this new sample variation.;

if  _classifier output changes_ then

return _Exit Code = Success_

end if

if  _classifier output changes_ then

while  _classifier output does not change_ do

Iterate over all SNR decreases from 6-10 Get classifier output for the
modified sample;

$numAttempts+=1$;

if  _$numAttempts >k$_ then

return _Exit Code = Failure_

end if

if  _classifier output changes_ then

return _Exit Code = Success_

end if

end while

end if

end for

return _Exit Code = Failure_

Algorithm 1 Pseudo-code for testing model robustness. Exit Code is Success when the algorithm finds a noise-augmented version of the sample that the model changes prediction for. Exit Code is Failure when the model maintains its predictions over the any of the noise-augmented versions tried. Table 5.1: The table shows the ratio of samples marked by human evaluators as imperceptible to difference in emotion perception. V: Valence, A: Activation, $\delta$V:Average change in Valence on addition of noise, $\delta$A:Average change in Activation on addition of noise. | V | A |  $\delta$V |  $\delta$A
---|---|---|---|---
Environmental Noise
NatSt |  | 0.01 | 0.00 |  |
NatdB (Co) | -10dB | 0.01 | 0.00 |  |
| Same | 0.02 | 0.00 |  |
| +10dB | 0.03 | 0.01 |  |
HumSt |  | 0.01 | 0.00 |  |
HumdB (Co) | -10dB | 0.03 | 0.00 |  |
| Same | 0.02 | 0.00 |  |
| +10dB | 0.04 | 0.01 |  |
IntSt |  | 0.05 | 0.01 |  |
IntdB (Co) | -10dB | 0.02 | 0.01 |  |
| Same | 0.02 | 0.00 |  |
| +10dB | 0.04 | 0.01 |  |
Signal Distortion
SpeedSeg |  | 0.01 | 0.0 |  |
Fade | In | 0.04 | 0.01 |  |
| Out | 0.04 | 0.00 |  |
DropWord |  | 0.01 | 0.00 |  |
DropLetters |  | 0.01 | 0.00 |  |
Reverb |  | 0.04 | 0.01 |  |
Filler | L | 0.10 | 0.06 |  |
| S | 0.06 | 0.03 |  |
Laugh |  | 0.16 | 0.17 | \+ .11 | \+ .26
Cry |  | 0.20 | 0.22 | \- .20 | \- .43
SpeedUtt | 1.25x | 0.13 | 0.03 | \- .10 | \- .13
| 0.75x | 0.28 | 0.06 | \- .18 | \- .23
Pitch | 1.25x | 0.22 | 0.07 | \- .11 | \+ .19
| 0.75x | 0.29 | 0.10 | \- .07 | \- .15
Table 5.2: State of the art model performance in terms of UAR when using the general versions of traditional deep learning and end-to-end deep learning models. No noise refers to clean speech, all noises refers to the combined set of perception-retaining and perception-altering noise. The environmental and signal distortion categories shown include only the perception-retaining noises. As a reminder, samples in the all noises category have an uncertain ground truth, the row is marked with two stars ($\ast\ast$). V: Valence, A: Activation, Clean: Training on clean dataset, Clean$+$Noise — Mismatch: Cleaning on noisy dataset and testing on mismatched noisy partition, Clean$+$Noise — Match: Cleaning on noisy dataset and testing on matched noisy partition. Random chance UAR is 0.33.  |  |  |  |  | Traditional Deep Neural Network |  | End-To-End Deep Neural Network
---|---|---|---|---|---|---|---
|  |  |  |  | Clean |  | Clean+Noise |  | Clean |  | Clean+Noise
|  |  |  |  |  | Mismatch | Match |  |  | Mismatch | Match
|  |  |  |  | A | V |  | A | V | A | V |  | A | V |  | A | V | A | V
No Noise |  | 0.67 | 0.59 |  | - | - | - | - |  | 0.70 | 0.63 |  | - | - | - | -
All Noises** |  | 0.40 | 0.38 |  | 0.55 | 0.42 | 0.66 | 0.59 |  | 0.70 | 0.63 |  | 0.44 | 0.38 | 0.67 | 0.60
Perception Retaining Noises |  | 0.50 | 0.42 |  | 0.57 | 0.48 | 0.60 | 0.52 |  | 0.53 | 0.45 |  | 0.60 | 0.50 | 0.62 | 0.54
Environmental Category | Nature | At Start |  |  | 0.50 | 0.45 |  | 0.61 | 0.53 | 0.63 | 0.55 |  | 0.56 | 0.48 |  | 0.64 | 0.55 | 0.66 | 0.58
|  Cont. | -5dB |  | 0.45 | 0.39 |  | 0.55 | 0.44 | 0.59 | 0.50 |  | 0.49 | 0.42 |  | 0.58 | 0.47 | 0.62 | 0.53
| -10dB |  | 0.42 | 0.35 |  | 0.56 | 0.46 | 0.59 | 0.49 |  | 0.47 | 0.38 |  | 0.57 | 0.48 | 0.61 | 0.51
| -20dB |  | 0.40 | 0.35 |  | 0.51 | 0.44 | 0.55 | 0.47 |  | 0.47 | 0.39 |  | 0.52 | 0.46 | 0.56 | 0.50
Interior | At Start |  |  | 0.53 | 0.44 |  | 0.61 | 0.52 | 0.64 | 0.57 |  | 0.57 | 0.47 |  | 0.64 | 0.56 | 0.66 | 0.59
|  Cont | -5dB |  | 0.46 | 0.36 |  | 0.55 | 0.44 | 0.58 | 0.49 |  | 0.49 | 0.39 |  | 0.59 | 0.49 | 0.62 | 0.51
| -10dB |  | 0.44 | 0.36 |  | 0.54 | 0.43 | 0.57 | 0.48 |  | 0.49 | 0.39 |  | 0.56 | 0.45 | 0.59 | 0.52
| -20dB |  | 0.40 | 0.35 |  | 0.52 | 0.44 | 0.55 | 0.49 |  | 0.46 | 0.37 |  | 0.56 | 0.45 | 0.58 | 0.51
Human | At Start |  |  | 0.52 | 0.45 |  | 0.60 | 0.51 | 0.63 | 0.55 |  | 0.58 | 0.47 |  | 0.62 | 0.52 | 0.66 | 0.57
|  Cont | -5dB |  | 0.45 | 0.37 |  | 0.52 | 0.43 | 0.55 | 0.48 |  | 0.49 | 0.40 |  | 0.56 | 0.44 | 0.57 | 0.50
| -10dB |  | 0.42 | 0.34 |  | 0.51 | 0.43 | 0.53 | 0.47 |  | 0.49 | 0.38 |  | 0.53 | 0.45 | 0.55 | 0.49
| -20dB |  | 0.40 | 0.34 |  | 0.50 | 0.41 | 0.53 | 0.46 |  | 0.46 | 0.38 |  | 0.54 | 0.43 | 0.56 |  0.48
Signal Distortion | Speed Segment |  | 0.61 | 0.52 |  | 0.63 | 0.53 | 0.64 | 0.55 |  | 0.63 | 0.55 |  | 0.64 | 0.55 | 0.67 | 0.58
Fade | In |  | 0.62 | 0.53 |  | 0.65 | 0.55 | 0.67 | 0.58 |  | 0.64 | 0.55 |  | 0.67 | 0.58 | 0.68 | 0.59
|  | Out |  | 0.61 | 0.51 |  | 0.62 | 0.54 | 0.64 | 0.57 |  | 0.63 | 0.54 |  | 0.64 | 0.56 | 0.66 | 0.59
DropWord |  | 0.64 | 0.56 |  | 0.65 | 0.56 | 0.67 | 0.59 |  | 0.65 | 0.58 |  | 0.67 | 0.58 | 0.69 | 0.61
DropLetters |  | 0.65 | 0.58 |  | 0.69 | 0.60 | 0.71 | 0.62 |  | 0.66 | 0.59 |  | 0.72 | 0.60 | 0.74 | 0.63
Reverb |  | 0.43 | 0.37 |  | 0.50 | 0.43 | 0.53 | 0.45 |  | 0.35 | 0.34 |  | 0.51 | 0.42 | 0.55 |  0.46
Table 5.3: Noise-Robust (NR) state of the art model performance in terms of UAR when using the noise-robust versions of traditional deep learning and end-to-end deep learning models. No noise refers to clean speech, all noises refers to the combined set of perception-retaining and perception-altering noise. The environmental and signal distortion categories shown include only the perception-retaining noises. As a reminder, samples in the all noises category have an uncertain ground truth, the row is marked with two stars ($\ast\ast$). V: Valence, A: Activation, Clean: Training on clean dataset, Clean$+$Noise — Mismatch: Cleaning on noisy dataset and testing on mismatched noisy partition, Clean$+$Noise — Match: Cleaning on noisy dataset and testing on matched noisy partition, NR: Noise Robust versions of the corresponding models Random chance UAR is 0.33.  |  |  |  |  | NR-Traditional Deep Neural Network |  | NR-End-To-End Deep Neural Network
---|---|---|---|---|---|---|---
|  |  |  |  | Clean |  | Clean+Noise |  | Clean |  | Clean+Noise
|  |  |  |  |  | Mismatch | Match |  |  | Mismatch | Match
|  |  |  |  | A | V |  | A | V | A | V |  | A | V |  | A | V | A | V
No Noise |  | 0.67 | 0.59 |  | - | - | - | - |  | 0.70 | 0.63 |  | - | - | - | -
All Noises** |  | 0.44 | 0.40 |  | 0.58 | 0.44 | 0.68 | 0.60 |  | 0.50 | 0.40 |  | 0.50 | 0.40 | 0.72 | 0.61
Perception Retaining Noises |  | 0.52 | 0.44 |  | 0.59 | 0.50 | 0.61 | 0.51 |  | 0.55 | 0.48 |  | 0.61 | 0.52 | 0.63 | 0.54
Environmental Category | Nature | At Start |  |  | 0.54 | 0.49 |  | 0.63 | 0.55 | 0.63 | 0.55 |  | 0.57 | 0.50 |  | 0.66 | 0.55 | 0.67 | 0.56
|  Cont. | -5dB |  | 0.50 | 0.42 |  | 0.58 | 0.50 | 0.60 | 0.52 |  | 0.49 | 0.42 |  | 0.61 | 0.51 | 0.63 | 0.53
| -10dB |  | 0.48 | 0.38 |  | 0.59 | 0.48 | 0.61 | 0.51 |  | 0.50 | 0.46 |  | 0.63 | 0.52 | 0.65 | 0.53
| -20dB |  | 0.44 | 0.38 |  | 0.55 | 0.49 | 0.59 | 0.51 |  | 0.50 | 0.43 |  | 0.53 | 0.46 | 0.58 | 0.47
Interior | At Start |  |  | 0.53 | 0.44 |  | 0.61 | 0.52 | 0.65 | 0.54 |  | 0.58 | 0.52 |  | 0.63 | 0.56 | 0.66 | 0.58
|  Cont | -5dB |  | 0.46 | 0.36 |  | 0.55 | 0.44 | 0.59 | 0.47 |  | 0.51 | 0.42 |  | 0.58 | 0.48 | 0.62 | 0.52
| -10dB |  | 0.44 | 0.36 |  | 0.54 | 0.43 | 0.57 | 0.43 |  | 0.49 | 0.44 |  | 0.57 | 0.48 | 0.61 | 0.50
| -20dB |  | 0.40 | 0.35 |  | 0.52 | 0.44 | 0.55 | 0.46 |  | 0.46 | 0.40 |  | 0.55 | 0.48 | 0.58 | 0.49
Human | At Start |  |  | 0.52 | 0.45 |  | 0.60 | 0.51 | 0.63 | 0.52 |  | 0.59 | 0.49 |  | 0.63 | 0.53 | 0.66 | 0.56
|  Cont | -5dB |  | 0.45 | 0.37 |  | 0.52 | 0.43 | 0.55 | 0.46 |  | 0.49 | 0.42 |  | 0.56 | 0.48 | 0.58 | 0.50
| -10dB |  | 0.42 | 0.34 |  | 0.51 | 0.43 | 0.54 | 0.44 |  | 0.47 | 0.38 |  | 0.54 | 0.49 | 0.55 | 0.45
| -20dB |  | 0.40 | 0.34 |  | 0.50 | 0.41 | 0.52 | 0.43 |  | 0.43 | 0.38 |  | 0.54 | 0.45 | 0.58 | 0.49
Signal Distortion | Speed Segment |  | 0.63 | 0.55 |  | 0.65 | 0.57 | 0.67 | 0.58 |  | 0.66 | 0.58 |  | 0.67 | 0.59 | 0.67 | 0.60
Fade | In |  | 0.64 | 0.55 |  | 0.66 | 0.57 | 0.68 | 0.59 |  | 0.65 | 0.58 |  | 0.67 | 0.59 | 0.69 | 0.60
|  | Out |  | 0.64 | 0.56 |  | 0.65 | 0.57 | 0.67 | 0.58 |  | 0.66 | 0.57 |  | 0.68 | 0.59 | 0.69 | 0.63
DropWord |  | 0.67 | 0.60 |  | 0.66 | 0.58 | 0.66 | 0.59 |  | 0.68 | 0.60 |  | 0.69 | 0.60 | 0.69 | 0.60
DropLetters |  | 0.67 | 0.60 |  | 0.65 | 0.62 | 0.66 | 0.60 |  | 0.68 | 0.64 |  | 0.69 | 0.60 | 0.69 | 0.60
Reverb |  | 0.48 | 0.41 |  | 0.56 | 0.47 | 0.58 | 0.48 |  | 0.52 | 0.40 |  | 0.55 | 0.45 | 0.60 | 0.45
Table 5.4: Success of misclassification attempts on different models with varying number of allowed attempts (lower is better). As a reminder, samples in the all noises category have an uncertain ground truth, the row is marked with two stars ($\ast\ast$). Reverberation (reverb) is a perception-retaining noise that is also analyzed separately. Trad: Traditional Deep Learning Model, E2E: End to End deep learning model, NR: Noise Robust version of the deep learning model. |  Noise Set |  No. of Attempts | Activation | Valence
---|---|---|---|---
| Trad | E2E | NR-Trad | NR-E2E | Trad | E2E | NR-Trad | NR-E2E
Performance Impact Per Noise Is Unknown |  All Noises** | 5 | 0.29 | 0.22 | 0.15 | 0.10 | 0.11 | 0.15 | 0.13 | 0.05
15 | 0.31 | 0.33 | 0.31 | 0.32 | 0.23 | 0.24 | 0.22 | 0.22
25 | 0.40 | 0.28 | 0.40 | 0.33 | 0.22 | 0.19 | 0.21 | 0.20
inf | 0.43 | 0.41 | 0.44 | 0.35 | 0.25 | 0.20 | 0.26 | 0.18
Perception-Retaining | 5 | 0.18 | 0.11 | 0.07 | 0.05 | 0.11 | 0.05 | 0.02 | 0.02
15 | 0.25 | 0.12 | 0.25 | 0.15 | 0.14 | 0.08 | 0.18 | 0.10
25 | 0.32 | 0.24 | 0.32 | 0.20 | 0.13 | 0.10 | 0.11 | 0.09
inf | 0.40 | 0.26 | 0.36 | 0.23 | 0.19 | 0.14 | 0.17 | 0.14
Reverb | 5 | 0.33 | 0.15 | 0.22 | 0.18 | 0.20 | 0.14 | 0.22 | 0.09
15 | 0.40 | 0.23 | 0.30 | 0.21 | 0.30 | 0.13 | 0.34 |  0.16
Performance Impact Per Noise Is Known |  All Noises** | 5 | 0.33 | 0.28 | 0.24 | 0.22 | 0.15 | 0.14 | 0.12 | 0.12
15 | 0.38 | 0.38 | 0.32 | 0.33 | 0.20 | 0.21 | 0.18 | 0.16
25 | 0.52 | 0.32 | 0.44 | 0.37 | 0.25 | 0.16 | 0.18 | 0.16
inf | 0.54 | 0.42 | 0.46 | 0.41 | 0.24 | 0.20 | 0.22 | 0.22
Perception-Retaining | 5 | 0.29 | 0.16 | 0.22 | 0.13 | 0.14 | 0.10 | 0.15 | 0.08
15 | 0.32 | 0.32 | 0.32 | 0.28 | 0.14 | 0.15 | 0.16 | 0.12
25 | 0.47 | 0.30 | 0.47 | 0.29 | 0.22 | 0.18 | 0.23 | 0.16
inf | 0.51 | 0.36 | 0.50 | 0.32 | 0.22 | 0.19 | 0.25 | 0.17
Reverb | 5 | 0.38 | 0.22 | 0.33 | 0.22 | 0.28 | 0.20 | 0.31 | 0.21
15 | 0.47 | 0.29 | 0.41 | 0.28 | 0.30 | 0.23 | 0.35 |  0.26

###  5.6 Methods

Table 5.5: Hyper-parameters used to select the best performing model on validation subset whilst training the traditional deep learning model. Hyper-parameter | Values
---|---
Traditional |
No. of Convolution Kernels | {64, 128}
Convolution Kernels Width | {2}
Number of Convolution Layers | {5}
Number of GRU layers | {1, 2, 3}
Pooling Kernel Width | {2, 4}
GRU Layers Width | {32, 64}
Number of Dense Layers | {1, 2, 3}
End to End |
No. of Dense Layers | {1, 2}

We now describe the emotion recognition approaches, presenting two separate
pipelines, one that relies upon direct feature extraction (Section 5.6.2) and
the other that is end-to-end (Section 5.6.3). This allows us to investigate
whether noise has a consistent effect. We discuss approaches to improve noise
robustness by training models with noise-augmented data or denoised data
(Section 5.6.4). Finally, we describe the setup and evaluation of the model
robustness using an untargeted model misclassification test, which measures a
model’s fragility in terms of how likely it is that the model’s decisions will
change when specific types of noise are observed at test time (Section 5.6.5).

####  5.6.1 Creation of Data Partitions

We use a subject-independent five-fold cross validation scheme to select our
train, test and validation sets. In the first iteration, sessions 1-3 are used
for training, session 4 is used as validation, and session 5 is used for
testing. This is repeated in a round-robin fashion, resulting in each session
serving as a validation and a test fold. We also divide possible noises in two
different categories based on results of crowdsourcing study (see Section
5.7.1). The first category is _perception-altering_ , those that changed
perception of humans and hence cannot be used for model training or evaluation
with the old annotations. The second category is _perception-retraining_ ,
those that did not change human perception, and hence, the model should
produce no change in predictions when using those noise categories for sample
augmentation.

We use the noise categories (seeSection 5.4) in two varying circumstances. The
first category is matched, where both the training and testing sets are
augmented with same kinds of noise (e.g., both have nature-based sounds in
them). The second category is mismatched, where the testing set is augmented
with a noise _category_ not used for augmenting the training set (e.g., only
the test set is augmented with nature-based noise while the train set is
augmented with human or interior noises).

####  5.6.2 Traditional Deep Learning Network

We first explore a common “traditional” deep learning network that is used in
speech emotion recognition. In this method we extract Mel Filterbank (MFB)
features as input to a model composed of convolutional and gated recurrent
unit (GRU) layers.

#####  5.6.2.1 Features

We extract 40-dimensional Mel Filterbank (MFB) features using a 25-millisecond
Hamming window with a step-size of 10-milliseconds using python-speech-
features 777https://github.com/jameslyons/python_speech_features. Each
utterance is represented as a sequence of 40-dimensional feature vectors. We
$z$-normalize the acoustic features using parameters extracted from the
training dataset. During each cross-validation fold, the parameters are chosen
from the training data and are applied to both the validation and testing
data.

#####  5.6.2.2 Network

Our baseline network is a state-of-art single utterance emotion classification
model which has been used in previous research [9, 116, 126]. The extracted
MFBs are processed using a set of convolution layers and GRUs (see Table 5.5
for the hyperparameters used for these layers). The output of these layers is
then fed through a mean pooling layer to produce an acoustic representation
which is then fed into a set of dense layers to classify activation or
valence.

#####  5.6.2.3 Training.

We implement the models using the Keras library [51]. We use a cross-entropy
loss function for each task (e.g., valence or activation). We learn the model
parameters using the RMSProp optimizer. We train our networks for a maximum of
50 epochs and use early stopping if the validation loss does not improve after
five consecutive epochs. Once the training process ends, we revert the
network’s weights to those that achieved the lowest validation loss. We repeat
the experiment five times. We report the results in terms of Unweighted
Average Recall (UAR, chance is 0.33), averaged over all test samples and five
repetitions. We compare the performance of different models or the same model
in different noisy conditions/partitions using a paired t-test using the
Bonferroni correction, asserting significance when $p\leq 0.05$.

####  5.6.3 End-to-End Deep Learning Networks

Next, we explore a transformer-based model. In this method the raw audio
signal is used as input to a pre-trained and fine-tuned network and the
emotion prediction is directly obtained as an output. These models do not
require us to perform manual or domain knowledge-based extraction of features.
They instead have a feature encoder component inside the model, which is
dynamic in nature, and hence, can change its output for the same signal based
on the dataset and nature of the task.

#####  5.6.3.1 Features

For the end-to-end deep learning models, we do not need to extract audio
features. Instead we rely on the network itself to both normalize and extract
features, that are later passed onto the deeper layers of the network. The
feature set here is the original wav files that are not modified in any
capacity. The eventual representations are of size 512, reproducing the setup
in the state-of-the-art implementation [175].

#####  5.6.3.2 Network

Our baseline network is the state-of-the-art wav2vec2.0 emotion recognition
model [175]. The wav2vec model is comprised of three parts: (i) a
convolutional neural network (CNN) that acts as feature encoder, (ii) a
quantizier module, and (iii) a transformer module. The input to the model is
raw audio data (16kHz) that is passed to a multi-block 1-d CNN to generate
audio representations (25ms). The quantizer is similar to a variational
autoencoder that encodes and extracts features using a contrastive loss. The
transformer is used for masked sequence prediction and encodes the bi-
directional temporal context of the features.

We use the base model, which has not been fine-tuned for ASR (wav2vec2.0-PT).
We then fine-tune the base model to predict the binned emotion labels. We use
the final representation of the output as an input to dense layers to produce
the final output.

#####  5.6.3.3 Training