# Best Practices for Noise-Based Augmentation to Improve the Performance of
Emotion Recognition “In the Wild”

Mimansa Jaiswal
University of Michigan, Ann Arbor
<EMAIL_ADDRESS>
Emily Mower Provost
University of Michigan, Ann Arbor
<EMAIL_ADDRESS>

###### Abstract

Emotion recognition as a key component of high-stake downstream applications
has been shown to be effective, such as classroom engagement or mental health
assessments. These systems are generally trained on small datasets collected
in single laboratory environments, and hence falter when tested on data that
has different noise characteristics. Multiple noise-based data augmentation
approaches have been proposed to counteract this challenge in other speech
domains. But, unlike speech recognition and speaker verification, in emotion
recognition, noise-based data augmentation may change the underlying label of
the original emotional sample. In this work, we generate realistic noisy
samples of a well known emotion dataset (IEMOCAP) using multiple categories of
environmental and synthetic noise. We evaluate how both human and machine
emotion perception changes when noise is introduced. We find that some
commonly used augmentation techniques for emotion recognition significantly
change human perception, which may lead to unreliable evaluation metrics such
as evaluating efficiency of adversarial attack. We also find that the trained
state-of-the-art emotion recognition models fail to classify unseen noise-
augmented samples, even when trained on noise augmented datasets. This finding
demonstrates the brittleness of these systems in real-world conditions. We
propose a set of recommendations for noise-based augmentation of emotion
datasets and for how to deploy these emotion recognition systems “in the
wild”.

##  1 Introduction

Emotion recognition is considered a low resourced domain due to the
limitations in the size of available datasets. As such, the performance of
emotion recognition models is often improved when techniques such as data
augmentation or transfer learning are applied. Accurate performance is
critical because the predictions made by emotion recognition models are often
used for downstream applications such as advertising, mental health monitoring
Khorram et al. (2018), hiring emo  and surveillance Martin (2019). However,
when used in such sensitive human-centered tasks, incorrect emotion
predictions can have devastating consequences.

Data used to train emotion recognition systems are often collected in
laboratory environments designed to avoid as much noise variation as possible.
This collection method ensures that the collected data are variable only due
to the presence of emotion or predefined control variables. This ensures that
the trained models avoid spurious correlations between noise and emotion
labels, which results in datasets that are usually clean and small.

However, these pristine conditions generally do not hold in the real world.
Researchers have addressed this concern by augmenting emotion datasets via
signal manipulation to enhance generalizability to real-world noise conditions
or avoid overfitting of ML models as is common in other speech domains Zheng
et al. (2016). For example, many speech domains, such as speech recognition
and speaker identification, tackle this challenge by manipulating training
samples to encourage models to learn only salient features Mošner et al.
(2019). But, unlike these tasks, where the desired output remains consistent
in the presence of noise, this same consistency cannot be assumed for emotion
labels. This paper investigates how common sources of noise affect both human
and machine perception of emotion.

Prior researchers has analyzed robust augmentation methods for automatic
speech recognition Li et al. (2014). The results have shown that humans have
surprisingly robust speech processing systems that can easily overcome the
omission of letters, changes in the speed of speech, and the presence of
environmental noise through psychoacoustic masking Biberger and Ewert (2016)
during speech comprehension. However, the same analyses have not been
completed within the domain of computational paralinguistic applications.
While it is known that the comprehension of lexical content in noisy data
examples is not significantly affected, the same effect in emotion perception
is understudied. Instead, previous works have focused on other confounders,
ranging from distribution shift Abdelwahab and Busso (2018) to physiological
factors such as stress Jaiswal et al. (b), or have studied how pink or white
noise affects emotion perception Parada-Cabaleiro et al. (2017); Scharenborg
et al. . Important outstanding questions include the impact of real world
noise, e.g., footsteps, running water, and how variation across both type and
loudness impacts human and machine perception.

In this paper, we study how human’s emotion perception changes as a factor of
noise. We find that multiple commonly used methods for speech emotion dataset
augmentation that assume consistent ground truth, like, speeding of the
utterance or the adding fillers and pauses, actually alter human emotion
perception. We show how the addition of noise, such as, footsteps, coughing,
rain, clock ticks, that does not alter human emotion perception, do result in
incorrect emotion recognition performance. This is critical because these
types of noise are common in real world applications (e.g., in inputs to
virtual agents such as Siri or Alexa). The fragility of these models can pose
a security risk, especially when the change in signal isn’t perceived by
humans. Adversarial attacks are designed to test the robustness of model. They
might be inaccurately evaluated if they introduce noise that do change the
perception of human, but are also considered as successful if the model output
changes. We end the paper with a set of recommendations for emotion dataset
augmentation techniques and and deployment of these emotion recognition
algorithms. We investigate the following research questions:

  * •

Q1: How does the presence of noise affect the perception of emotions as
evaluated by _human raters_? How does this effect vary based on the utterance
length, loudness and the added noise type, and the original emotion?

  * •

Q2: How does the presence of noise affect the performance of _automatic
emotion recognition systems_? Does this effect vary based on the type of the
added noise in agreement with changes in human perception?

  * •

Q3: Does dataset augmentation or sample denoising help improve ML models’
robustness to unseen noise?

  * •

Q4 : How does difference in allowed set of noises for adversarial perturbation
change evaluation of adversarial attack efficiency?

  * •

Q5: What are the best practices for speech emotion dataset augmentation and
model deployment, based on crowdsourced experiments on human perception?

Our findings suggest that human emotion perception is usually unaffected by
the presence of environmental noise or common modulations such as
reverberations and fading loudness. We find that human perception is changed
by signal manipulations such as speed, pitch change, and even the presence of
pauses and fillers, and hence are not good noise augmentation methods. We show
that the trained models though, are brittle to the introduction of various
kinds of noise, which do not change human perception, especially
reverberation. We show that some kinds of noise can help through either
dataset augmentation or sample denoising, or a combination of both, but, the
performance still drops when an unseen noise category is tested. We finally
show how noise augmentation practices can lead to a false improvement in
adversarial test efficiency, thus leading to an unreliable metric. We end the
paper with a set of augmentation and deployment suggestions.

##  2 Relation to Prior Work

Previous work focused on understanding how noise impacts machine learning
models can be classified into three main directions: (a) robustness in
automatic speech recognition or speaker verification; (b) noise-based
adversarial example generation; and, (c) improvement in performance of machine
learning models when augmented with noise.

Previous work has looked at how speech recognition systems can be built so
that they are robust to various kinds and levels of noise Li et al. (2014).
The most common themes in these papers are to concentrate on either data
augmentation or gathering more real-world data to produce accurate transcripts
Zheng et al. (2016). Other lines of work have looked into preventing various
attacks, e.g., spoofing or recording playback, on speaker verification systems
Shim et al. . Noise in these systems is usually considered to be caused by
reverberations or channel-based modulations Zhao et al. (2014). Researchers
have also looked into building noise robust emotion recognition models, using
either signal transformation or augmentation Aldeneh and Provost (2017). But
data augmentation in this analysis was formulated in a similar manner to those
for speech recognition, or, speaker identification systems, which have a fixed
ground truth and are independent of human perception. This paper attempts to
establish methods for noise-based data augmentation of emotion datasets that
are grounded in human perception.

Previous works have also investigated adversarial example generation that aims
to create audio samples that change the output of the classifier. However,
these methods usually assume white-box access to the network and create
modifications in either feature vectors or actual wave files Carlini and
Wagner ; Gong and Poellabauer (2017). This generally results in samples that
either have percetible differences when played back to a human, or are
imperceptible to a human but fail to attack the model when played over-air
Carlini and Wagner .

The line of work closest to ours is data augmentation aimed for training more
generalizable models. Noise and manipulation-based data augmentation is a
prime research field in the space of vision and text. Sohn et al. (2020) have
looked at how rotating and blackening out random pixels in images to create
augmented datasets leads to better performance on the test set for digit and
object recognition. Authors Sohn et al. (2020) have also looked at how data
augmentation can help with zero shot object recognition in sparse classes of
the dataset, providing new options for sub sampling of classes. In the field
of NLP, researchers have looked at how replacing words with their semantic
synonyms can lead to better performance on tasks such as POS tagging and
semantic parsing Wallace et al. (2019). Researchers have also looked at
techniques for data augmentation using noise augmentation Kim and Kim , mostly
focusing on how model training can be improved to yield better performance.
However, these augmentation techniques are mostly used for acoustic event
detection, speaker verification or speech recognition Ko et al. (2015); Rebai
et al. (2017), and has been sparingly used in audio classification tasks.

To the best of our knowledge, this is the first work that looks at the effect
of different kinds of real world noise and varying amounts of noise
contamination on the human perception of emotion and its implication for
noise-based augmentation for emotion datasets.

##  3 Datasets And Network

###  3.1 Dataset

For our study, we use the Interactive Emotional Dyadic MOtion Capture
(IEMOCAP) dataset Busso et al. (2008), which is commonly used for emotion
recognition. The IEMOCAP dataset was created to explore the relationship
between emotion, gestures, and speech. Pairs of actors, one male and one
female (five males and five females in total), were recorded over five
sessions (either scripted or improvised) The data were segmented by speaker
turn, resulting in a total of 10,039 utterances (5,255 scripted turns and
4,784 improvised turns). It contains audio, video, and associated manual
transcriptions.

###  3.2 Network

To focus on the effect of types of noise contamination on performance of
emotion classifiers, we use the state-of-art single utterance emotion
classification model which has widbeen used in previous research Khorram et
al. (2017); Krishna et al. .
Acoustic Feature. We extract 40-dimensional Mel Filterbanks (MFB) features
using a 25-millisecond Hamming window with a step-size of 10-milliseconds.
Each utterance is represented as a sequence of 40-dimensional feature vectors.
We $z$-normalize the acoustic features by each speaker.
Emotion Labels. The target emotion labels represented in a dimensional format
are binned into three classes to represent {low, medium, high} for both
activation and valence.
Architecture. The extracted MFBs are processed using a set of convolution
layers and Gated Recurrent Units (GRU), which are fed through a mean pooling
layer to produce an acoustic representation which is then fed into a set of
dense layers to classify activation or valence.
Training. We implement models using the Keras library Chollet (2015). We use a
subject independent five-fold cross validation scheme to select our train,
test and validation sets. We generate corresponding noisy examples for each
sample in the test using the method described in Section4. We use a weighted
cross-entropy loss function for each task and learn the model parameters using
the RMSProp optimizer. We train our networks for a maximum of 50 epochs and
monitor the validation loss from the emotion classifier after each epoch for
early stopping, stopping the training if the validation loss does not improve
after five consecutive epochs. Once the training process ends, we revert the
network’s weights to those that achieved the lowest validation loss on the
emotion classification task. We measure performance using Unweighted Average
Recall (UAR) (chance is 0.33). We report averaged performance over all samples
considered as test in multiple runs.
Sample Selection. We select 900 samples from the IEMOCAP dataset for the human
perception part of the study. The sample size is far larger than the ones used
for previous perception studies Parada-Cabaleiro et al. (2017); Scharenborg et
al. . To ensure that the 900 samples covered multiple variations, we select
100 samples from each activation and valence pair bin, i.e., 100 samples from
all the samples that have activation bin _low_ , and valence bin _low_ ; 100
samples which have activation bin _low_ , and valence bin _mid_ , and so on.

We ensure that out of the 100 samples, 30 of them are shorter than first
quartile or greater than fourth quartile of utterance length in seconds to
cover both extremities of the spectrum, and the rest of the 70 belong in the
middle. We also ensure that the selected samples had a 50-50 even split
amongst gender.

##  4 Noise

###  4.1 Environmental Noise

We define environmental noises (ENV) as additive background noises. These
noises are obtained from the ESC-50 datasetPiczak , which has often been used
for both, noise contamination and environmental sound classification, out of
which, we choose:

  * •

Natural soundscapes (Nat), e.g., rain, wind.

  * •

Human, non-speech sounds (Hum) for e.g. sneezing, coughing etc.

  * •

Interior/domestic sounds (Int) for e.g. door creaks, clock ticks etc

These environmental sounds are representative of many factors in “in the wild”
audio, especially in the context of virtual and smart home conversational
agents. We manipulate two factors in addition to noise:

  * •

ENVPos: We vary the position of the introduction of sound that (i) starts and
then fades out in loudness, represented by St, or (ii) occurs during the
entirety of the duration of the utterance, represented by Co. Complete
additive background would represent a consistent noise source in real world,
e.g., fan rotation.

  * •

ENVdB: We vary the signal to noise ratio (SNR) of additive background noise
for the Co method at levels of 20dB, 10dB and 0dB.

  * •

ENVLen: We vary the length of the introduced background noise, from a short
blip to covering the entire clip. Short length is represented by Sh, medium
length by Me, and complete length by Co. A complete additive background
represents a consistent noise source in real world (e.g. fan rotation).

###  4.2 Synthetic Noise

We define synthetic noise as modulations that aren’t additive in background.
These kinds of noises in audio signal can occur from linguistic/paralinguistic
factors, room environment, internet lags, or the physical locomotion of the
speaker. We add ten variations of synthetic noise in our dataset.

  * •

SpeedSeg: We speed up a random segment of the utterance by 1.25$\times$.

  * •

Fade: We fade the loudness of the utterance by 2% every second, which emulates
the scenario of a user moving away from the speaker. We increase the loudness
for fade in, and decrease for fade out.

  * •

Filler: Insertion of non-verbal short fillers such as ‘uh’, ‘umm’ (from the
same speaker) in the middle of a sentence. The insertion is either just the
filler (S) or succeeded and preeceeded by a long pause (L).

  * •

DropW: Dropping all non-essential word belonging to the set :{a, the, an, so,
like, and}.

  * •

DropLt: Phonological deletion or dropping of letters has been widely studied
as a part of native US-English dialect pho ; Yuan and Liberman . Hence, we
drop letters in accordance with various linguistic styles chosen from the
set:{/h/+vowel, vowel+/nd/+consonant(next word), consonant+/t/+consonant(next
word), vowel+/r/+consonant, /ihng/}.

  * •

Laugh/Cry: We add “sob” and “short-laughter” sounds obtained from AudioSet
Gemmeke et al. (2017) to the end of utterance.

  * •

SpeedUtt: Speed up the entire utterance by 1.25$\times$ or 0.75$\times$.

  * •

Pitch: Change the pitch by $\pm$ 3 half octaves.

  * •

Rev: Add room reverbration to the utterance.

##  5 Analysis

###  5.1 Qualitative human study

For the human perception part of the study, we introduce noise into the
selected 900 samples in ten ways each, leading a total of 9000 samples for
comparison. Each sample is introduced to the four kinds of environmental noise
and six out of ten randomly chosen synthetic noise methods to create ten noisy
samples.

Crowdsourcing Setup. We recruited annotators using MTurk from a population of
workers in the United States who are native English speakers, to reduce the
impact of cultural variability. We ensured that each worker had $>98$%
approval rating and more than 500 approved Human Intelligence Tasks (HITs). We
ensured that all workers understood the meaning of activation and valence
using a qualification task that asked workers to rank emotion content, as in
previous work of Jaiswal et al. (a). All HIT workers were paid a minimum wage
($\$9.45/$hr), pro-rated to the minute. Each HIT was annotated by three
workers.

For our main task, we created pairs of samples, i.e., the original and
modulated version. For each pair, we asked three workers to annotate if they
thought that the samples contained the same emotional content. If the worker
selected yes for both activation and valence, the noisy example was labeled
same and they could move to the next HIT. If the worker selected no, the noisy
example was labeled different. If a clip was labeled different, the annotator
was asked to assess the activation and valence of the noisy sample using Self
Assessment Manikins Bradley and Lang (1994) on a scale of one to five (as in
original IEMOCAP annotation).

We ensured the quality of the annotations by paying bonuses to the workers
based on time (removing the incentive to mark same every time) and by not
inviting annotators to continue if their level of agreement with our attention
checks was low.

We created final labels for the noisy examples by taking the majority vote
over each pair. The final label was either same emotion perception or
different emotion perception and an averaged valence and activation score. The
annotators agreed in 79% of the cases. All the samples along with the paired
noisy examples and their annotations will be made available for further
research.

Table 1: The original activation (Act) and valence (Val) performance is 0.67 and 0.59, respectively. The table shows the change in UAR on IEMOCAP (Iem) and the Augmented IEMOCAP (Iem(Aug)) datasets. The rows are augmentation methods that were evaluated as imperceptible for emotion by human annotators. | UAR
---|---
Iem | Iem(Aug)
Act | Val | Act | Val
Environmental Noise
NatSt |  | -.25 | -.24 | .22 | .18
NatdB (Co) | 20dB | -.33 | -.34 | .22 | .13
| 10dB | -.37 | -.41 | .33 | .31
| 0dB | -.40 | -.41 | .27 | .26
HumSt |  | -.22 | -.24 | .15 | .13
HumdB (Co) | 20dB | -.33 | -.37 | .16 | .16
| 10dB | -.37 | -.42 | .21 | .26
| 0dB | -.40 | -.42 | .25 | .21
IntSt |  | -.21 | -.25 | .15 | .18
IntdB (Co) | 20dB | -.31 | -.39 | .20 | .22
| 10dB | -.34 | -.39 | .23 | .19
| 0dB | -.40 | -.41 | .30 | .26
Synthetic Noise
SpeedSeg |  | -.09 | -.12 | .03 | .02
Fade | In | -.07 | -.10 | .05 | .04
| Out | -.09 | -.14 | .02 | .06
DropW |  | -.04 | -.05 | .02 | .00
DropLt |  | -.03 | -.02 | .06 | .03
Reverb |  | -.36 | -.37 | .05 | .04

Human Perception Study Results. We report our human perception results below.
The values represent the ratio of the utterances that were marked as different
based on majority vote amongst all the utterances introduced with that noise.
We also find that the presence of environmental noise, even when loud, rarely
affects human perception, verifying our initial judgement that humans are able
to psycho-acoustically mask the background noise. We verify expected patterns,
such as the presence of laughter or crying often changes human perception. The
results for the five synthetic modulations that significantly change human
perception of either activation or valence are as follows:

  * •

Filler(L)\- Act: 0.10, Val: 0.06, Filler(S)\- Act: 0.06, Val: 0.03;

  * •

Laugh\- Act: 0.16, Val: 0.17

  * •

Cry\- Act:0.20, Val: 0.22;

  * •

SpeedUtt(1.25x)\- Act: 0.13, Val: 0.03, SpeedUtt(1.25x)\- Act: 0.28, Val:
0.06;

  * •

Pitch(1.25x)\- Act: 0.22, Val: 0.07, Pitch(0.75x)-Act: 0.29, Val:0.10.

This suggests that modifications that do not change human perception and
wouldn’t raise suspicion, should ideally not impact performance of an ideal
machine learning model.

###  5.2 Variation in changes in human perception

In further experiments, we aim to find if there is a notable difference in
human perception of emotion based on the original length of utterance,
original emotion classification and gender of the speaker. We find that the
original emotion of the utterance, the gender of the speaker, or the length of
the utterance, plays no significant role in predicting if the perception of
noisy example would change. In line with the hypothesis tested by Parada et.al
in Parada-Cabaleiro et al. (2017), we test how noise shapes the direction of
perception change.

We find that the addition of laughter or crying increases perceived activation
while laughter increases perceived valence and crying decreases it. We also
find that increasing the pitch usually increases perceived activation and
vice-a-versa. Further, increases in the speed of an utterance increases
activation and vice-versa Busso et al. (2009). This evaluation gives us an
insight into what methods can be used for robust data augmentation, which,
unlike for speech recognition, do not always convey the same material.

###  5.3 Model performance

We study how introducing varying kinds of noise affects the performance of the
model trained on a clean dataset. The test samples of each fold are introduced
to all the types of noise (Section 4), except those that were also found to
often affect human perception (Pitch, SpeedUtt, Laugh, Cry, Filler), which are
then used to evaluate the performance of the trained network. Given that, the
domain of the trained network, the speaker independent fold distribution
remains the same, a drop in performance can be singularly attributed to the
addition of a particular noise.

We report the performance of emotion recognition model trained as described in
Section 3 in Table 1. We find that the machine learning model’s performance
decreases greatly given environmental noise, fading, and reverberation. There
is also a smaller drop in performance for speeding up parts of the utterance
and dropping words, showing the brittleness of these models.

###  5.4 Denoising Algorithms and Model Performance

The common way to deal with noise in any audio signal is to use denoising
algorithms. Hence, it is a valid counterpoint to understand how machine
learning models trained to recognize emotions perform if they are tested on
denoised samples. To this end, we look at two major approaches to denoising
algorithms in the audio space: Denoising Feature Space Valin (2018) and Speech
Enhancement Chakraborty et al. (2019). Denoising feature space algorithms seek
to remove noise from the extracted front end features. Speech enhancement
algorithms seek to convert noisy speech to more intelligible speech. Both
techniques are associated with known challenges, including signal degradation
Valin (2018) to harmonic disintegration Valin (2018).

We use the most common denoising methods from each category, Recurrent Neural
Network Noise Suppression (RNNNoise, denoising feature space) Valin (2018) and
Vector Taylor Series (VTS, speech enhancement) Chakraborty et al. (2019).

We train a system composed of the emotion recognition models, described in
Section 3.2. We then create two separate denoising approaches, one that
combines the emotion recognition model with RNNNoise and the other with VTS.
In both cases, we refer to the different approaches using RNNNoise and VTS,
respectively. We then use these denoising algorithms on the extracted features
for the noise augmented samples as shown in Section 4. The combination ensures
that the models are trained to maximize the probability of predicting the
correct emotion as well as maximizing the speech quality.

We use the noise augmented training, validation, and testing datasets,
described in Section 5.5. We then evaluate the performance of pre-trained
emotion recognition models from Section 3.2 on the denoised test samples using
a leave-one-noise-out cross-validation approach. This assumes that the system
doesn’t have an individual sample of noise prototype introduced in the sample,
akin to real world conditions. We do not compare with other denoising
algorithms that assume a priori knowledge of noise category Tibell et al.
(2009).

We find that the addition of a denoising component to the models leads to
significant improvement (average of $23\%\pm 3\%$ across all environmental
noise categories) in continuous noise introduction at 20dB SNR as compared to
when there is no denoising or augmentation performed. We observe a decline in
performance when dealing with other signal to noise ratios of continuous noise
additions, possibly due to masking of emotional information from the speech to
maintain legibility. We further show that the addition of a denoising
component didn’t significantly improve performance when samples were faded in
or out or segments were sped up. While we did see an improvement in
performance for _unseen_ reverberation contaminated samples as compared to
data augmentation, the performance is still significantly lower ($-28\%$) than
when tested on a clean test set. Finally, we observe an increase in emotion
recognition performance, compared to when the model is trained on a clean
training set, which supports the findings from dataset augmentation.

###  5.5 Model performance with augmentation

We then augment the dataset with noise to increase the robustness of the
trained model. Hence, we test environmental and synthetic noise separately. We
perform leave-one-noise-out cross-validation, augmenting each fold with one
kind of noise from the set: {Hum, Int, Nat}. We train on two kinds of
environmental noise and test on the third. We do the same with synthetic
noise, performing leave-one-noise-out cross-validation, augmenting each fold
with the most impactful augmentations from the set: {SpeedSeg, Fade and
Reverb}.

We find that data augmentation improves performance, especially when the
environmental noise is introduced at the start of the utterance, e.g., when
the test set is introduced with NatSt and the train set is introduced with
HumSt and IntSt, we see a performance jump of 22%, compared to when the model
is trained on clean data and tested on noisy samples. We can speculate that
the network learns to assign different weights to the start and end of the
utterance to account for the initial noise.

Though the performance increase on test data when introduced to continuous
background noise is aided by addition of training samples that have a
continuously present background noise as well, the performance is still
affected due to introduction of new noise in the test set.

We also find that it is hard to improve performance on utterances contaminated
with reverberation, a common use case, even when the training set is augmented
with other types of noise. This can be because, reverberation adds a
continuous human speech signal in the background delayed by a small period of
time. None of the other kind of noises have speech in them, and hence
augmentation doesn’t aid the model to learn robustness to this kind of noise.

###  5.6 Use Case: Adversarial Attack Evaluation

The common way to evaluate the robustness of a model to noise or an attacker
with malicious intent, is adding noise to the dataset and testing performance
of the model. This method is commonly used for various tasks, such as, speech
recognition, or, speaker identification Abdullah et al. (2019), which aren’t
perception dependent as discussed before. The closest task to ours where the
ground truth varies based on noise introduction is sentiment analysis. In this
case, the adversarial robustness is usually tested by flipping words to their
synonyms, such that the meaning of the text remains the same, and analyzing
how the predictions of the model change. Unlike the lexical modality, speech
cannot be broken down into such discrete components with obvious replacements
while maintaining speech feature integrity, and hence, introduction of noise
for emotion recognition while assuring that the perception remains the same is
more difficult.

Instead, we formulate the problem as a decision boundary untargted attack,
which tries to find the minimal amount of noise perturbation possible in _k-
queries_ such that the decision of a model changes, while having no knowledge
about the internal workings of the model (probabilities or gradients). A
decision-based attack starts from a large adversarial perturbation and then
seeks to reduce the perturbation while staying adversarial. In our case,
because we are considering noise augmentations, we define the distance between
an adversarial sample and the original sample as the degradation in signal to
noise ratio (SNR). The input to this attack model is a set of allowed noise
categories (e.g., white, gaussian, or categorical), and the original signal.
We also modify the decision boundary attack to first use the any random sound
from four categories at lowest SNR. If the attack model is successful, it then
shifts to using other additive noise options from that category. In case, the
attacker has knowledge about the relation between noise category and
performance degradation, we also provide an ordering to the attack model, such
that it is more successful given the same limited budget of _k-queries_. The
sample is considered to have been successfully attacked only when the noise
addition chosen by the algorithm is at SNR of $>$ 10dB. We use the Foolbox
Tooklit Rauber et al. (2017) to implement a decision boundary attack Brendel
et al. (2017).

For our purpose, we assume that an attacker has access to a subset of labelled
data from the set of users (_U_) in IEMOCAP and the training and the testing
conditions remain the same as that of the black-box emotion recognition model.
We calculate the average probability of a successful attack, $P_{IF}$, for any
sample from user _$U_{a}$_ using _k_ queries.

We calculate $P_{IF}$ for four conditions varying in two: 1) knowledge about
the correlation between the performance deterioration of emotion recognition
and the noise category (_Corr_) and 2) evaluation only on those noise
categories whose addition doesn’t change human perception (_Eval_). This leads
to four experimental conditions, i.e. _{ ordering knowledge based on
degradation (Y/N)} x {allowed noise categories that change human perception
(Y/N)}_. The results are reported in Table 2. We see that the efficiency is
always higher when the attack model can use noises that change human
perception, but is an unreliable way of evaluating due to change in ground
truth label. This shows that not accounting for human perception in evaluation
can lead to faux increase in attack efficiency, and hence improper
benchmarking of proposed adversarial attack methods proposed in future.

Table 2: Attacker efficiency ($P_{IF}$) using noise-based adversarial methods. _k_ refers to the number of black box queries an attacker can access, whereas, Corr refers to the knowledge of correlation between performance deterioration of emotion recognition and noise category, and Eval refers to whether the attack efficiency is evaluated only on those noise categories that human perception doesn’t change.  Bold shows significant improvement. Higher values are better. Significance is established using paired t-test, adjusted p-value$<0.05$ |  _k_ =5 |  _k_ =15 |  _k_ =25
---|---|---|---
Corr: No, Eval: No | 0.22 | 0.29 | 0.39
Corr: Yes, Eval: No | 0.32 | 0.38 | 0.53
Corr: No, Eval: Yes | 0.31 | 0.36 | 0.45
Corr: Yes, Eval: Yes | 0.34 | 0.43 | 0.58

##  6 Recommendations

We propose a set of recommendations, for both augmentation and deployment of
emotion recognition models in the wild, that are grounded in human perception.
For augmentation, we suggest that:

  1. 1.

Environmental noise can be added to datasets to improve generalizability to
varied noise conditions, whether using denoising or augmentation method or a
combination of both.

  2. 2.

It is good to augment datasets with fading loudness of segments, dropping
letters or words, and speeding up small (no more than 25% of the total sample
length) segments of the complete sound samples in the dataset. But it is
important to note that these augmented samples should not be passed through
the denoising component as the denoised version loses emotion information.

  3. 3.

One should not change the speed of the entire utterance more than 5% and to
not add intentional pauses or any background noises that elicit emotion
behavior, e.g., sobs or laughter.

Regarding deployment, we suggest that:

  1. 1.

Noisy starts and ends of utterances can be handled by augmentation, hence, if
the training set included these augments, there is no issue for deployed
emotion recognition systems.

  2. 2.

Reverberation, in its various types is hard to handle for even augmented
emotion recognition models, and hence, the samples must either be cleaned to
remove reverberation effect, or must be identified as low confidence for
classification.

##  7 Conclusion

In this work, we study how the introduction of real world noise, either
environmental or synthetic, affects human emotion perception. We identify
noise sources that do not affect human perception, and hence can be used for
data augmentation, leading to more robust models. We then study how models
trained on the original dataset are affected in performance when tested on
these noisy samples and if augmentation of the training set leads to
improvement in performance of the machine learning model. We conclude that,
unlike humans, machine learning models are extremely brittle to the
introduction of many kinds of noise. While the performance of the machine
learning model on noisy samples is aided from augmentation, the performance is
still significantly lower when the noise in the train and test environments
does not match. In this paper, we demonstrate fragility of the emotion
recognition systems and valid methods to augment the datasets, which is a
critical concern in real world deployment.

##  8 Ethical Considerations

Data augmentation is often applied to speech emotion recognition to impove
robustness. Better augmentation methods are an important way to not only
ensure reliability and robustness of these models, but also improve the real-
life adoption in high-stakes downstream applications. Knowing when human
perception of emotion can change in the presence of noise is needed to design
better model unit tests and adversarial tests for verifying the reliability of
the system. However, emotion variability is often dependent on multiple
factors, such as, culture, race, gender, age etc, some of which are highly
protected variables. These models can also encode stereotypical expected
behavior from a certain group, and hence have a higher error rate for other
groups. It is important to note that this paper considers a small set of
crowd-sourced workers as human raters of emotion perception, who belong to US
and are well versed in English, the language this dataset is collected in, and
the model is trained on.

## References

  * (1) How emotion ai can transform large-scale recruitment processes shorturl.at/gcs15, note = Accessed: 10-21-2019.
  * (2) Phonological history of english consonant clusters= https://bit.ly/3l08zbu, Accessed: 10-21-2019.
  * Abdelwahab and Busso (2018) Mohammed Abdelwahab and Carlos Busso. 2018.  Domain adversarial for acoustic emotion recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Abdullah et al. (2019) Hadi Abdullah, Washington Garcia, Christian Peeters, Patrick Traynor, Kevin RB Butler, and Joseph Wilson. 2019.  Practical hidden voice attacks against speech and speaker recognition systems.  _arXiv preprint arXiv:1904.05734_.
  * Aldeneh and Provost (2017) Zakaria Aldeneh and Emily Mower Provost. 2017.  Using regional saliency for speech emotion recognition.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2741–2745. IEEE.
  * Biberger and Ewert (2016) Thomas Biberger and Stephan D Ewert. 2016.  Envelope and intensity based prediction of psychoacoustic masking and speech intelligibility.  _The Journal of the Acoustical Society of America_.
  * Bradley and Lang (1994) Margaret M Bradley and Peter J Lang. 1994.  Measuring emotion: the self-assessment manikin and the semantic differential.  _Journal of behavior therapy and experimental psychiatry_.
  * Brendel et al. (2017) Wieland Brendel, Jonas Rauber, and Matthias Bethge. 2017.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  _arXiv preprint arXiv:1712.04248_.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_.
  * Busso et al. (2009) Carlos Busso, Murtaza Bulut, and Sungbok Lee. 2009.  Shrikanth narayanan fundamental frequency analysis for speech emotion processing.  _The role of prosody in affective speech_ , 97:309.
  * (11) Nicholas Carlini and David Wagner.  Audio adversarial examples: Targeted attacks on speech-to-text.  In _2018 IEEE Security and Privacy Workshops (SPW)_.
  * Chakraborty et al. (2019) Rupayan Chakraborty, Ashish Panda, Meghna Pandharipande, Sonal Joshi, and Sunil Kumar Kopparapu. 2019.  Front-end feature compensation and denoising for noise robust speech emotion recognition.  In _INTERSPEECH_ , pages 3257–3261.
  * Chollet (2015) François Chollet. 2015.  keras.  https://github.com/fchollet/keras.
  * Gemmeke et al. (2017) Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events.  In _Proc. IEEE ICASSP 2017_ , New Orleans, LA.
  * Gong and Poellabauer (2017) Yuan Gong and Christian Poellabauer. 2017.  Crafting adversarial examples for speech paralinguistics applications.  _arXiv preprint arXiv:1711.03280_.
  * Jaiswal et al. (a) Mimansa Jaiswal, Zakaria Aldeneh, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost. a.  Muse-ing on the impact of utterance ordering on crowdsourced emotion annotations.  In _IEEE ICASSP 2019_.
  * Jaiswal et al. (b) Mimansa Jaiswal, Zakaria Aldeneh, and Emily Mower Provost. b.  Controlling for confounders in multimodal emotion classification via adversarial learning.  In _2019 ICMI_.
  * Khorram et al. (2017) Soheil Khorram, Zakaria Aldeneh, Dimitrios Dimitriadis, Melvin McInnis, and Emily Mower Provost. 2017.  Capturing long-term temporal dependencies with convolutional networks for continuous emotion recognition.  _arXiv preprint arXiv:1708.07050_.
  * Khorram et al. (2018) Soheil Khorram, Mimansa Jaiswal, John Gideon, Melvin McInnis, and Emily Mower Provost. 2018.  The priori emotion dataset: Linking mood to emotion detected in-the-wild.  _arXiv preprint arXiv:1806.10658_.
  * (20) Hyoung-Gook Kim and Jin Young Kim.  Acoustic event detection in multichannel audio using gated recurrent neural networks with high-resolution spectral features.  _ETRI Journal_.
  * Ko et al. (2015) Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015.  Audio augmentation for speech recognition.  In _Sixteenth Annual Conference of the International Speech Communication Association_.
  * (22) Kalpesh Krishna, Liang Lu, Kevin Gimpel, and Karen Livescu.  A study of all-convolutional encoders for connectionist temporal classification.  In _2018 IEEE ICASSP_.
  * Li et al. (2014) Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. 2014.  An overview of noise-robust automatic speech recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Martin (2019) Nicole Martin. 2019.  The major concerns around facial recognition technology.
  * Mošner et al. (2019) Ladislav Mošner, Minhua Wu, Anirudh Raju, Sree Hari Krishnan Parthasarathi, Kenichi Kumatani, Shiva Sundaram, Roland Maas, and Björn Hoffmeister. 2019.  Improving noise robustness of automatic speech recognition via parallel data and teacher-student learning.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6475–6479. IEEE.
  * Parada-Cabaleiro et al. (2017) Emilia Parada-Cabaleiro, Alice Baird, Anton Batliner, Nicholas Cummins, Simone Hantke, and Björn W Schuller. 2017.  The perception of emotions in noisified nonsense speech.  In _INTERSPEECH_.
  * (27) Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification.  In _Proceedings of the 23rd Annual ACM Conference on Multimedia_.
  * Rauber et al. (2017) Jonas Rauber, Wieland Brendel, and Matthias Bethge. 2017.  Foolbox: A python toolbox to benchmark the robustness of machine learning models.  _arXiv preprint arXiv:1707.04131_.
  * Rebai et al. (2017) Ilyes Rebai, Yessine BenAyed, Walid Mahdi, and Jean-Pierre Lorré. 2017.  Improving speech recognition using data augmentation and acoustic model fusion.  _Procedia Computer Science_.
  * (30) Odette Scharenborg, Sofoklis Kakouros, and Jiska Koemans.  The effect of noise on emotion perception in an unknown language.  In _Proc. 9th International Conference on Speech Prosody 2018_.
  * (31) Hye-Jin Shim, Jee-Weon Jung, Hee-Soo Heo, Sung-Hyun Yoon, and Ha-Jin Yu.  Replay spoofing detection system for automatic speaker verification using multi-task learning of noise classes.  In _2018 TAAI_.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _arXiv preprint arXiv:2001.07685_.
  * Tibell et al. (2009) Kajsa Tibell, Hagen Spies, and Magnus Borga. 2009.  Fast prototype based noise reduction.  In _Scandinavian Conference on Image Analysis_ , pages 159–168. Springer.
  * Valin (2018) Jean-Marc Valin. 2018.  A hybrid dsp/deep learning approach to real-time full-band speech enhancement.  In _2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP)_ , pages 1–5. IEEE.
  * Wallace et al. (2019) Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019.  Allennlp interpret: A framework for explaining predictions of nlp models.  _arXiv preprint arXiv:1909.09251_.
  * (36) Jiahong Yuan and Mark Liberman.  Automatic detection of “g-dropping” in american english using forced alignment.  In _2011 IEEE Workshop on Automatic Speech Recognition & Understanding_. IEEE.
  * Zhao et al. (2014) Xiaojia Zhao, Yuxuan Wang, and DeLiang Wang. 2014.  Robust speaker identification in noisy and reverberant conditions.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Zheng et al. (2016) Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. 2016.  Improving the robustness of deep neural networks via stability training.  In _Proceedings of CVPR_.