# Human-Imitating Metrics for Training and Evaluating Privacy Preserving
Emotion Recognition Models Using Sociolinguistic Knowledge

Mimansa Jaiswal, Emily Mower Provost

###### Abstract

Privacy preservation is a crucial component of any real-world application.
But, in applications relying on machine learning backends, privacy is
challenging because models often capture more than what the model was
initially trained for, resulting in the potential leakage of sensitive
information. In this paper, we propose an automatic and quantifiable metric
that allows us to evaluate humans’ perception of a model’s ability to preserve
privacy with respect to sensitive variables. In this paper, we focus on
saliency-based explanations, explanations that highlight regions of the input
text, to infer internal workings of a black box model. We use the degree with
which differences in interpretation of general vs privacy preserving models
correlate with sociolinguistic biases to inform metric design. We show how
certain commonly-used methods that seek to preserve privacy do not align with
human perception of privacy preservation leading to distrust about model’s
claims. We demonstrate the versatility of our proposed metric by validating
its utility for measuring cross corpus generalization for both privacy and
emotion. Finally, we conduct crowdsourcing experiments to evaluate the
inclination of the evaluators to choose a particular model for a given purpose
when model explanations are provided, and show a positive relationship with
the proposed metric. To the best of our knowledge, we take the first step in
proposing automatic and quantifiable metrics that best align with human
perception of model’s ability for privacy preservation, allowing for cost-
effective model development.

##  1 Introduction

Privacy has been an important concern in deploying black box machine learning
algorithms in the real world. This concern stems from the ability of multi-
parameter neural networks to memorize and replicate training data, and its
impact on both the re-identification of samples (Carlini et al. 2019) used for
training the model and potential to base classification or generation
decisions on these “unintentionally” learned sensitive variables. Many web
based applications claim are trained using de-identified user data and hence
private to build implying trustworthiness. While it might be true that that
these models do not explicitly make use of any protected variables, they often
end up learning demographic variables in the generated representations (Sun et
al. 2019). In this paper, we evaluate how metrics commonly used to evaluate a
model’s prediction accuracy and privacy preservation ability do not
necessarily correlate to human trust and judgement of how private a model’s
working is. We then propose a cost-effective human-imitating explainable
metric that can be used to maximize the model’s ability to preserve privacy as
perceived by humans, without incurring the huge costs of human evaluations and
maintaining performance on standardized metrics.

Previous studies have verified that representations learned from machine
learning models trained to predict emotion also encode demographic factors
(Jaiswal and Provost 2020). This can range from effects such as the model
using gender encoding in the input representation for text auto-completion to
sound a ‘particular’ way. But these models are also used to make decisions
that can have serious consequences on a person’s quality of life, such as the
model using gender-based information in making hiring decisions (Pliev 2019).
Knowing that publicly deployed ML models can imbibe this hidden demographic
information leads to humans’ distrust in the model’s ability to preserve
privacy. This may lead to users be unwilling to adopt helpful applications
such as passive mental health monitoring, for the fear of revealing their
personal information (Gerke, Minssen, and Cohen 2020).

To counter the problem of data leakage or privacy preservation, researchers
have proposed methods spanning multiple fields such as in vision, language,
and speech (Bengio, Courville, and Vincent 2013; Wang and Chang 2020). One
common way to mitigate these concerns is to introduce privacy preserving
mechanisms that avoid implicit encoding of protected variables in the
generated representations, through either introduction of noise in the
training data or intentionally training the model to not learn a private
variable. In the field of natural language processing, researchers have
proposed methods such as debiasing, model re-training, model distillation2 to
improve privacy preservation of the model for different conditions and
required privacy protections (Liu et al. 2021). Most of these models though
are evaluated on two major criterion, performance on the desired prediction
task (primary task, which is emotion recognition, in our case), and, the
performance of the model to be able to predict the private variable (in our
case, gender), either from the perspective of the original model or an
adversarial action. Hence, the most common metric used in these papers are
usually f1-scores or accuracy values. While these evaluation methods are good
for non-human-facing prediction models (such as, dam water level prediction),
they miss the mark in being able to quantify a user’s perception of the model,
both, for the primary and the privacy preservation task, which is a necessary
component for the adoption of AI-based decision systems. But, a challenge
remains: the major hindrance towards using human judgement as a metric has
been non-quantifiable and expensive to obtain human evaluations.

In this paper, we introduce a metric, which we call the emotion-privacy (EP)
metric that will quantify human judgement of the performance of, and,
preference for, any given model with respect to both emotion recognition and
privacy preservation. For the privacy preservation task, we focus on gender
prediction, acknowledging that the benchmark emotion recognition datasets are
restricted to binary gender labels, which may not align with how individuals
self-identify. To do this, we consider two lists of word-tokens both of which
are weighted. The first list has word tokens that have been identified to be
indicative of gender from sociolinguistic literature. The second list contains
word tokens that the model learns to unlearn, again with associated weights,
when being trained to preserve privacy. We identify words that occur in both
lists and, for each word, multiply the weight of that word in each list. Next,
we identify words that occur in only one list, multiplying that word’s weight
by -1, indicating the mismatch. This creates an importance value for each
word. Finally, for an input sample, we sum the importance values for each word
in the text and divide by the total number of tokens. However, the EP metric
refers to the dataset as a whole. We calculate the EP metric over the complete
dataset by taking an average of this value over all samples in the dataset.

We then conduct both, empirical and crowdsourcing-based experiments to test
the goodness of the proposed metric. We imbue the emotion recognition models
with privacy preservation through multiple state-of-the-art strategies:
adversarial training, bias-fine tuning, and dataset augmentation. We first
evaluate the performance of these trained models on three major criterion:
generalizability, performance of emotion prediction, and performance of an
adversary aiming to obtain private information. We then conduct crowdsourcing
experiments to find how these multiple privacy preservation techniques are
perceived by humans, asking them questions related to trust in the model and
preference of a model.

We find that our metric has consistently significant high correlation with
user trust and preference, and hence can be used to train or evaluate privacy
preservation models. We remind that the computation of the proposed metric is
independent of the private variable and can be applied as long as there exist
sociolinguistic studies to draw from. This suggests that this approach can be
used for alternative variables of interest. Additionally, the computation of
our metric is independent of the chosen method for model interpretation, and
relies only on having a weighted word list, irrespective of the reason and
motivation behind the obtained ordering. We offer some suggestions on how to
obtain these lists for other commonly studied protected variables, such as
race and age.

##  2 Related Work

Model interpretability is a critical component of machine learning systems
that are designed to interact with humans. Models that are interpretable have
the ability to explain _why_ they are making decisions, or in the context of
privacy, _why_ users should have faith that they are preserving users’
sensitive information. We divide our related work section into three parts: 1)
privacy preservation or debiasing, 2) interpretability methods, and 3) human
trustworthiness.

###  2.1 Privacy Preservation or Debiasing

Researchers in fair algorithmic approaches have investigated approaches to
develop models that are invariant to particular sensitive attributes, to
obtain debiased word embedding (Bolukbasi et al. 2016), ensure fairness
parities (Corbett-Davies and Goel 2018), and train debiased hate speech
classification (Davidson, Bhattacharya, and Weber 2019). These approaches
often use the accuracy of the demographic variable prediction as the fairness
metric. In this paper, we instead evaluate how interpretations can be used as
an evaluation metric, one that measures the model’s performance on the primary
task and also how closely the rules learned by the machine learning system for
the optimized variable (e.g., emotion, privacy, etc.) are aligned with human
trust.

Researchers have investigated the impact of privacy preserving approaches by
creating artificially augmented datasets that change the data distribution.
For example, Sun et al. created an augmented dataset that by changing the bias
towards opposite gender in their original data set. They then trained on the
union of the original and data-swapped sets (Sun et al. 2019) and found that
the resultant models capture less racial information, and are more secure to
adversarial effects used to obtain private information.

Researchers have also investigated bias fine-tuning to reduce bias in machine
learning models by initializing models with a large unbiased dataset with the
aim of retaining that ‘unbiased’ nature when fine-tuned (Sun et al. 2019).
However, work by (Jaiswal and Provost 2020) shows how these models though can
still introduce privacy violations while obtaining promising performance over
privacy evaluation metrics.

###  2.2 Model Interpretation and Explanation

New tools are being developed to create deep learning models that are more
interpretable (Kim, Khanna, and Koyejo 2016; Szegedy et al. 2013).
Interpretability has been used for auditing algorithms’ predictions (Belinkov
and Glass 2019). Recently, explanation methods have also been used as human
aids, i.e., using summarization for medical records, such as Subjective,
Objective, Assessment and Plan (SOAP) notes (Krishna et al. 2020) or using
explanations for model debugging (Nushi, Kamar, and Horvitz 2018). Other
methods, beyond input saliency, include multiple other approaches such as
representative samples and influence functions, which is succinctly summarized
by (Belinkov and Glass 2019). While major interpretation methods cater to
explaining models to a general audience, our work aligns more with use of
interpretability to assess and debug a trained model by a model designer.
Researchers have previously looked at using interpretability as a mechanism
for auditing a model and checking the learned decision making flows (Engler
2021).

###  2.3 Human Perception of ML Models

To be deployed at a large scale, machine learning models must be robust and
generalizable, while at the same time assuring users that the algorithms will
appropriately handle their data. One component of this goal is accomplished
through the development of algorithms with strong privacy guarantees. Another
component is to create models that users will trust. Previous research has
shown that humans are more likely to trust a model’s decision if its
explanation are aligned with how they, themselves, make predictions (Sperrle
et al. 2019; Ferrario, Loi, and Viganò 2019; Schmidt and Biessmann 2019).
Further, it is easier for the user to trust a model and its capabilities if
they are provided with information about how the model processes their data,
and explanations from the model regarding how predictions were generated
(Schmidt and Biessmann 2019). This allows users to personally verify whether
the model is implicitly making inferences tied to their sensitive demographic
information (e.g., age or gender). Thus, explanations must be both plausible
and faithful to the internal model workings. Researchers have previously
investigated the utility of explanations for various stakeholders (Preece et
al. 2018), for inducing trustworthiness (Heuer and Breiter 2020) and have also
explored the possibility of false privacy guarantees (Pruthi et al. 2019).

In this work we look at how saliency methods, applied for model
interpretability combined with sociolinguistic knowledge, can be used as a
proxy metric for human perception of a model’s ability for a given task. This
combination of model interpretation and expert knowledge allows us to have a
cost-effective metric that not only captures the aspects of human judgement,
but is also quantifiable and hence optimizable for machine learning
algorithms.

##  3 Research Questions

RQ1: Is there a difference in saliency-based interpretations obtained from a
model that is trained just for emotion recognition vs. another that is trained
for emotion recognition while preserving privacy?
RQ2: Does the difference in the saliency-based interpretations, explanations
that highlight regions of the input text and should thus match human judgment,
actually align with human perception of goodness of privacy preservation?
RQ3: How can we quantify this difference to obtain a metric that is a proxy
for human judgement of the goodness of privacy preservation - one that encodes
both the model’s ability to avoid private variable leakage and a person’s
belief that the model preserves privacy?
RQ4: Is the proposed metric versatile for quantifying human trust in other
tasks beyond privacy preservation? We look at the primary task of emotion
recognition, and investigate:
RQ4a: Does the proposed metric correlate with human’s trust in the ability of
a model to effectively recognize emotion? RQ4b: If yes, does the use of this
metric, which has been designed to be both optimizable and human-aligned, in
model training encourage generalizability?

##  4 Datasets

We use three datasets for emotion recognition purposes, IEMOCAP (Busso et al.
2008), MSP-Improv (Busso et al. 2016), and MuSE (Jaiswal et al. 2020),
focusing on only the text modality of each dataset. IEMOCAP, an audio-visual +
motion capture dataset consisting of 10,039 utterances, was collected to
understand how emotion expressions shape behavioral patterns. The data consist
of interactions between five mixed-gender pairs of actors (one male and one
female, ten actors total). The data were labeled by 5 human annotators using
emotion categories and dimensional attributes respectively. MuSE, an audio-
visual dataset consisting of 2,648 utterances, was collected to understand the
interplay between stress and emotion in natural spoken communication. The data
were labeled by 232 human annotators, 3 per utterance using dimensional
attributes. MSP-Improv, an audio-visual dataset consisting of 8,438
utterances, was collected to capture naturalistic emotions by 12 actors (6
male, 6 female). The data was annotated by 5 evaluators per utterance for both
emotion categories and dimensional attributes. We restrict our analysis to the
data collected from the improvised turns (4,381 utterances). These data
consist of fixed prompts, initiated first by one actor, and then the other.
This provides a controlled environment for the collection of emotional
sentences, where the topic distribution is consistent across both genders,
reducing the gender-bias. This will allow us to train a model that is also
less biased due to lexical content for the primary emotion recognition task.
The transcripts in both MuSE and IEMOCAP were generated by humans, the
transcripts in MSP-Improv were generated using Microsoft Azure Speech-to-Text.
IEMOCAP is used to train the set of privacy preserving methods whose
predictions and explanations are used in the crowdsourcing experiment. MSP-
Improv is used as an auxiliary dataset in bias-fine tuning, one of the privacy
preservation methods. MuSE is used to assess the generalizability of the
models learned on the IEMOCAP or MSP-Improv+IEMOCAP (initialized on MSP, fine-
tuned on IEMOCAP).

###  4.1 Labels

The dimensional emotion labels are binned into 3 classes to represent {low,
mid, high} for valence using binned averaged rating. The classes are defined
as {(low:[1,2.75]), (mid:(2.75,3.25]), (high:(3.25,5])} for IEMOCAP and MSP-
Improv and {(low:[1,3.75]), (mid:(3.75,4.25]), (high:(4.25,9])} for MuSE. We
treat the problem as a three-way classification problem, where the goal is to
assign a label from {_low_ , _mid_ , _high_} to a given utterance.

##  5 Methods

###  5.1 General Emotion Recognition Model  _:General_

We train all of the following models on IEMOCAP* (original set or
modifications as mentioned below). For within dataset evaluation, we use the
test dataset from IEMOCAP, whereas, for out of distribution data evaluation,
we test the same model on MuSE dataset. Our classification model is based on
the base version of Bidirectional Encoder Representations from Transformers
(BERT) model due to the prevalence of this approach (Devlin et al. 2018). We
also use a pre-trained BeRT tokenizer for the model. We implement and fine-
tune the model using the HuggingFace library (Wolf et al. 2019). We replace
the pretraining head of the BERT model with a classification head, which is
randomly initialized. We use the _Trainer_ module that takes in
_TrainingArguments_ to then fine-tune the model for our purposes. We use
unweighted average recall (UAR) as our evaluation metric for the emotion
classification to account for class imbalance (Rosenberg 2012). We train our
models using PyTorch with RMSprop and a weighted cross-entropy loss function,
and, use the validation set for early stopping after the results do not
improve for five consecutive epochs. We run each experiment three times to
account for random initialization of the parameters.

###  5.2 Privacy by Adversarial Training  _:PrivAdv_

We use an adversarial paradigm to train the models to preserve the privacy of
the generated embeddings with respect to the demographic variable of gender.
The main network is trained to unlearn gender using a Gradient Reversal Layer
(GRL) (Ganin and Lempitsky 2014). GRLs are a multi-task approach to train
models that are invariant to specific properties (Meng et al. 2018). We place
the GRL function between the embedding sub-network and the gender classifier
to obtain gender-invariant representations.

###  5.3 Privacy by Data Augmentation  _:PrivAug_

We create an augmented data set using gender-swapping to compare our proposed
method and resultant metric to other successful approaches (Iosifidis and
Ntoutsi 2018). We use a pronoun-based word list and create a gender-swapped
equivalent for each sentence, for e.g., replacing “he” with “she”, or “his”
with “hers” and so on. Data augmentation does come with its own issues: double
the training data size with no added primary task information, expensive list
creation of gender based words in the dataset to be replaced, and nonsensical
sentence creation (Belinkov and Glass 2019).

###  5.4 Privacy by Bias Fine-Tuning  _:PrivBias_

Bias fine-tuning assumes the existence of an additional dataset that is
unbiased with respect to the sensitive variable of interest, which is used to
train an initial model. The goal is that the resulting model should then no
longer encode bias due solely to dataset characteristics. We use the
improvised turns subset of MSP-Improv for unbiased fine-tuning (the improvised
prompts are unbiased with respect to gender). We then fine-tune this model on
IEMOCAP.

###  5.5 Control Models

We train and use the two models that follow to have maximally poor performance
for privacy preservation. This provides a quality control check for the
crowdsourcing experiments, we can monitor annotation quality, and a negative
training baseline for our proposed metric. _GenderCtrl_ : Explicitly training
for gender classification The GenderExp model is a multi-task model, which
adds gender prediction on top of the general model. It is used as a control
mechanism for any metric calculation, because it should ideally capture
correlations that we are trying to avoid, leading to a low evaluation score on
the chosen metrics (Section 6.2).

ArtNoise: Artificially Noisy Model We create a parallel corpus of IEMOCAP to
use as a control baseline. We add six artificially ”noisy” features, i.e,
{’zq0’, ’zq1’, ’zq2’, ’zx0’, ’zx1’, ’zx2’}, such that they correlate
specifically with both emotion and gender. For example, zq0 would be added to
a random selection of male-low valence samples. We then train a classification
model to predict emotion on this dataset, and highlight salient features.
Because the added signals are completely correlated to various sub-classes in
the dataset, it ensures that the model always learns these added tokens as
salient features. Given these added tokens are meaningless, significant
saliency of these tokens implies that the model explanations are neither good
for judging privacy preservation nor for judging ability of emotion
recognition (Kaushik, Hovy, and Lipton 2019). This acts as a controlled
mechanism, specifically an attention check, allowing us to discard any
crowdsourced task where the crowdsourced worker selected the result from this
model, because the words highlighted by this model are always nonsensical.

##  6 Privacy Metrics via Human Perception

People often rely on gender-based patterns when assessing emotion expression
(Brescoll 2016). The term, perceptual bias, is defined as the probability of
any sample being specifically understood as a particular emotion based on the
speaker’s gender and not on how the information was expressed. We aim to
design a metric such that it can capture this bias to tease apart the
linguistic indicators through which humans attribute gender to any written
text (Fosch-Villaronga et al. 2021). We focus on two types of sources to
obtain a desired keyword list, (a) using word lists that have been tested
using Implicit Association Test (IAT) to have population level bias between
the genders and (b) sociological studies (Newman et al. 2008). The IAT is
designed to reveal attitudes and other automatic associations even for
subjects who prefer not to express those attitudes. We use the gender bias IAT
list provided by WEAT (Word Embedding Association Test) (Swinger et al. 2019)
to identify words that are known to have significant differential attitude
towards a word. Sociologists have discussed that humans recognize gender in
written communication using certain word categories (Brescoll 2016). For
example, women tend to use more hedge words but men are likely to use more
referential language. We consider hedge words, tags, referential language,
profanity, first person pronouns and politeness markers as the categories. We
use Linguistic Word Inquiry Count (LIWC) to find keywords or phrases that fall
into the defined categories of biased perception indicators as mentioned in
these studies.

###  6.1 Sociolinguistically-Informed Wordlists

While we focus on gender in this study (presently binary due to the manner in
which gender was categorized in the target datasets), there are other
variables that researchers have aimed to preserve privacy for, e.g., age, race
etc. While we do not consider these variables in our study to the expansive
scope, we collate some sources that can be used to inform development of these
word lists for other variables. Differences based on age in linguistic choices
and perception have been extensively studied in which provides pointers to
creating these lists, including words such as (Eckert 2017). Similarly, race
based linguistic differences have been documented in various population level,
group level and case studies. These papers and books also provide a good
source for keywords, such as (Rickford 2016), that can be used to create a ‘
_prior knowledge_ ’ based list.

###  6.2 Model Explanations

We use the Captum interpretability library for PyTorch (Kokhlikyan et al.
2020). Captum provides state-of-the-art algorithms to identify how the input
features, hidden neurons, and layers contribute to a model’s output. We use
the attribution algorithms implemented via integrated gradients (Sundararajan,
Taly, and Yan 2017). Integrated gradients represent the integral of gradients
with respect to inputs along the path from a given baseline (absence of the
cause) to input sample. The integral can be approximated using a Riemann Sum
or Gauss Legendre quadrature rule. The output is a set of words for each model
instance that contribute towards the prediction along with their attribution
weights. We refer to these sets as attribution word sets.

We will compare these sets from pairs of models and, in the rest of the paper,
we will refer to differences between the sets as the  _model attribution
difference_. It has two parts, the first is a weighted sum of the words the
privacy preserving model avoids that are correlated to gender when compared to
the general model, and, the second is a penalty for including words that are
correlated to gender that weren’t included in the general model (Section 5.1).
We calculate the Emotion-Privacy (EP) metric, which is an extension of general
expectation overlap, which captures the overlap of salient words with the list
of ‘expected’ words, often used to measure agreement. We extend our EP metric
to not only capture the aforementioned overlap, but also be able to attribute
change in overlap to the particularly introduced intervention (in this paper,
privacy preservation) to provide meaningful evidence towards approaching
causality. The EP metric calculation is based on four components: (i) the
number of samples in the model ($N$), (ii) the explanation set obtained from
the generally trained model ($E(M_{g})$) and the comparison model
($E(M_{x})$), (iii) the number of words in the explanation set produced by the
generally trained model ($G(M_{g})$), and (iv) the above created list of
implicitly gender-biased perception words ($L$) as:

$\displaystyle G(M_{g})$ $\displaystyle=$
$\displaystyle\left\lvert\bigcup_{j\in L}\\{\lvert w_{j}\rvert:w_{j}\in
E(M_{g})\\}\right\rvert$ (1) $\displaystyle E_{d}$ $\displaystyle=$
$\displaystyle\left(E(M_{g})-E(M_{x})\right)\cap L\hfill$ $\displaystyle
E_{a}$ $\displaystyle=$ $\displaystyle\left(E(M_{x})-E(M_{g})\right)\cap
L\hfill$ EP $\displaystyle=$
$\displaystyle{\frac{1}{N}}\sum_{i=1}^{N}\frac{\left(\sum_{j=1}^{\lvert
E_{d}\rvert}\lvert w_{j}\rvert-\sum_{j=1}^{\lvert E_{a}\rvert}\lvert
w_{j}\rvert\right)}{G(M_{g})}$

To show the versatility of the metric, we also show a variant of the EP
metric, as, emotion-generalization (EG), that captures the degree and
direction of overlap of the salient words with the list of expected words. We
perform a linguistic analysis on the samples that the model gets right for the
new dataset, vs. the ones it gets wrong, to investigate the difference in the
generalizability of the model when either using or not using the EG metric. We
use the National Research Council Canada - Valence, Arousal, and Dominance
(NRC-VAD) Lexicon (Mohammad 2018) corpus consisting of 20,000 unigrams that
are annotated for the emotion axes.

Conventionally, generalizability is assessed by training the model on one
dataset and testing on another for the same task. We calculate the probability
that sample $i$ relies on spurious correlations ($Px_{i,sc}$) as the ratio of
words in the machine explanation that are correlated in the wrong direction
with the words in NRC-VAD over the total number of words. The probability of a
model relying on spurious correlation (${{EG}_{not}}$) is the average of all
sample probabilities. Consequentially, Emotion-Generalization ($EG$) is the
estimated generalization capability of a model for emotion recognition.

$\displaystyle Px_{i,sc}={\frac{1}{sN}}(c_{n}\left(\lvert
w_{j}\rvert)\right)+c_{p}\left(\lvert w_{j}\rvert\right)$ (2)
$\displaystyle{{EG}_{not}}={\frac{1}{N}}\sum_{i=1}^{N}Px_{i,sc}$
$\displaystyle EG=1-{{EG}_{not}}$

where $sN$ is the number of words in the sample, $c_{n}$ are words in the
model explanation for a sample that are negatively attributed to prediction,
and have the same binned value in NRC-VAD as ground truth label, $c_{p}$ are
words in the model explanation for a sample that are positively attributed to
prediction, and have the opposite binned value in NRC-VAD as ground truth
label, $w_{j}$ is the weight of the attribution for word $j$ as obtained from
CAPTUM integrated gradients, $N$ is the number of samples correctly predicted
by the model only trained for the primary task of emotion recognition.

We compare the $EG$ for samples that the generally trained model correctly
classifies. This allows us to create a baseline system from which we can
understand the increase in spurious correlations. This will also allow us to
understand how increases in spurious correlations lead to performance
differences both within and across datasets.

Table 1: Results for valence (Val) prediction on IEMOCAP, _I_ , and MuSE, _M_. The model names on the left refer to the models trained in their corresponding as described in Section 5. $EP$ is the proposed metric to corrleate interpretation difference with curated lists (Section 6.2). $EG$ is estimated generalization ability of the model. C(Em) and C(Pv) refer to the proportion of people who showed preference for a particular model for the task of emotion recognition and privacy preservation respectively. Random, Low and High EP refer to the sample categories as described in Section 7.3 Bold shows significant improvement.Higher values for valence, $EP$ and $EG$ are better, and lower value for gender is better.. Significance is established using paired t-test, adjusted p-value$<0.05$.  | IEMOCAP | MuSE | Proposed Metric | Random EP | High EP | Low EP
---|---|---|---|---|---|---
| V-UAR | G-UAR | V-UAR | G-UAR | _EP_ | _$EG$_ | _C(Em)_ | _C(Pv)_ | _C(Em)_ | _C(Pv)_ | _C(Em)_ | _C(Pv)_
General | 0.648 | 0.711 | 0.577 | 0.742 | - | 0.65 | 0.35 | 0.1 | 0.3 | 0.06 | 0.46 | 0.28*
PrivAdv | 0.671 | 0.562 | 0.532 | 0.622 | 0.653 | 0.62 | 0.30 | 0.33 | 0.36 | 0.38 | 0.24 | 0.24*
PrivAug | 0.636 | 0.551 | 0.591 | 0.643 | 0.680 | 0.68 | 0.26 | 0.39 | 0.31 | 0.43 | 0.18 | 0.21
PrivBias | 0.620 | 0.621 | 0.528 | 0.688 | 0.563 | 0.60 | 0.02 | 0.12 | 0.01 | 0.10 | 0.06 | 0.15
GenderCtrl | 0.631 | 0.820 | 0.517 | 0.756 | -0.210 | 0.59 | 0.09 | 0.06 | 0.02 | 0.03 | 0.06 | 0.11

##  7 Experimental Setup

We analyze the performance of these models within and cross-dataset, and then
from the perspective of user preference. We use Amazon Mechanical Turk (MTurk)
for crowdsourcing annotations to understand how the combination of model
explanations and predictions shape evaluators’ trust as measured by their
selection of a specific model for a given task amongst the choices mentioned
in Section 7.3.

###  7.1 Model Construction

We use IEMOCAP to train the emotion recognition models in a speaker
independent training and testing paradigm. We train six different models on
the original train subset of the IEMOCAP dataset: {PrivAdv: Privacy
Preservation using Adversarial Training, PrivAug: Privacy Preservation using
Data Augmentation, PrivBias: Privacy Preservation using Bias Fine-Tuning,
GenderExp: Explicitly training for gender as a multi-task classification
problem, Gen: Generally trained model for just emotion recognition, and,
ArtNoise: Generally trained model on artificially introduced noisy data}.

###  7.2 Model Generalizability

Previous research has shown that multi-objective training on one dataset, with
poor choices of the tasks, might lead to poorer domain adaptation, because the
model tends to rely more on spurious correlations (Sagawa et al. 2020).
Therefore, we look at how multiple privacy preservation methods, perform when
tested on a different dataset, both for gender and emotion prediction. We
train the model on the IEMOCAP training set and then test the model on the
MuSE dataset.

###  7.3 Crowdsourcing Experiment

We show the predictions and explanations for a subset of the IEMOCAP testing
partition in a crowdsourcing experiment. We aim to select a representative
sample population by considering various levels of valence, as well as, gender
for a total of six bins. We select 20 random samples from each of the six
classes to use as our baseline and then choose additional 30 random samples
each from the set of samples that have a high EP score and those that have a
low EP score as obtained in Section 6, for a total 540 samples.

We extract both explanations and predictions using Captum for each the 540
samples, using each of the five models (PrivAdv, PrivAug, PrivBias,
GenderCtrl, ArtNoise, described above). The ArtNoise is used as a control
baseline for human evaluation, ensuring that the evaluators are paying
attention to the task at hand. We consider this option as an attention check
and discard any response where the crowdsourced worker preferred the result
and/or the explanation from this model (7.31% samples were discarded in
total).

We recruited annotators using MTurk from a population of workers with the
following characteristics: 1) $>98$% approval rating, 2) $>$ 500 approvals, 3)
in the United States, and 4) native English speakers. Each Human Intelligence
Task (HIT) was annotated by three workers. We ensured that all workers
understood the meaning of valence using the popular qualification tests. Each
HIT took an average of one-minute. The compensation was $\$9.45/$hr. We
present the predictions from these models and heat map based explanations to
the workers and ask them to choose between these five models for each of these
three questions: 1) Which model are you more likely to trust for emotion
prediction? 2) Which model are you more likely to trust for privacy
preservation?

##  8 Results and Discussion

###  8.1 RQ1: Quantifying Interpretation Differences

Hypothesis: The differences in salient tokens obtained for general models vs
those obtained for privacy preserving models should show avoidance of gender-
related terms.
Reason: Previous researches have shown how inducing privacy preservation in a
natural language avoids learning tokens indicative of gender (Baron and
Musolesi 2020).
Result: We show the shifts in the performance between models that are trained
just for emotion recognition (General, Table 1) and those that are trained for
emotion recognition and privacy preservation (adversarial training, PrivAdv,
augmentation, PrivAug, bias fine-tuning, PrivBias, in Table 1).

We first examine the change in the performance of the within-IEMOCAP valence
classification task (column _I_ Valence UAR in Table 1, chance is 0.33). We
find that while there is a drop in valence prediction performance, moving from
the General model to the privacy-preserving models, the difference lies in
between $\pm$2%. We also observe that while the General model can be used to
recognize gender, the privacy-preserving models obstruct this information
(column _I_ Gender UAR in Table 1, chance is 0.5). We find that the top 65% of
the intersection between the EP and the implicit bias lists come from LIWC
category-based words (for example: words implying hesitation, vagueness) and
the other from IAT proposed lists (for e.g.: words implying self-confidence,
arrogance) which indicates the usefulness of incorporating expert knowledge in
model evaluation.

###  8.2 RQ2: Crowdsourcing Validation of EP Metric

Hypothesis: EP is significantly correlated with evaluators’ perception of the
ability of model for privacy preservation.
Reason: Previous researches have shown that humans perceive a model as more
trustworthy and capable if its explanations align with how they would form
their own judgments (Schmidt and Biessmann 2019). Given that the metric is
intentionally aimed to capture this information using expert curation, it
should be indicative of evaluators’ judgment.
Result: Table 1 shows 2 values obtained from crowdsourcing, segmented by EP
ranges: 1) T(V) - the percentage of people who reported that they trust the
model _X_ to predict valence, 2) T(Priv) - the percentage of people who trust
the model _X_ to preserve privacy. To answer this particular question, we
analyze how evaluators choose the model that they are most likely to trust
with their data for the purpose of evaluating perception of privacy
preservation [T(Priv)]. The two privacy preservation methods, i.e.,
adversarial training and augmentation, are chosen 30% and 26% of the time,
respectively. When asked to focus on privacy preservation the annotators
choose in most of the cases. To test the effectiveness of EP as a proxy for
relative model choice, we evaluate whether the evaluators’ model choices for
privacy preservation change as a factor of EP. Table 1 shows that there is a
positively correlated trend between the calculated EP metric and the
likelihood of a particular model being preferred. We also show how EP
correlates with a random sampling of crowdsourced preferences.

###  8.3 RQ3: Comparing the EP and General Metrics

Hypothesis: A metric that uses sociological information to integrate the
degree of alignment and misalignment with known gender-biased tokens will also
align with model artifacts that can be exploited to infer gender.
Reason: Unintentional capture of gender-based information in models is usually
an artifact of the data used for model training (Sun et al. 2019). Given that
the emotion dataset used is collected from a subset of human population, the
gender biased artifacts should be captured by the proposed metric which aims
to align with known sociological indicators of demographics.
Result: We calculate EP model interpretation difference as defined in Section
6.2. We expect GenderExp to have the lowest EP score, because it should
explicitly capture gender-based information, which is should be highly
correlated with the bias list described in Section 6. We find that this model
has a EP score of -.21 (Table 1), thus supporting the claim that our metric
captures the perception of gender. We find that augmentation methods (PrivAug)
are most similar to sociolinguistic gender biases, as measured by EP. However,
we remind that both the creation of the paired augmentation lists and the
resulting training time can be burdensome. The adversarial training method
(PrivAdv) has the next closest EP score (relative difference of 3.97%) but
does not require the additional demographic variable replacement list
annotation effort and maintains the same training time.

To show that this metric can be used as a stand-in for human judgment while
still capturing human perception, we train the model (Sec 5.1) on IEMOCAP
dataset for valence generally, just optimizing for cross-entropy loss for
valence prediction. For the privacy preservation model comparison, we use the
adversarial variant of the model (Sec 5.2), because it relies only on the
knowledge of the gender label (rather than replacement lists etc.). We
incorporate our metric for privacy preservation model by two methods, (i)
replacement, in which we completely remove the adversarial component of the
model, and instead, use a weighted loss for maximizing both EP and Valence
UAR, and, (ii) _addition_ , in which we retain the adversarial gender
classification component, while adding an additional loss optimization for
maximizing EP. When training and testing on IEMOCAP, the replacement method
yields a valence UAR of 0.643 and gender UAR of 0.582, whereas the _addition_
method gives a valence UAR of 0.672 and gender UAR of 0.548.

###  8.4 RQ4: Versatility of the EP Metric

#### Privacy Preservation Generalization Correlation with Emotion-Privacy
Metric (EP)

Hypothesis: Models that have higher EP score would generalize better in their
ability for privacy preservation.
Reason: Privacy preserving ability of models across datasets is has been found
to be dependent on using a sample set that ideally has all associations for
any demographic variable (Sun et al. 2019). Because EP is dependent on an
externally curated list of such assosications by experts, rather than just
relying on internally learnt associations in a black box model, it should be
indicative of a model’s performance on out of distribution dataset when
presented with ‘unseen’ gender-biased artificats.
Results: In Table 1, we show the generalizability of the models, training on
IEMOCAP or MSP-Improv+IEMOCAP (for bias fine-tuning) and testing on MuSE. We
find that the generalizability of the models with respect to both the
performance of the emotion recognition (column _M_ Valence UAR) and efficacy
of the privacy preservation (column _M_ Gender UAR) is affected by the method
of privacy preservation. For cross corpus performance for privacy preservation
(gender UAR), we find that adversarial training paradigms are most effective
for continuing to mask gender out of training distribution, compared to other
privacy preserving methods. This is in contrast to the within-corpus setting,
where data augmentation was found to be most effective.This may be because
adversarial training method can learn additional replicable gender-based
correlation patterns that do not agree or are unobserved in demographic
studies. For future work it would be interesting to see whether these repeated
patterns are dataset artifacts or new gateways into presently unknown
associations. We see a maximal drop in privacy preservation performance for
PrivBias model (highest increase in attacker’s ability to predict gender on
MuSE use the initially trained model), which corresponds with it having the
lowest _EP_ value amongst all privacy preservation methods.

To show that the metric is a good stand-in or supplement to privacy
preservation method, we train the two models as mentioned in Section 8.3 that
introduce privacy either by _replacement_ or by _addition_. When using these
models trained on IEMOCAP, but testing on MuSE dataset, we obtain a valence
UAR of 0.563 along with a gender UAR of 0.611, and, a valence UAR of 0.581
along with a gender UAR of 0.587 for the replacement and _addition_ setup
respectively. We see a significantly substantial decrease in performance of
the _addition_ setup model when predicting gender on out of distribution data,
demonstrating the utility of the metric for enhanced privacy preservation
while still maintaining the alignment with human perception.

#### Emotion-Generalization (EG): Correlation with Emotion Recognition
Generalization

Hypothesis: EG has a positive relationship with performance of a model for
cross-corpus emotion recognition.
Reason: Studies have shown that emotion recognition models have better
generalization performance if they have a diverse dataset that is
representative of the general emotion perception, and that the model learns
these generalized patterns (Kaushik, Hovy, and Lipton 2019). Because EG
measures the directional amount of overlap between saliency (learnt patterns)
and known perception (NRC-VAD), it should be indicative of a model’s
generalization capability.
Result: As mentioned above, in Table 1, we see that efficacy of emotion
recognition is affected when inducing privacy preservation. We find a decline
in emotion recognition accuracy across corpus when inducing privacy seeing the
maximal drop (10%) in case of bias fine-tuning. We find a positive
relationship between the probability that a model does not rely on spurious
correlations in IEMOCAP, $EG$, and the cross-corpus UAR obtained on the MuSE
dataset for the emotion recognition task (Table 1). For example, in Table 1,
we show that EG has the highest value for Privacy Augmentation modeling along
with having the highest performance for valence prediction on MuSE to validate
out of data distribution.

##  9 Conclusion

We study how interpretation mechanisms can be used to evaluate privacy
preserving models. and how they can be used to inform metric design that
correlates with humans’ perception of model’s ability for a given task. We
analyse patterns in interpretation differences that are introduced due to an
additional constraint of privacy preservation. and look at how this difference
can be directionally correlated with known sociological indicators of gender
to inform our metric design. We conduct crowdsourcing studies to validate that
this metric is representative of humans’ perception of model’s ability for
privacy preservation. Finally, we show how the same metric design concept can
be used for measuring generalization capability for both emotion recognition
performance and privacy preservation in cross-corpus settings.

## References

  * Baron and Musolesi (2020) Baron, B.; and Musolesi, M. 2020.  Interpretable machine learning for privacy-preserving pervasive systems.  _IEEE Pervasive Computing_ , 19(1): 73–82.
  * Belinkov and Glass (2019) Belinkov, Y.; and Glass, J. 2019.  Analysis methods in neural language processing: A survey.  _Transactions of the Association for Computational Linguistics_ , 7: 49–72.
  * Bengio, Courville, and Vincent (2013) Bengio, Y.; Courville, A.; and Vincent, P. 2013.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Bolukbasi et al. (2016) Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and Kalai, A. T. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _Advances in neural information processing systems_.
  * Brescoll (2016) Brescoll, V. L. 2016.  Leading with their hearts? How gender stereotypes of emotion lead to biased evaluations of female leaders.  _The Leadership Quarterly_ , 27(3): 415–428.
  * Busso et al. (2008) Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_.
  * Busso et al. (2016) Busso, C.; Parthasarathy, S.; Burmania, A.; AbdelWahab, M.; Sadoughi, N.; and Provost, E. M. 2016.  MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception.  _IEEE Transactions on Affective Computing_ , 8(1): 67–80.
  * Carlini et al. (2019) Carlini, N.; Liu, C.; Erlingsson, Ú.; Kos, J.; and Song, D. 2019.  The Secret Sharer: Evaluating and testing unintended memorization in neural networks.  In _28th $\\{$USENIX$\\}$ Security Symposium)_.
  * Corbett-Davies and Goel (2018) Corbett-Davies, S.; and Goel, S. 2018.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _arXiv preprint arXiv:1808.00023_.
  * Davidson, Bhattacharya, and Weber (2019) Davidson, T.; Bhattacharya, D.; and Weber, I. 2019.  Racial Bias in Hate Speech and Abusive Language Detection Datasets.  _arXiv preprint arXiv:1905.12516_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Eckert (2017) Eckert, P. 2017.  Age as a sociolinguistic variable.  _The handbook of sociolinguistics_ , 151–167.
  * Engler (2021) Engler, A. 2021.  Auditing employment algorithms for discrimination.  _Brookings Institute, Center for Technology Innovation_.
  * Ferrario, Loi, and Viganò (2019) Ferrario, A.; Loi, M.; and Viganò, E. 2019.  In AI we trust incrementally: a multi-layer model of trust to analyze human-artificial intelligence interactions.  _Philosophy & Technology_, 1–17.
  * Fosch-Villaronga et al. (2021) Fosch-Villaronga, E.; Poulsen, A.; Søraa, R. A.; and Custers, B. 2021.  A little bird told me your gender: Gender inferences in social media.  _Information Processing & Management_, 58(3): 102541.
  * Ganin and Lempitsky (2014) Ganin, Y.; and Lempitsky, V. 2014.  Unsupervised domain adaptation by backpropagation.  _arXiv preprint arXiv:1409.7495_.
  * Gerke, Minssen, and Cohen (2020) Gerke, S.; Minssen, T.; and Cohen, I. G. 2020.  Ethical and Legal Challenges of Artificial Intelligence-Driven Health Care.  _Forthcoming in: Artificial Intelligence in Healthcare, 1st edition, Adam Bohr, Kaveh Memarzadeh (eds.)_.
  * Heuer and Breiter (2020) Heuer, H.; and Breiter, A. 2020.  More Than Accuracy: Towards Trustworthy Machine Learning Interfaces for Object Recognition.  In _Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization_ , 298–302.
  * Iosifidis and Ntoutsi (2018) Iosifidis, V.; and Ntoutsi, E. 2018.  Dealing with bias via data augmentation in supervised learning scenarios.  _Jo Bates Paul D. Clough Robert Jäschke_ , 24.
  * Jaiswal et al. (2020) Jaiswal, M.; Bara, C.-P.; Luo, Y.; Burzo, M.; Mihalcea, R.; and Provost, E. M. 2020\.  MuSE: a Multimodal Dataset of Stressed Emotion.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , 1499–1510.
  * Jaiswal and Provost (2020) Jaiswal, M.; and Provost, E. M. 2020.  Privacy Enhanced Multimodal Neural Representations for Emotion Recognition.  In _AAAI_ , 7985–7993.
  * Kaushik, Hovy, and Lipton (2019) Kaushik, D.; Hovy, E.; and Lipton, Z. C. 2019.  Learning the difference that makes a difference with counterfactually-augmented data.  _arXiv preprint arXiv:1909.12434_.
  * Kim, Khanna, and Koyejo (2016) Kim, B.; Khanna, R.; and Koyejo, O. O. 2016.  Examples are not enough, learn to criticize! criticism for interpretability.  In _Advances in Neural Information Processing Systems_ , 2280–2288.
  * Kokhlikyan et al. (2020) Kokhlikyan, N.; Miglani, V.; Martin, M.; Wang, E.; Alsallakh, B.; Reynolds, J.; Melnikov, A.; Kliushkina, N.; Araya, C.; Yan, S.; et al. 2020.  Captum: A unified and generic model interpretability library for pytorch.  _arXiv preprint arXiv:2009.07896_.
  * Krishna et al. (2020) Krishna, K.; Khosla, S.; Bigham, J. P.; and Lipton, Z. C. 2020.  Generating SOAP Notes from Doctor-Patient Conversations.  _arXiv preprint arXiv:2005.01795_.
  * Liu et al. (2021) Liu, B.; Ding, M.; Shaham, S.; Rahayu, W.; Farokhi, F.; and Lin, Z. 2021.  When machine learning meets privacy: A survey and outlook.  _ACM Computing Surveys (CSUR)_ , 54(2): 1–36.
  * Meng et al. (2018) Meng, Z.; Li, J.; Chen, Z.; Zhao, Y.; Mazalov, V.; Gang, Y.; and Juang, B.-H. 2018\.  Speaker-invariant training via adversarial learning.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 5969–5973. IEEE.
  * Mohammad (2018) Mohammad, S. 2018.  Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 174–184.
  * Newman et al. (2008) Newman, M. L.; Groom, C. J.; Handelman, L. D.; and Pennebaker, J. W. 2008.  Gender differences in language use: An analysis of 14,000 text samples.  _Discourse Processes_ , 45(3): 211–236.
  * Nushi, Kamar, and Horvitz (2018) Nushi, B.; Kamar, E.; and Horvitz, E. 2018.  Towards accountable ai: Hybrid human-machine analyses for characterizing system failure.  _arXiv preprint arXiv:1809.07424_.
  * Pliev (2019) Pliev, G. 2019.  How Emotion AI Can Transform Large-Scale Recruitment Processes.  Accessed: 10-21-2019.
  * Preece et al. (2018) Preece, A.; Harborne, D.; Braines, D.; Tomsett, R.; and Chakraborty, S. 2018.  Stakeholders in explainable AI.  _arXiv preprint arXiv:1810.00184_.
  * Pruthi et al. (2019) Pruthi, D.; Gupta, M.; Dhingra, B.; Neubig, G.; and Lipton, Z. C. 2019.  Learning to Deceive with Attention-Based Explanations.  _arXiv preprint arXiv:1909.07913_.
  * Rickford (2016) Rickford, J. R. 2016.  _Raciolinguistics: How language shapes our ideas about race_.  Oxford University Press.
  * Rosenberg (2012) Rosenberg, A. 2012.  Classifying skewed data: Importance weighting to optimize average recall.  In _Thirteenth Annual Conference of the International Speech Communication Association_.
  * Sagawa et al. (2020) Sagawa, S.; Raghunathan, A.; Koh, P. W.; and Liang, P. 2020.  An Investigation of Why Overparameterization Exacerbates Spurious Correlations.  _arXiv preprint arXiv:2005.04345_.
  * Schmidt and Biessmann (2019) Schmidt, P.; and Biessmann, F. 2019.  Quantifying interpretability and trust in machine learning systems.  _arXiv preprint arXiv:1901.08558_.
  * Sperrle et al. (2019) Sperrle, F.; Schlegel, U.; El-Assady, M.; and Keim, D. 2019.  Human Trust Modeling for Bias Mitigation in Artificial Intelligence.  In _ACM CHI 2019 Workshop: Where is the Human? Bridging the Gap Between AI and HCI_.
  * Sun et al. (2019) Sun, T.; Gaut, A.; Tang, S.; Huang, Y.; ElSherief, M.; Zhao, J.; Mirza, D.; Belding, E.; Chang, K.-W.; and Wang, W. Y. 2019.  Mitigating gender bias in natural language processing: Literature review.  _arXiv preprint arXiv:1906.08976_.
  * Sundararajan, Taly, and Yan (2017) Sundararajan, M.; Taly, A.; and Yan, Q. 2017.  Axiomatic attribution for deep networks.  _arXiv preprint arXiv:1703.01365_.
  * Swinger et al. (2019) Swinger, N.; De-Arteaga, M.; Heffernan IV, N. T.; Leiserson, M. D.; and Kalai, A. T. 2019.  What are the biases in my word embedding?  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , 305–311.
  * Szegedy et al. (2013) Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; and Fergus, R. 2013.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_.
  * Wang and Chang (2020) Wang, S.; and Chang, J. M. 2020.  Privacy-Preserving Image Classification in the Local Setting.  _arXiv preprint arXiv:2002.03261_.
  * Wolf et al. (2019) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ , arXiv–1910.