# Disentangled Variational Autoencoder for Emotion Recognition in
Conversations

Kailai Yang, Tianlin Zhang, Sophia Ananiadou Kailai Yang, Tianlin Zhang, and
Sophia Ananiadou are with NaCTeM and the Department of Computer Science, The
University of Manchester, United Kingdom.
E-mail<EMAIL_ADDRESS>
E-mail<EMAIL_ADDRESS>
E-mail<EMAIL_ADDRESS>
###### Abstract

In Emotion Recognition in Conversations (ERC), the emotions of target
utterances are closely dependent on their context. Therefore, existing works
train the model to generate the response of the target utterance, which aims
to recognise emotions leveraging contextual information. However, adjacent
response generation ignores long-range dependencies and provides limited
affective information in many cases. In addition, most ERC models learn a
unified distributed representation for each utterance, which lacks
interpretability and robustness. To address these issues, we propose a VAD-
disentangled Variational AutoEncoder (VAD-VAE), which first introduces a
target utterance reconstruction task based on Variational Autoencoder, then
disentangles three affect representations Valence-Arousal-Dominance (VAD) from
the latent space. We also enhance the disentangled representations by
introducing VAD supervision signals from a sentiment lexicon and minimising
the mutual information between VAD distributions. Experiments show that VAD-
VAE outperforms the state-of-the-art model on two datasets. Further analysis
proves the effectiveness of each proposed module and the quality of
disentangled VAD representations. The code is available at
https://github.com/SteveKGYang/VAD-VAE.

###### Index Terms:

Emotion Recognition in Conversations, Variational Autoencoder, Valence-
Arousal-Dominance, Disentangled Representations.

##  1 Introduction

Emotion Recognition in Conversations (ERC) aims to identify the emotion of
each utterance within a dialogue from pre-defined emotion categories [1]. As
an extension of traditional emotion detection from text, ERC attracts
increasing research interest from the NLP community since it is more suitable
for usage in real-world scenarios. For example, ERC aids dialogue systems in
generating emotionally coherent and empathetic responses [2]. It has also been
widely utilised in emotion-related social media analysis [3, 4] and opinion
mining from customer reviews [5, 6].

Unlike sentence-level emotion recognition, the emotion of each utterance is
dependent on contextual information in ERC. Some works enhance the context
modelling ability by training the model to reconstruct the dialogue. For
example, [7, 8] pre-train the utterance encoders on large-scale conversation
data and transfer the pre-trained weights to ERC. More recent works utilise
Pre-trained Language Models (PLMs) [9] to model the dialogue and avoid pre-
training from scratch [10, 11]. [12] combine both methods by fine-tuning pre-
trained BART [13] with an auxiliary response generation task on the dialogue,
which trains the model to generate the next sentence given the target
utterance. This task aims to force the model to recognise emotions considering
context information.

Figure 1: A dialogue example. Solid lines show the influence of previous
utterances on the emotion of the target utterance (marked blue), and dashed
lines denote the change of the discussion topic in next utterance. As
illustrated, the emotion of the target utterance is dependent on long-range
history. As the topic changes, response generation provides limited affective
information.

However, response generation only mines the dependencies between two adjacent
utterances, while the influence of long-range history on the target utterance
is ignored. Generating the next sentence also provides limited affective
information for the target utterance in many cases, such as the sudden change
of the discussion topic. An example is shown in Figure 1 to prove the above
points. As illustrated, the emotion of the target utterance is dependent on
long-range history, and response generation provides limited affective
information as the discussion topic changes in the next utterance. We argue
that a context-aware reconstruction of the target utterance itself is more
appropriate since ERC is centred on the target utterance representations. In
addition, current ERC methods mostly learn a unified distributed
representation for each target utterance. Though achieving impressive results,
entangled features are proved to lack interpretability and robustness [14].
Affective text generation models [2, 15] also postulate that emotions are
independent of the content they modify, and their success shows the viability
of disentangling emotion features from content representations.

To address these issues, we propose a VAD-disentangled Variational AutoEncoder
(VAD-VAE) for ERC. Firstly, instead of generating the response, we introduce a
target utterance reconstruction task based on the Variational Autoencoder
(VAE) [16] generative model. We devise a PLM-based context-aware encoder to
model the dialogue, and sample the latent representations from a Gaussian
distribution estimated from the utterance representations. The Gaussian
distribution also aims to regularise the latent space. Then another PLM-based
decoder is leveraged to reconstruct the target utterance from the latent
representations. VAD-VAE outperforms the state-of-the-art model on two ERC
datasets.

Secondly, we utilise disentangled representation learning [17] techniques to
disentangle critical features from the utterance representations. Studies in
affect representation models in psychology point out that Valence-Arousal-
Dominance (VAD) are both orthogonal and bipolar, which are appropriate to
define emotion states [18, 19]. Therefore, we propose to disentangle the three
VAD features from the latent space of the VAE, where we also sample each of
the VAD representations from a corresponding Gaussian distribution estimated
from the utterance representations. Then the disentangled features are
combined for both ERC and target utterance reconstruction tasks.

Thirdly, two techniques are used to enhance the disentangled VAD
representations. We boost their _informativeness_ [20] by introducing
supervision signals from NRC-VAD [21], a sentiment lexicon that contains human
ratings of VAD for all emotions. To enforce the _independence_ [17] of latent
spaces, we minimise the Mutual Information (MI) between VAD representations.
During training, we estimate and minimise the variational Contrastive Log-
ratio Upper-Bound (vCLUB) [22] of MI. Further analysis proves the quality of
the disentangled VAD representations.

To summarise, this work mainly makes the following contributions:

  * •

We propose a VAE-based target utterance reconstruction auxiliary task for ERC,
which improves model performance and regularises the latent spaces.

  * •

For the first time in ERC, We explicitly disentangle the three VAD features
from the utterance representations. Analysis shows it benefits
interpretability and robustness, and bears potential in the affective text
generation task.

  * •

We enhance the _informativeness_ of the disentangled representations with VAD
supervision signals from the lexicon NRC-VAD and minimise the vCLUB estimate
of their mutual information to improve _independence_.

##  2 Related Work

###  2.1 Emotion Recognition in Conversations

For ERC, the emotion of a dialogue participant is primarily influenced by the
dialogue history, which makes context modelling a key challenge. Early works
utilised Recurrent Neural Networks (RNN) to model each participant’s dialogue
flow as a sequence and revise them as memories at each time step [23, 24].
Considering multi-party relations, [25] leveraged another global-state RNN to
model inter-speaker dependencies and emotion dynamics. To avoid designing
complex model structures, more recent works leveraged the strong context-
modelling ability of PLMs to model the conversation as a whole [26, 10]. Some
other works [27, 28] built a graph upon the dialogue with each utterance as a
node and used graph neural networks to model ERC as a node-classification
task.

Enhancing the utterance representations is also crucial for ERC. Some works
managed to incorporate task-related information. For example, commonsense
knowledge was introduced [29, 11] to enrich the semantic space. To enhance the
conversation modelling ability, some methods pre-trained the model on large-
scale conversation data and transferred the weights to ERC [8, 7]. Multi-task
learning was also leveraged to introduce topic information [30, 31], discourse
roles [32] and speaker-utterance relations [26] to aid emotion reasoning. [33,
34] incorporated VAD information to introduce fine-grained sentiment
supervision. Contrastive learning [12, 35, 36] was also devised to distinguish
utterances with similar emotions.

###  2.2 Disentangled Representation Learning

Disentangled Representation Learning (DRL) aims to map key data features into
distinct and independent low-dimensional latent spaces [17]. Current DRL
methods are mainly divided into unsupervised and supervised disentanglement.
Early unsupervised methods mainly designed constraints on the latent space to
enforce the independence of each dimension, such as information capacity [37]
and mutual information gap [38]. Supervised methods introduced supervision
signals to different parts of the latent space to enforce _informativeness_.
Some works utilised ground-truth labels of the corresponding generative
factors, such as syntactic parsing trees [39] and style labels [40]. In
contrast, other works used weakly-supervised signals, including pairwise
similarity between representations [41] and semi-supervised ground-truth
labels [42]. Still, supervised methods devised techniques such as mutual
information minimisation [42] and adversarial learning [39, 43] to enforce
_independence_ and _invariance_ [44] of the disentangled representations.

##  3 Methodology

###  3.1 Task Definition

The ERC task is defined as follows: a dialogue $D$ contains $n$ utterances
$\\{u_{1},u_{2},...,u_{n}\\}$, with the corresponding ground-truth emotion
labels $\\{y_{1},y_{2},...,y_{n}\\}$, where $y_{i}\in E$, $E$ is the pre-
defined emotion label set. Each $u_{i}$ contains $m_{i}$ tokens:
$\\{u_{i}^{1},u_{i}^{2},...,u_{i}^{m_{i}}\\}$. The dialogue is also
accompanied by a speakers list $S(D)=\\{S(u_{1}),S(u_{2}),...,S(u_{n})\\}$,
where $u_{i}$ is uttered by $S(u_{i})\in S$, and $S$ is the set of dialogue
participants. With the above information, ERC aims to identify the emotion of
each target utterance $u_{i}$, which is formalised as:
$\hat{y}_{i}=f(u_{i},D,S(D))$.

###  3.2 Target Utterance Reconstruction

This section introduces the target utterance reconstruction auxiliary task.
Based on the context-aware utterance encoder, we also disentangle VAD latent
representations from the utterance representations and build a VAE-based
generative model to reconstruct the target utterance, which is the backbone of
VAD-VAE, as illustrated in Figure 2.

Figure 2: Main components of VAD-VAE. Each latent representation
$z^{(\mathcal{R})}$ is sampled from a Gaussian distribution estimated from the
context-aware utterance representation $r$. The VAD prediction
$P^{(\mathcal{R})}$ is obtained from $z^{(\mathcal{R})}$ via the VAD
predictors. The concatenated representation $z$ is utilised for both ERC and
target utterance reconstruction. In the utterance decoder, “$\langle
cls\rangle$” denotes the start-of-sentence token, and “$\langle eos\rangle$”
denotes the end-of-sentence token.

####  3.2.1 Context-Aware Utterance Encoder

To introduce speaker information explicitly, we prepend the speaker $S(u_{j})$
to each utterance: $u_{j}=\\{S(u_{j}),u_{j}^{1},u_{j}^{2},...,u_{j}^{k}\\}$,
where $u_{j}^{k}$ is the $k$-th token of $u_{j}$. Then the target utterance
$u_{i}$ is concatenated with both past and future dialogues to obtain the
context-aware input $\hat{u}_{i}$:

$\hat{u}_{i}=\\{\langle cls\rangle;u_{i-W_{p}};...;\langle
sep\rangle;u_{i};\langle sep\rangle;...;u_{i+W_{f}};\langle eos\rangle\\}$

where $\\{;\\}$ denotes concatenation, $W_{p}$ and $W_{f}$ denotes the past
and future context window size, $\langle cls\rangle$ and $\langle eos\rangle$
denote the start-of-sentence and end-of-sentence token. Two $\langle
sep\rangle$ tokens are added at the start and end place of $u_{i}$ to identify
the target utterance. Utilising a PLM-based encoder, we obtain the context-
aware utterance embeddings:

$r_{i}=Encoder(\hat{u}_{i})$ (1)

where $Encoder$ denotes the RoBERTa-Large [45] utterance encoder,
$r_{i}\in\mathbb{R}^{L\times D_{h}}$ is the utterance representations, $L$
denotes the sequence length, and $D_{h}$ is the hidden states dimension. We
leverage the embedding of the start-of-sentence token at position 0:
$r_{i}^{[CLS]}\in\mathbb{R}^{D_{h}}$ as the utterance-level representation of
$u_{i}$.

####  3.2.2 VAE-based Generative Model

We build a VAE-based generative model and disentangle three latent features,
Valence-Arousal-Dominance (VAD) from the utterance representation, where
Valence reflects the pleasantness of a stimulus, Arousal reflects the
intensity of emotion provoked by a stimulus, and Dominance reflects the degree
of control exerted by a stimulus [46]. We also define a “Content” feature that
controls the content generation of the target utterance.

A VAE is utilised to estimate this model, which imposes a standard Gaussian
prior distribution on each latent space $Z$. The deterministic utterance
representation is replaced with an approximation of the posterior
$q_{\phi}(z|x)$, which is parameterised by a neural network. We utilise four
feed-forward neural networks to map $x=r_{i}^{[CLS]}$ to four sets of Gaussian
distribution parameters $(\mu,\sigma)$, which parameterise the latent
distributions of Valence, Arousal, Dominance, and Content, denoted as
$\mathcal{R}\in\\{V,A,D,C\\}$. For each feature, we sample the latent
representation $z^{(\mathcal{R})}$ from the Gaussian distribution defined by
the corresponding $(\mu^{(\mathcal{R})},\sigma^{(\mathcal{R})})$ using the re-
parameterisation trick [16]:

$z^{(\mathcal{R})}_{i}=\mu^{(\mathcal{R})}\odot\sigma^{(\mathcal{R})}+\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
(2)

where $z^{(\mathcal{R})}_{i}\in\mathbb{R}^{D_{(\mathcal{R})}}$,
$D_{(\mathcal{R})}$ is the pre-defined latent space dimension. Then the latent
representations are concatenated:
$z_{i}=[z^{V}_{i};z^{A}_{i};z^{D}_{i};z^{C}_{i}]$. $z_{i}$ is used to
initialise the decoder and reconstruct the target utterance:

$u_{i}^{j}=Softmax(Decoder(z_{i},u_{i}^{<j}))$ (3)

where $Softmax$ denotes the softmax operation, $u_{i}^{j}$ denotes the $j$-th
generated tokens, and $u_{i}^{<j}$ denotes the previously generated tokens. We
utilise BART-Large decoder [13] as $Decoder$, since it shares the vocabulary
with RoBERTa-Large in Huggingface [47] implementations and is proved powerful
in many generative tasks. As in standard VAE, we include a KL-divergence term
for each latent space to keep the approximate posterior close to the prior
distribution. During training, we utilise the Evidence Lower BOund (ELBO) as
the training objective:

$\begin{split}\mathcal{L}_{ELBO}(\phi,\theta)=-\mathbb{E}_{q_{\phi}(z_{i}|x)}\left[log\
p_{\theta}(x|z_{i})\right]+\\\
\sum_{\mathcal{R}\in\\{V,A,D,C\\}}\alpha_{\mathcal{R}}KL\left[q_{\phi}^{(\mathcal{R})}(z_{i}^{(\mathcal{R})}|x)||p(z_{i}^{(\mathcal{R})})\right]\end{split}$
(4)

where $\phi$ and $\theta$ denote the parameters of the encoder and decoder,
each $\alpha_{\mathcal{R}}$ weights the corresponding KL-divergence term, and
standard Gaussian prior is used for each $p(z_{i}^{(\mathcal{R})})$.

###  3.3 Enhancing VAD Representations

We aim to enhance the disentangled VAD representations considering the
following two aspects: (a). _informativeness_ : the representation should
include enough information to predict the corresponding generative factor well
[48, 20]. (b). _Independence_ : for each generative factor, the representation
should lie in an independent latent space [17]. Therefore, we introduce
supervision signals from a sentiment lexicon to enforce _informativeness_ and
a mutual information minimisation objective to enforce _independence_.

####  3.3.1 Informativeness

To enhance the representation’s ability to predict the corresponding
generative factor, we introduce supervision signals from NRC-VAD [21], a VAD
sentiment lexicon that contains reliable human ratings of VAD for 20,000
English terms. All NRC-VAD terms denote or connote emotions and are selected
from commonly used sentiment lexicons and tweets. Each term is strictly
annotated via best-worst scaling with crowdsourcing annotators, and an
aggregation process calculates the VAD for each term ranging from 0 to 1. For
example, the emotion _happiness_ is assigned
$vad_{happiness}=\\{0.960,0.732,0.850\\}$. More details about NRC-VAD are in
Sec. 4.1.

With the pre-defined categorical emotion set $E$, we extract the VAD score
$vad_{e_{j}}=\\{vad_{e_{j}}^{V},vad_{e_{j}}^{A},vad_{e_{j}}^{D}\\}$for each of
the emotion $e_{j}\in E$ from NRC-VAD, where $j\in[1,|E|]$. Since fine-grained
VAD supervision signals are introduced, we expect to improve both the
_informativeness_ of VAD representations and model performance on ERC.

Specifically, for each $\hat{\mathcal{R}}\in\\{V,A,D\\}$, we compute the
corresponding prediction from the latent representation using a feed-forward
neural network predictor, and a sigmoid function is utilised to map the
prediction to the range $(0,1)$:

$P_{i}^{(\hat{\mathcal{R}})}=\frac{1}{1+\emph{e}^{-(z_{i}^{(\mathcal{\hat{R}})}W^{(\mathcal{\hat{R}})}+b^{(\mathcal{\hat{R}})})}}$
(5)

where $W^{(\hat{\mathcal{R}})}$ and $b^{(\hat{\mathcal{R})}}$ are parameters
of the predictor corresponding to $\hat{\mathcal{R}}$. As the training
objective, we compute the mean squared error loss between the predictions and
the supervision signals:

$\mathcal{L}_{INFO}(\phi,\lambda)=\frac{1}{N}\sum_{i=1}^{N}\sum_{\mathcal{\hat{R}}\in\\{V,A,D\\}}(P_{i}^{(\mathcal{\hat{R}})}-vad_{y_{i}}^{(\mathcal{\hat{R}})})^{2}$
(6)

where $\phi$ and $\lambda$ denote the parameters of the encoder and the
predictor, $y_{i}$ denotes the emotion label of $i$-th utterance, and $N$
denotes the batch size.

####  3.3.2 Independence

We improve the independence of all disentangled latent spaces by making their
distributions as dissimilar as possible. A common method is to minimise the
Mutual Information (MI) [49] between each pair of latent variables:
$\mathbf{I}(\mathcal{\hat{R}}_{i};\mathcal{\hat{R}}_{j})$, which is defined
as:

$\begin{split}\mathbf{I}(\mathcal{\hat{R}}_{i};\mathcal{\hat{R}}_{j})=\mathbb{E}_{p(\mathcal{\hat{R}}_{i},\mathcal{\hat{R}}_{j})}\Big{[}log\frac{p(\mathcal{\hat{R}}_{i},\mathcal{\hat{R}}_{j})}{p(\mathcal{\hat{R}}_{i})p(\mathcal{\hat{R}}_{j})}\Big{]}\end{split}$
(7)

where $\mathcal{\hat{R}}_{i}$, $\mathcal{\hat{R}}_{j}\in\\{V,A,D\\}$ and
$i\neq j$. However, MI is hard to calculate in high-dimensional spaces. The
conditional distribution between each pair of latent variables is also
unavailable in our cases. Therefore, we utilise the variational Contrastive
Log-ratio Upper-Bound (vCLUB) proposed by Cheng et al. [22] to estimate and
minimise an upper bound of MI, which is calculated as follows:

$\begin{split}\mathbf{I}_{vCLUB}(\mathcal{\hat{R}}_{i};\mathcal{\hat{R}}_{j}):=\mathbb{E}_{p(\mathcal{\hat{R}}_{i},\mathcal{\hat{R}}_{j})}[log\
q_{\delta}(\mathcal{\hat{R}}_{j}|\mathcal{\hat{R}}_{i})]-\\\
\mathbb{E}_{p(\mathcal{\hat{R}}_{i})}\mathbb{E}_{p(\mathcal{\hat{R}}_{j})}[log\
q_{\delta}(\mathcal{\hat{R}}_{j}|\mathcal{\hat{R}}_{i})]\end{split}$ (8)

A variational distribution $q_{\delta}(y|x)$ with parameter $\delta$ is used
to approximate $p(y|x)$. In practice, we separately use a feed-forward neural
network as an estimator to approximate the conditional distribution between
each pair in VAD variables: $P(\mathcal{\hat{R}}_{i}|\mathcal{\hat{R}}_{j})$
where $i\neq j$, and the parameters are updated along with VAD-VAE at each
time step. Unbiased vCLUB estimates between each pair are summed to get the MI
minimisation loss as the training objective:

$\begin{split}\mathcal{L}_{MI}(\phi,\delta)=\frac{1}{N}\sum_{k=1}^{N}\sum_{i,j}\
\Big{[}log\
q_{\delta_{ij}}(z_{k}^{(\mathcal{\hat{R}}_{i})}|z_{k}^{(\mathcal{\hat{R}}_{j})})-\\\
\frac{1}{N}\sum_{l=1}^{N}log\
q_{\delta_{ij}}(z_{l}^{(\mathcal{\hat{R}}_{i})}|z_{k}^{(\mathcal{\hat{R}}_{j})})\Big{]}\end{split}$
(9)

where $\delta_{ij}$ denotes the parameters of the corresponding estimator. The
detailed proof of Eqn. 8 and 9 is available in [22]. Since no extra
supervision signals are introduced, we expect the model to still achieve
comparable performance as a trade-off for more _independence_ of each latent
space.

###  3.4 Model Training

For ERC task, the concatenated latent representation $z_{i}$ is utilised to
compute the final classification probability:

$\hat{y}_{i}=Softmax(z_{i}W_{0}+b_{0})$ (10)

where $W_{0}$ and $b_{0}$ are learnable parameters. Then we compute the ERC
loss $\mathcal{L}_{ERC}$ using standard cross-entropy loss:

$\mathcal{L}_{ERC}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{|E|}y_{i}^{j}log\
\hat{y}_{i}^{j}$ (11)

where $y_{i}^{j}$ and $\hat{y}_{i}^{j}$ are $j$-th element of $y_{i}$ and
$\hat{y}_{i}$. Finally, we combine all proposed modules and train in a multi-
task learning manner:

$\mathcal{L}=\mathcal{L}_{ERC}+\mu_{E}\mathcal{L}_{ELBO}+\mu_{I}\mathcal{L}_{INFO}+\mu_{MI}\mathcal{L}_{MI}$
(12)

where the $\mu$s are the pre-defined weight coefficients.

##  4 Experimental Settings

###  4.1 Datasets

We evaluate our model on the following three benchmark datasets, where the
statistics are listed in Table I:

TABLE I: Statistics of the datasets. Conv. and Utter. denotes the conversation and utterance number. Utter./Conv denotes the average utterance number per dialogue. Dataset | Conv.(Train/Val/Test) | Utter.(Train/Val/Test) | Utter./Conv
---|---|---|---
IEMOCAP | 100/20/31 | 4,778/980/1,622 | 49.2
MELD | 1,038/114/280 | 9,989/1,109/2,610 | 9.6
DailyDialog | 11,118/1,000/1,000 | 87,170/8,069/7,740 | 7.9

IEMOCAP [50]: A two-party multi-modal conversation dataset derived from the
scenarios in the scripts of the two actors. For all datatsets, we only utilise
the text modality in our experiments. The pre-defined categorical emotions are
_neutral, sad, anger, happy, frustrated, excited_.

MELD [51]: A multi-party multi-modal dataset enriched from _EmotionLines_
dataset, collected from the scripts of American TV show _Friends_. The pre-
defined emotions are _neutral, sad, anger, disgust, fear, happy, surprise_.

DailyDialog [52]: A dataset compiled from human-written daily conversations
with only two parties involved and no speaker information. The pre-defined
emotion labels are the Ekman’s emotion types: _neutral, happy, surprise, sad,
anger, disgust, fear_.

TABLE II: The NRC-VAD assignments to all emotions in the four datasets.

IEMOCAP | neutral | frustrated | sad | anger | excited | happy | –
---|---|---|---|---|---|---|---
Valence | 0.469 | 0.060 | 0.052 | 0.167 | 0.908 | 0.960 | –
Arousal | 0.184 | 0.730 | 0.288 | 0.865 | 0.931 | 0.732 | –
Dominance | 0.357 | 0.280 | 0.164 | 0.657 | 0.709 | 0.850 | –
MELD | neutral | joy | surprise | anger | sad | disgust | fear
Valence | 0.469 | 0.980 | 0.875 | 0.167 | 0.052 | 0.052 | 0.073
Arousal | 0.184 | 0.824 | 0.875 | 0.865 | 0.288 | 0.775 | 0.840
Dominance | 0.357 | 0.794 | 0.562 | 0.657 | 0.164 | 0.317 | 0.293
DailyDialog | neutral | anger | disgust | fear | happy | sad | surprise
Valence | 0.469 | 0.167 | 0.052 | 0.073 | 0.960 | 0.052 | 0.875
Arousal | 0.184 | 0.865 | 0.775 | 0.840 | 0.732 | 0.288 | 0.875
Dominance | 0.357 | 0.657 | 0.317 | 0.293 | 0.850 | 0.164 | 0.562

We utilise information from NRC-VAD [21] in our methodology, a sentiment
lexicon that contains human ratings of Valence, Arousal, and Dominance for
more than 20,000 English words. All the terms in NRC-VAD denote or connote
emotions. Specifically, the terms are collected from sentiment lexicons such
as NRC emotion lexicon [53], General Inquirer [54], and ANEW [46]. These terms
are first very strictly annotated via best-worst scaling with crowdsourcing
annotators. Then an aggregation process calculates the VAD for each term
ranging from 0 to 1. Take Valence as an example. The annotators are presented
with four words at a time (4-tuples) and asked to select the word with the
highest and lowest Valence. The questionnaire uses paradigm words that signify
the two ends of the valence dimension. The final VAD scores are calculated
from the responses: For each item, the score is the proportion of times the
item was chosen as the best (highest V/A/D) minus the proportion of times the
item was selected as the worst (lowest V/A/D). The scores were linearly
transformed from interval 0 (lowest V/A/D) to 1 (the highest V/A/D). In NRC-
VAD, the emotion prototypes of the labels for the three ERC datasets are
listed in Table II. According to the assignments, most cluster centres reflect
appropriate positions of the corresponding emotions in VAD space, where
similar emotions are measurably closer to each other while maintaining a fine-
grained difference to facilitate the model to distinguish them. For example,
_happy_ stays closer to _excited_ than _anger_ in IEMOCAP. In addition, for
all four datasets, positive and negative emotions are separated mainly by
_neutral_ in the dimension Valence, while the emotions within each sentiment
polarity mainly differ in Arousal and Dominance.

We also notice that human-labeled utterance-level VAD scores are available in
IEMOCAP. Therefore, we further explore utilising these utterance-level VAD
scores instead of NRC-VAD to supervise informativeness. The aggregation
process calculates the VAD for each utterance ranging from 1 to 5. In our
experiments, we linearly transform all VAD scores to the range $[0,1]$ during
inference.

###  4.2 Baseline Models

We select the following baseline models for comparison:

TL-ERC [7]: The method pre-trains an encoder-decoder architecture on large-
scale conversation data, then the weights of the encoder are transferred to
ERC.

BERT-Large [55]: Following the standard pretraining-finetuning paradigm, the
model is initialised from the pre-trained weights of BERT-Large, then a linear
classifier follows the output of the BERT encoder for ERC task.

DialogXL [10]: Based on the XLNet [56], this work combines four types of
dialogue-aware self-attention (global self-attention, local self-attention,
speaker self-attention, listener self-attention) to model inter- and intra-
speaker dependencies and proposes an utterance recurrence mechanism to model
the long-range contexts.

DAG-ERC [27]: Utilising RoBERTa-Large [45] as the context-independent
utterance encoder, this model builds a directed acyclic graph on the dialogue
and uses a multi-layer graph neural network to aggregate the information on
the graph. The node representations are utilised for ERC classification.

SKAIG [28]: This work extracts psychological commonsense knowledge and builds
a graph on the dialogue according to different aspects of the knowledge. The
corresponding knowledge representations are used to enhance the edge
representations.

COSMIC [29]: This work uses the RNN to model the dialogue history for each
participant and the context information. It also extracts utterance-level
commonsense knowledge to model the speakers’ mental states and attentively
infuses the knowledge into the utterance representations.

Dis-VAE [32]: This work utilises a VAE to model discourse information between
utterances in an unsupervised manner and combines the learnt latent
representations to the ERC encoder.

SGED [57]: This method proposes a speaker-guided encoder-decoder framework for
ERC to model the intra- and inter-speaker dependencies in a dynamic manner.

CoG-BART [12]: Based on BART-Large, this work utilises contrastive learning
and a response generation task to enhance utterance representations.

SPCL [36]: This work proposes a supervised prototypical contrastive learning
to improve performance on class-imbalanced data. Curriculum learning is also
utilised to further enhance the model.

CoMPM [58]: This work proposes a context embedding module and a speaker-aware
memory module to efficiently model the conversation and utterance-speaker
relations.

###  4.3 Implementation Details

We conduct all experiments using a single Nvidia Tesla A100 GPU with 80GB of
memory. We initialise the pre-trained weights of all PLMs and use the
tokenisation tools both provided by Huggingface [47]. We leverage AdamW
optimiser [59] to train the model. We use the weighted-F1 measure as the
evaluation metric for MELD and IEMOCAP. Since “neutral” occupies most of
DailyDialog, we use micro-F1 for this dataset and ignore the label “neutral”
when calculating the results as in the previous works [27, 12].

The BART decoder brings about 1/3 more parameters to VAD-VAE than most of the
selected baselines. Therefore, we also provide the results with a single-layer
uni-directional LSTM utterance decoder to reduce the parameters and provide a
more fair comparison. All hyper-parameters are tuned on the validation set. We
also provide more implementation details: The batch size of experiments on all
datasets is set to 4 except in DailyDialog, which is 16. Each iteration of
training takes less than 0.75 GPU hours. We use a linear warm-up learning rate
scheduling [60] of warm-up ratio 20$\%$ and a peak learning rate 1e-5. We set
a dropout rate of 0.1 and an L2-regularisation rate of 0.01 to avoid over-
fitting. We do hyper-parameter search and determine
$D_{(V)}=D_{(A)}=D_{(D)}=64$, $D_{(C)}=832$, $D_{h}$ = 1024, $\mu_{E}=0.8$,
$\mu_{I}=1.0$. $\mu_{MI}$ is tuned between $[0.001,0.01]$ on each validation
set. More hyper-parameters will be listed within the source code. The names
and versions of relevant software libraries and frameworks will be described
in the source code. All reported results are averages of five random runs.

TABLE III: Test results on IEMOCAP, MELD and DailyDialog. “-vCLUB” denotes VAD-VAE trained without the vCLUB loss $\mathcal{L}_{MI}$. VAD-VAELSTM denotes the results with an LSTM utterance decoder. VAD-VAEH denotes the results using the human-labeled utterance-level VAD scores. The numbers with $*$ indicate that the improvement over all baselines is statistically significant with p $<$ 0.05 under t-test. Best values are highlighted in bold. Model | IEMOCAP | MELD | DailyDialog
---|---|---|---
TL-ERC | 59.30 | 57.46 | 52.46
BERT-Large | 60.98 | 61.50 | 54.09
DialogXL | 65.94 | 62.41 | 54.93
COSMIC | 65.28 | 65.21 | 58.48
Dis-VAE | 68.23 | 65.34 | 60.95
SGED | 68.53 | 65.46 | –
CoMPM | 66.33 | 66.52 | 60.34
DAG-ERC | 68.03 | 63.65 | 59.33
SKAIG | 66.96 | 65.18 | 59.75
CoG-BART | 66.18 | 64.81 | 56.29
SPCL | 69.74 | 67.25 | –
VAD-VAE |  70.22*($\pm$0.85) | 64.96($\pm$0.19) |  62.14*($\pm$0.23)
-vCLUB | 69.19($\pm$0.66) | 65.94($\pm$0.31) | 61.23*($\pm$0.77)
VAD-VAELSTM | 69.92 | 64.6 | 61.96*
-vCLUB | 69.72 | 64.89 | 61.4*
VAD-VAEH | 70.2* | – | –

##  5 Results and Analysis

###  5.1 Overall Performance

We present the performance of VAD-VAE and the baseline models on the three
benchmark datasets in Table III. According to the results, BERT-Large and
DialogXL outperform TL-ERC on all datasets, showing the advantages of PLM-
based methods over RNN-based models that pre-train from scratch. Following
this trend, the rest of the baseline models and our VAD-VAE all utilise
RoBERTa as the utterance encoder (except CoG-BART which uses BART). COSMIC
explicitly introduces mental state information to enrich the contexts, and the
performance improves significantly on simple-context datasets MELD and
DailyDialog. Dis-VAE, SGED, and CoMPM implicitly introduce speaker-related
information and achieve over 68% on IEMOCAP. To enhance the utterance
representations, DAG-ERC and SKAIG build dialogue-level graphs to introduce
priors on context modelling and perform well on all datasets. The competitive
performance of CoG-BART and SPCL also proves the effectiveness of contrastive
learning and response generation.

Overall, VAD-VAE achieves new state-of-the-art performance 70.22% on IEMOCAP,
62.14% on DailyDialog, and very competitive performance 65.94% on MELD. Our
model reaches an impressive 4.04% improvement on IEMOCAP and a 5.85%
improvement on DailyDialog over CoG-BART, showing the advantage of VAE-based
target utterance reconstruction over response generation. Test results also
show that VAD-VAE outperforms Dis-VAE on all three datasets. We notice that
Dis-VAE only builds a VAE to model the discourse information, while VAD-VAE
uses VAE to reconstruct the target utterance directly. The results prove that
target utterance reconstruction is appropriate to leverage the full potential
of VAE. In addition, our method introduces NRC-VAD information and outperforms
several strong knowledge-enhanced methods. For example, VAD-VAE outperforms
COSMIC by over 4% on both IEMOCAP and DailyDialog, which also utilises
RoBERTa-Large as the utterance encoder and introduces mental state knowledge.
These advantages reflect the effectiveness of the VAD supervision signals.

With the LSTM utterance decoder, VAD-VAELSTM still outperforms previous state-
of-the-art models on both IEMOCAP and DialyDialog. These results further prove
the effectiveness of our proposed VAD-VAE structure under approximately the
same number of parameters. In addition, VAD-VAELSTM also achieves slightly
worse performances on all datasets compared to VAD-VAE, which shows that a
more powerful utterance decoder can benefit more on ERC. However, as VAD-
VAELSTM is more efficient in training, we recommend using the LSTM utterance
decoder in most cases.

We can only test VAD-VAEH on IEMOCAP since all other datasets do not provide
human-labeled utterance-level VAD scores. According to the results, VAD-VAEH
only achieves a comparable performance of 70.2% with VAD-VAE, which does not
correspond with our hypothesis that fine-grained utterance-level VADs would
provide more useful information. We notice that NRC-VAD follows strict best-
worst scaling annotation and aggregation processes with a minimum of 6
annotators per word. In comparison, the IEMOCAP VAD annotation process follows
a rough rule with two annotators for each utterance, which can bring
inaccuracy to many labels and weaken the advantage of context-dependent VAD
labels.

TABLE IV: Results of ablation study. Best values are highlighted in bold.

Method | IEMOCAP | MELD | DailyDialog
---|---|---|---
VAD-VAE | 70.22 | 64.96($\downarrow$0.98) | 62.14
-vCLUB | 69.19($\downarrow$1.03) | 65.94 | 61.23($\downarrow$0.91)
-VAE Decoder | 67.92($\downarrow$2.30) | 63.99($\downarrow$1.95) | 60.16($\downarrow$1.98)
-V, A, D Sup. | 66.85($\downarrow$3.37) | 63.99($\downarrow$1.95) | 61.01($\downarrow$1.13)
-V Sup. | 67.60($\downarrow$2.62) | 64.20($\downarrow$1.74) | 61.92($\downarrow$0.22)
-A Sup. | 68.06($\downarrow$2.16) | 64.59($\downarrow$1.35) | 61.38($\downarrow$0.76)
-D Sup. | 67.16($\downarrow$3.06) | 64.03($\downarrow$1.91) | 61.66($\downarrow$0.48)
Utter. Encoder | 66.52($\downarrow$3.70) | 63.7($\downarrow$2.24) | 59.80($\downarrow$2.34)

###  5.2 Ablation Study

To investigate the effect of each module, we provide ablation analysis in
Table IV. “-” denotes removing a module. “vCLUB” denotes the MI minimisation
modules. “VAE Decoder” denotes the VAE decoder module for target utterance
reconstruction. “V Sup.”, “A Sup.”, and “D Sup.” denote the NRC-VAD
supervision signals corresponding to Valence, Arousal, and Dominance. “Utter.
Encoder” directly trains an ERC model on the context-aware utterance encoder.

According to the results, VAD-VAE achieves comparable performance with
“-vCLUB”, which corresponds with our early hypothesis. With vCLUB loss added,
the performance improves on IEMOCAP and DailyDialog. A possible reason is that
MI minimisation enforces the model to learn dissimilar representations for
VAD, which corresponds with the orthogonal nature of VAD in psychology.
Without the VAE Decoder, the performance drops significantly on all datasets,
which further indicates the effectiveness of the target utterance
reconstruction task.

“-V, A, D Sup.” leads to significant drops in all datasets, which proves our
hypothesis that NRC-VAD supervision signals also provide fine-grained
information to enhance ERC performance. In further comparisons of separately
removing supervision signals for Valence, Arousal, and Dominance, the
performance drops most with “D Sup.” removed for IEMOCAP and MELD and “A Sup.”
removed for DailyDialog. We notice that the sentiment polarity of emotions is
primarily determined by Valence, and similar emotions mainly differ in Arousal
and Dominance. Therefore, these results show that our model benefits more from
fine-grained information from Arousal and Dominance to distinguish similar
emotions.

Figure 3: The performance of VAD-VAE on IEMOCAP with different context window
sizes in the input sequence. We separately present results with both past and
future contexts and only past/future contexts. ”all” denotes setting the
corresponding window sizes as wide as possible.

###  5.3 Context Influence

In previous experiments, we set both past and future context window sizes
$W_{p}$ and $W_{f}$ as wide as possible (the maximum input length of RoBERTa
encoder is 512) to provide more contextual information. In Figure 3, we
further present VAD-VAE performances with different context window sizes to
investigate the influence of multi-range contextual information and guide
future research directions on context modeling.

According to the results, both past and future contexts significantly enhance
the model performance compared to no context (window size 0), but
$past\&future$ further outperforms both _only past_ and _only future_ on all
window sizes, which shows that VAD-VAE benefits from combined emotional
reasoning with past and future contexts. In addition, though the nearest 5
contexts provide most of the improvement (over 10%) for $past\&future$ and
_only past_ , their performances continuously increase with wider window
sizes, proving that long-range conversational history also provides useful
information. On the other hand, the performance of _only future_ reaches the
top with a window size of 10 but decreases with longer contexts. These results
denote near-future contexts as more indicative of emotional cues for the
target utterance, while unrelated longer future contexts can bring noises to
the emotional reasoning.

TABLE V: The Pearson’s correlation coefficients between predicted VAD scores and supervision signals on all test sets, and the average vCLUB estimates of MI between VAD latent distributions. Best values are highlighted in bold. | IEMOCAP | MELD | DailyDialog
---|---|---|---
Methods | V | A | D | MI | V | A | D | MI | V | A | D | MI
$\mathcal{L}_{ELBO}$ | 0.434 | 0.156 | 0.094 | 0.732 | 0.021 | 0.133 | 0.152 | 0.398 | 0.071 | 0.029 | -0.017 | 0.356
+$\mathcal{L}_{MI}$ | 0.306 | 0.132 | 0.150 | 0.133 | 0.044 | 0.230 | 0.141 | 0.068 | 0.115 | -0.026 | -0.016 | 0.094
+$\mathcal{L}_{INFO}$ | 0.882 | 0.708 | 0.743 | 0.531 | 0.565 | 0.643 | 0.557 | 0.885 | 0.424 | 0.348 | 0.321 | 0.789
+$\mathcal{L}_{INFO}$+$\mathcal{L}_{MI}$ | 0.872 | 0.715 | 0.765 | 0.355 | 0.568 | 0.639 | 0.558 | 0.312 | 0.433 | 0.355 | 0.330 | 0.226
Figure 4: The performance of VAD-VAE on ERC and the MI estimates between VAD
pairs with different vCLUB weight coefficients $\mu_{MI}$ on the three test
sets.

###  5.4 Disentanglement Evaluation

We analyse the effects of VAD supervision signals ($\mathcal{L}_{INFO}$) and
MI minimisation ($\mathcal{L}_{MI}$) on enhancing VAD disentanglement. In
Table V, we present the Pearson’s correlation coefficients between the
predicted VAD scores from latent representations and the supervision signals
from NRC-VAD on all three test sets. Higher values indicate more precise
predictions, denoting better _Informativeness_. We also provide the average
vCLUB estimates of MI between VAD latent distributions on each test set, with
lower values denoting lower estimates of MI upper bounds and better
_Independence_.

####  5.4.1 Informativeness

According to the results, the model performs poorly on all datasets (Pearson’s
correlation coefficients below 0.2 in most cases) with VAE reconstruction loss
($\mathcal{L}_{ELBO}$) or $\mathcal{L}_{MI}$ introduced, since VAD features
can be embedded in Content space without specific supervisions. We observe
significant improvement in _Informativeness_ for VAD with
$\mathcal{L}_{INFO}$, which brings over 0.5 Pearson’s correlation coefficients
gain for IEMOCAP and 0.3 for MELD and DailyDialog. These results reflect the
effectiveness of NRC-VAD supervision signals. On top of $\mathcal{L}_{INFO}$,
$\mathcal{L}_{MI}$ further improves the Pearson’s correlation coefficients
scores in most cases, which shows that MI minimisation also helps to enhance
_Informativeness_ of VAD representations to some extent.

####  5.4.2 Independence

For all datasets, The vCLUB estimates remain high with only
$\mathcal{L}_{ELBO}$ introduced since the unified distributed representation
in VAE encourages strong correlations between each part. With
$\mathcal{L}_{INFO}$, we observe even higher vCLUB in MELD and DailyDialog. In
this case, our model is only optimised for _Informativeness_ , and enforces
full utilisation of all latent spaces, which leads to high MI. Introducing
only $\mathcal{L}_{MI}$ has the lowest vCLUB in all datasets. However, it
achieves bad performance in _Informativeness_. With both $\mathcal{L}_{MI}$
and $\mathcal{L}_{INFO}$, VAD-VAE not only achieves the best results in VAD
predictions but also greatly decreases vCLUB compared with only
$\mathcal{L}_{INFO}$, showing the satisfactory trade-off between
_Informativeness_ and _Independence_.

####  5.4.3 ERC-Independence Trade-off

To further investigate the influence of MI minimisation on ERC performance, we
present the results of tuning vCLUB loss coefficients $\mu_{MI}$ on the three
test sets in Figure 4.

For all datasets, the MI estimates decrease as $\mu_{MI}$ increases, showing
the effectiveness of the vCLUB method. Meanwhile, ERC performance decreases at
acceptable rates. For IEMOCAP, with $\mu_{MI}$=0.005, all MI upper bounds
steadily drop below 0.3 while keeping a competitive ERC performance: over 67%.
As $\mu_{MI}$ further increases, there is no apparent decrease in MI
estimates. We have similar observations for MELD at $\mu_{MI}$=0.006. On the
other hand, for DailyDialog, all MI estimates drop rapidly below 0.1 with
small $\mu_{MI}$=0.001. According to the statistics in Table I, DailyDialog
has more abundant training data, which provides rich information for
understanding Valence, Arousal, and Dominance. Simpler contexts of DailyDialog
can also facilitate VAD disentanglement. From the perspective of each VAD
pair, DailyDialog achieves high disentanglement for all VAD pairs, while on
the other two datasets, the performances are uneven. Valence-Dominance in
IEMOCAP and Valence-Arousal in MELD remain above 0.2. These results show that
the pair-wise difficulties of VAD disentanglement are possibly related to the
complexity of the contexts.

Figure 5: UMAP visualisations of Valence, Arousal, Dominance and Content
representations for IEMOCAP test results. TABLE VI: Four cases where target
utterance reconstruction leverages the critical context of the target
utterance for emotion reasoning, while the next utterance provides limited
information. “CoG-BART”, “VAD-VAE” and “Human” denotes the corresponding
predictions for response generation (CoG-BART), target utterance
reconstruction (VAD-VAE) and the human-annotated emotion labels. Key
information in the context: bold.

Target Utterance | Next Utterance | Context | CoG-BART | VAD-VAE | Human
---|---|---|---|---|---
| #1 Chandler: It’s all finished!
---
| Joey: This was Carol’s favorite beer. She
---
always drank it out of can, I should have
known.
| Chandler: That’s OK, anyway.
---
Joey: Tell Monica we are done with
the bookcase at last!
_neutral_ | _joyful_ | _joyful_
| #2 James: OK, that’s OK. Improvise
---
is just good.
| James: Well, what about we go to a great
---
jazz show now, and um-
| James: Well, you’ve got to tell me
---
the details. What did he say?
Joey: I don’t-I don’t know how much
plan he had done.
_neutral_ | _excited_ | _excited_
| #3 Mary: It’s a good idea to live together
---
before you get married.
| James: I think so, too. Are you going
---
to get a house or-
| Linda: I have to think about moving now.
---
We are going to move in together!
Monica: I thought you two already
did that…
_surprised_ | _happy_ | _happy_
| #4 Chandler: Hey thanks. Joey is the uh-
---
fellow processor.
| Scott: No kidding.
---
| Joey: But don’t you need more experiences
---
for a job like that?
Chandler: It’s not that hard to learn. Hey,
you’re an actor, act like a processor.
James: Hey Chandler, this morning’s
projections here.
_frustrated_ | _angry_ | _angry_

###  5.5 Disentangled Representations Visualisation

To conduct a more interpretable analysis of the disentangled representations,
we present the UMAP [61] visualisations of VAD representations in the IEMOCAP
test set for four representative emotions in Figure 5. Their corresponding
NRC-VAD assignments are presented in Table II. As shown, for Valence and
Dominance, positive and negative emotions are well separated, while emotions
within one polarity overlap. In the visualisation of Arousal, “happiness”,
“excited” and “frustration” lie close while “sadness” separates away. These
observations correspond with the NRC-VAD assignments, indicating the quality
of the learnt VAD representations. In addition, the distribution of each
emotion shows continuity and completeness conditions.

We observe very regularised latent distributions for Content. The distribution
of each emotion is close to a Gaussian distribution and shows continuity and
completeness conditions for “frustration”, “sadness” and “happiness”
utterances. However, the representations of “excited” and “happiness” still
overlap, possibly due to the similar way of expressing excited and happy
emotions. Overall, with the well learnt disentangled VAD representations and
sound properties of the content space, one direction of future work is to
explore the potential of VAD-VAE in the controlled affective text generation
task. Unlike previous works, which control categorical emotions, our work aims
to control more fine-grained sentiments by adjusting Valence, Arousal, and
dominance separately. To provide a more intuitive view, we provide a case of
VAD-based controlled generation in Figure 6. The VAD representations of the
neutral target utterance are replaced by the VAD representations of a randomly
sampled excited utterance. As shown, the utterance decoder can adjust the
tones of the utterance according to the input VAD representations. For
example, with the new 0.908 Valence representation, the reconstructed
utterance ”so glad I got these at a reasonable price!” shows more
pleasantness.

Figure 6: A case where the target utterance is reconstructed via VAD-VAE by
separately adjusting Valence, Arousal, and Dominance. Figure 7: Performance
decrease rate of VAD-VAE and vanilla VAE on three ERC test sets with different
random replacement percentages of training labels.

###  5.6 Robustness Evaluation

In Figure 7, we evaluate the robustness of disentangled representations by
randomly replacing a percentage (0% to 50%) of training labels with false
labels, then comparing the performance decrease rate (For $\alpha\%$
replacement, the rate is computed as $\frac{F1\ for\ \alpha\%\
replacement}{F1\ for\ 0\%\ replacement}$) of VAD-VAE and the vanilla VAE
method. Higher rates indicate better performance and more robustness.

According to the results, disentangled VAD representations achieve higher
performance decrease rates than entangled representations on all datasets. For
example, VAD-VAE outperforms VAE by an average of 12.24% with all label
replacement percentages (orange shaded area). With 50% of random label
replacement, VAD-VAE retains over 75% of performances on all test sets while
the performances of VAE all fall below 70%. These results show that VAD-VAE
can tolerate a higher level of misinformation during training, indicating the
robustness of disentangled VAD representations over the entangled
representations. A possible reason is that disentangled representations are
explicitly trained to extract VAD information, which provides helpful guidance
during inference when other features are misleading.

###  5.7 Case Study for Target Utterance Reconstruction

Target utterance reconstruction enables VAD-VAE to learn long-range
dependencies and outperforms the response generation-based model CoG-BART. We
provide four cases from the test results of VAD-VAE and CoG-BART in Table VI
for further investigation.

According to our observation, there are mainly two scenarios where response
generation provides limited information and mispredicts the emotion: (a)
Change of the discussion topic in the response. In the next utterance of case
#1, the discussion topic changes from “Finish assembling the bookcase” to
“Carol’s favorite beer”. In the next utterance of case #2, the topic changes
from “His plan” to “Go to a great jazz show”. In these cases, emotion-related
clues in the response are irrelevant to the target utterance. (b) Indirect
response to the current utterance. In case #3, the next utterance merely
expresses agreement with the target utterance and expands the dialogue. In
case #4, the response is simply sarcasm. These responses provide crucial clues
for future utterances but little information for the target utterance. These
scenarios are more common in multi-party dialogues. In contrast, VAD-VAE can
learn complex inter- and intra-speaker influences since it learns critical
information via the context-aware target utterance reconstruction. For
example, VAD-VAE focuses on the previous discussion topic “done with the
bookcase”(case #1) and “his plan”(case #2). It can also extract key
information from long-range contexts, such as in cases #3 and #4. These
advantages allow VAD-VAE to make correct emotion predictions in these cases.

##  6 Conclusion

This paper proposes a VAD-disentangled Variational Autoencoder for emotion
recognition in conversations. We first introduce an auxiliary target utterance
reconstruction task via the VAE framework. Then we disentangle three key
features: Valence, Arousal, and Dominance, from the latent space. VAD
supervision signals and a mutual information minimisation task are also
utilised to enhance the disentangled VAD representations.

Experiments show that VAD-VAE outperforms the state-of-the-art model on two
ERC datasets. Ablation and case studies prove the effectiveness of the
proposed VAE-based target utterance reconstruction task and VAD supervision
signals. Further quantitative analysis and visualisation also show that VAD-
VAE learns disentangled VAD representations with informativeness, independence
and robustness. In the future, we will leverage these decent VAD
representations to explore fine-grained emotion control for affective text
generation by adjusting Valence, Arousal, and Dominance separately.

## Acknowledgments

This work was supported in part by the Alan Turing Institute, UK. This work
was also supported by the University of Manchester President’s Doctoral
Scholar award.

## References

  * [1] S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” _IEEE Access_ , vol. 7, pp. 100 943–100 953, 2019.
  * [2] Y. Ma, K. L. Nguyen, F. Z. Xing, and E. Cambria, “A survey on empathetic dialogue systems,” _Information Fusion_ , vol. 64, pp. 50–70, 2020.
  * [3] P. Nandwani and R. Verma, “A review on sentiment analysis and emotion detection from text,” _Social Network Analysis and Mining_ , vol. 11, no. 1, pp. 1–19, 2021.
  * [4] A. Chatterjee, U. Gupta, M. K. Chinnakotla, R. Srikanth, M. Galley, and P. Agrawal, “Understanding emotions in text using deep learning and big data,” _Computers in Human Behavior_ , vol. 93, pp. 309–317, 2019.
  * [5] S. Zad, M. Heidari, J. H. J. Jones, and O. Uzuner, “Emotion detection of textual data: An interdisciplinary survey,” in _AIIoT_ , 2021, pp. 0255–0261.
  * [6] J. Wang, J. Wang, C. Sun, S. Li, X. Liu, L. Si, M. Zhang, and G. Zhou, “Sentiment classification in customer service dialogue with topic-aware multi-task learning,” in _AAAI_. AAAI Press, 2020, pp. 9177–9184.
  * [7] D. Hazarika, S. Poria, R. Zimmermann, and R. Mihalcea, “Conversational transfer learning for emotion recognition,” _Information Fusion_ , vol. 65, pp. 1–12, 2021.
  * [8] E. Chapuis, P. Colombo, M. Manica, M. Labeau, and C. Clavel, “Hierarchical pre-training for sequence labelling in spoken dialog,” in _Findings of EMNLP_. Online: Association for Computational Linguistics, Nov. 2020, pp. 2636–2648.
  * [9] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” _Science China Technological Sciences_ , vol. 63, no. 10, pp. 1872–1897, 2020.
  * [10] W. Shen, J. Chen, X. Quan, and Z. Xie, “Dialogxl: All-in-one xlnet for multi-party conversation emotion recognition,” in _AAAI_. AAAI Press, 2021, pp. 13 789–13 797.
  * [11] Y. Xie, K. Yang, C. Sun, B. Liu, and Z. Ji, “Knowledge-interactive network with sentiment polarity intensity-aware multi-task learning for emotion recognition in conversations,” in _Findings of EMNLP_. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 2879–2889.
  * [12] S. Li, H. Yan, and X. Qiu, “Contrast and generation make BART a good dialogue emotion recognizer,” in _AAAI_. AAAI Press, 2022, pp. 11 002–11 010.
  * [13] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _ACL_. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880.
  * [14] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 8, pp. 1798–1828, 2013.
  * [15] T. Goswamy, I. Singh, A. Barkati, and A. Modi, “Adapting a language model for controlled affective text generation,” in _COLING_. Barcelona, Spain: International Committee on Computational Linguistics, Dec. 2020, pp. 2787–2801.
  * [16] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in _ICLR_ , 2014.
  * [17] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner, “Towards a definition of disentangled representations,” _arXiv preprint arXiv:1812.02230_ , 2018.
  * [18] J. A. Russell and A. Mehrabian, “Evidence for a three-factor theory of emotions,” _Journal of Research in Personality_ , vol. 11, no. 3, pp. 273–294, 1977.
  * [19] A. Mehrabian, “Framework for a comprehensive description and measurement of emotional states.” _Genetic, social, and general psychology monographs_ , 1995.
  * [20] C. Eastwood and C. K. I. Williams, “A framework for the quantitative evaluation of disentangled representations,” in _ICLR_. OpenReview.net, 2018.
  * [21] S. Mohammad, “Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words,” in _ACL_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 174–184.
  * [22] P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin, “CLUB: A contrastive log-ratio upper bound of mutual information,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 1779–1788.
  * [23] D. Hazarika, S. Poria, R. Mihalcea, E. Cambria, and R. Zimmermann, “ICON: Interactive conversational memory network for multimodal emotion detection,” in _EMNLP_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 2594–2604.
  * [24] D. Hazarika, S. Poria, A. Zadeh, E. Cambria, L.-P. Morency, and R. Zimmermann, “Conversational memory network for emotion recognition in dyadic dialogue videos,” in _NAACL_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2122–2132.
  * [25] N. Majumder, S. Poria, D. Hazarika, R. Mihalcea, A. F. Gelbukh, and E. Cambria, “Dialoguernn: An attentive RNN for emotion detection in conversations,” in _AAAI_. AAAI Press, 2019, pp. 6818–6825.
  * [26] J. Li, D. Ji, F. Li, M. Zhang, and Y. Liu, “HiTrans: A transformer-based context- and speaker-sensitive model for emotion detection in conversations,” in _COLING_. Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 4190–4200.
  * [27] W. Shen, S. Wu, Y. Yang, and X. Quan, “Directed acyclic graph network for conversational emotion recognition,” in _ACL_. Online: Association for Computational Linguistics, Aug. 2021, pp. 1551–1560.
  * [28] J. Li, Z. Lin, P. Fu, and W. Wang, “Past, present, and future: Conversational emotion recognition through structural modeling of psychological knowledge,” in _Findings of EMNLP_. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 1204–1214.
  * [29] D. Ghosal, N. Majumder, A. Gelbukh, R. Mihalcea, and S. Poria, “COSMIC: COmmonSense knowledge for eMotion identification in conversations,” in _Findings of EMNLP_. Online: Association for Computational Linguistics, Nov. 2020, pp. 2470–2481.
  * [30] J. Wang, J. Wang, C. Sun, S. Li, X. Liu, L. Si, M. Zhang, and G. Zhou, “Sentiment classification in customer service dialogue with topic-aware multi-task learning,” in _AAAI_. AAAI Press, 2020, pp. 9177–9184.
  * [31] L. Zhu, G. Pergola, L. Gui, D. Zhou, and Y. He, “Topic-driven and knowledge-aware transformer for dialogue emotion detection,” in _ACL_. Online: Association for Computational Linguistics, Aug. 2021, pp. 1571–1582.
  * [32] D. Ong, J. Su, B. Chen, A. T. Luu, A. Narendranath, Y. Li, S. Sun, Y. Lin, and H. Wang, “Is discourse role important for emotion recognition in conversation?” in _AAAI_. AAAI Press, 2022, pp. 11 121–11 129.
  * [33] S. Park, J. Kim, S. Ye, J. Jeon, H. Y. Park, and A. Oh, “Dimensional emotion detection from categorical emotion,” in _EMNLP_. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 4367–4380.
  * [34] R. Mukherjee, A. Naik, S. Poddar, S. Dasgupta, and N. Ganguly, “Understanding the role of affect dimensions in detecting emotions from tweets: A multi-task approach,” in _SIGIR_. ACM, 2021, pp. 2303–2307.
  * [35] K. Yang, T. Zhang, H. Alhuzali, and S. Ananiadou, “Cluster-level contrastive learning for emotion recognition in conversations,” _IEEE Transactions on Affective Computing_ , pp. 1–12, 2023.
  * [36] X. Song, L. Huang, H. Xue, and S. Hu, “Supervised prototypical contrastive learning for emotion recognition in conversation,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 5197–5206. [Online]. Available: https://aclanthology.org/2022.emnlp-main.347
  * [37] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, “Understanding disentangling in beta-vae,” in _NeurlPS Workshops_. Curran Associates, Inc., 2018\.
  * [38] R. T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” in _NeurlPS_ , vol. 31. Curran Associates, Inc., 2018\.
  * [39] Y. Bao, H. Zhou, S. Huang, L. Li, L. Mou, O. Vechtomova, X.-y. Dai, and J. Chen, “Generating sentences from disentangled syntactic and semantic spaces,” in _ACL_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6008–6019.
  * [40] P. Cheng, M. R. Min, D. Shen, C. Malon, Y. Zhang, Y. Li, and L. Carin, “Improving disentangled text representation learning with information-theoretic guidance,” in _ACL_. Online: Association for Computational Linguistics, Jul. 2020, pp. 7530–7541.
  * [41] J. Chen and K. Batmanghelich, “Weakly supervised disentanglement by pairwise similarities,” in _AAAI_. AAAI Press, 2020, pp. 3495–3502.
  * [42] J. Vasilakes, C. Zerva, M. Miwa, and S. Ananiadou, “Learning disentangled representations of negation and uncertainty,” in _ACL_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 8380–8397.
  * [43] V. John, L. Mou, H. Bahuleyan, and O. Vechtomova, “Disentangled representation learning for non-parallel text style transfer,” in _ACL_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 424–434.
  * [44] R. Shu, Y. Chen, A. Kumar, S. Ermon, and B. Poole, “Weakly supervised disentanglement with guarantees,” in _ICLR_. OpenReview.net, 2020.
  * [45] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [46] A. B. Warriner, V. Kuperman, and M. Brysbaert, “Norms of valence, arousal, and dominance for 13,915 english lemmas,” _Behavior Research Methods_ , vol. 45, pp. 1191–1207, 2013.
  * [47] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [48] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” in _ICLR_. OpenReview.net, 2017.
  * [49] B. Poole, S. Ozair, A. van den Oord, A. A. Alemi, and G. Tucker, “On variational bounds of mutual information,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 5171–5180.
  * [50] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, no. 4, pp. 335–359, 2008.
  * [51] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A multimodal multi-party dataset for emotion recognition in conversations,” in _ACL_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 527–536.
  * [52] Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, “DailyDialog: A manually labelled multi-turn dialogue dataset,” in _IJCNLP_. Taipei, Taiwan: Asian Federation of Natural Language Processing, Nov. 2017, pp. 986–995.
  * [53] S. M. Mohammad and P. D. Turney, “Crowdsourcing a word-emotion association lexicon,” _Comput. Intell._ , vol. 29, no. 3, pp. 436–465, 2013.
  * [54] P. J. Stone and E. B. Hunt, “A computer approach to content analysis: studies using the general inquirer system,” in _Proceedings of the 1963 spring joint computer conference, AFIPS 1963 (Spring), Detroit, Michigan, USA, May 21-23, 1963_ , E. C. Johnson, Ed. ACM, 1963, pp. 241–256.
  * [55] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
  * [56] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [57] Y. Bao, Q. Ma, L. Wei, W. Zhou, and S. Hu, “Speaker-guided encoder-decoder framework for emotion recognition in conversation,” in _IJCAI_. ijcai.org, 2022, pp. 4051–4057.
  * [58] J. Lee and W. Lee, “CoMPM: Context modeling with speaker’s pre-trained memory tracking for emotion recognition in conversation,” in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 5669–5679. [Online]. Available: https://aclanthology.org/2022.naacl-main.416
  * [59] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _ICLR_. OpenReview.net, 2019.
  * [60] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” _arXiv preprint arXiv:1706.02677_ , 2017.
  * [61] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” _arXiv preprint arXiv:1802.03426_ , 2018.

|  Kailai Yang received the B.E. degree in computer science and technology
from Harbin Institute of Technology, China, in 2021. He is currently pursuing
the Ph.D degree in computer science at the University of Manchester, United
Kingdom. His current research interests include natural language processing,
affective computing and deep learning.
---|---
|  Tianlin Zhang received the B.S. degree in software engineering from Nankai
University, China, in 2017 and the M.S. degree in computer science from
University of Chinese Academy of Sciences, China, in 2020. He is currently
pursuing the Ph.D. degree in computer science at the University of Manchester,
United Kingdom. His current research interests include natural language
processing, affective computing and deep learning.
---|---
|  Sophia Ananiadou is the Director of the UK National Centre for Text Mining
(NaCTeM) and a Professor of Computer Science with The University of
Manchester. She is also a Turing Fellow in the Alan Turing Institute and a
visiting professor in AIST/AIRC. Her main research contributions are in the
areas of biomedical text mining and natural language processing.
---|---