# M2R2: Missing-Modality Robust emotion Recognition framework with iterative
data augmentation

Wang Ning
Xi’an Jiaotong University
@ a
a
@

###### Abstract

This paper deals with the utterance-level modalities missing problem with
uncertain patterns on emotion recognition in conversation (ERC) task. Present
models generally predict the speaker’s emotions by its current utterance and
context, which is degraded by modality missing considerably. Our work proposes
a framework Missing-Modality Robust emotion Recognition (M2R2), which trains
emotion recognition model with iterative data augmentation by learned common
representation. Firstly, a network called Party Attentive Network (PANet) is
designed to classify emotions, which tracks all the speakers’ states and
context. Attention mechanism between speaker with other participants and
dialogue topic is used to decentralize dependence on multi-time and multi-
party utterances instead of the possible incomplete one. Moreover, the Common
Representation Learning (CRL) problem is defined for modality-missing problem.
Data imputation methods improved by the adversarial strategy are used here to
construct extra features to augment data. Extensive experiments and case
studies validate the effectiveness of our methods over baselines for modality-
missing emotion recognition on two different datasets. Impact Statement

##  1 Introduction

Figure 1: Possible causes of data missing at utterances in a conversation: 1)
target face out of image boundary; 3) audio corrupted by noise; 4) fail to
transform audio to text; 7) audio sensor fails; 8) camera fails.

Emotion recognition in conversation (ERC) has now attracted increasing
attention as its potential ability to mine and analyze the connections from
the heavily incremental volume of publicly available conversational
data[3][9]. Furthermore, ERC can be used as a tool for psychological analysis
in healthcare systems[28] and human-interacted emotion-aware conversation that
requires an understanding of the user’s emotions [31][6].

Traditional machine learning methods according to textual features such as
Naïve Bayes classifier[35], NLTK vader[5], etc, have been proposed for ERC
task [34]. As deep learning develops, many neural networks models are employed
to classify the emotion at each utterance. [27] uses autoencoder to detect
multiple emotions in conversation transcripts. Graph convolutional networks
are exploited by [23] and [42] to model the inter-relationship between
conversation topic and its different parties for better emotion
representations. [14] employs LSTM cells and Bert hierarchically for utterance
context considering the sequential essence of the conversation, which was
followed by [29] firstly using recurrent neural network (RNN) to recognize
emotions. Moreover, many NLP techniques and ideas are introduced in ERC. [10]
incorporates different elements of commonsense and builds upon them to learn
interactions between all conversation parties, and [44] proposes context-aware
affective graph attention mechanism to interpret contextual utterances
hierarchically.

While solving the ERC task by unimodal (mostly textual) data is not enough
because not only that the emotions are subjective mixed feelings [6], but also
the labeling difficulty in real-life conversation scenarios. Through multiple
channels, humans express emotions and feelings (such as voice speech, textual
meaning, facial expression, body gestures, movements) provides multimodal
data. Multimodal emotion recognition systems are studied alongside unimodal
solutions as they offer higher accuracy of classification [1]. Many systems
consider multimodal data for ERC. [13], [12] and [21] introduce attention
mechanism in conversation context extraction by pooling all proceeding global
states related to concatenated utterance modalities. To better fuse multi
modalities under different representation spaces, [32], [36], [15] propose
kinds of novel cross-modality modules to capture intra- and inter-modal
influence.

In the real world, however, data often suffer from utterance-level
incompleteness [19] with various patterns, which greatly degrades the
integration and complement between different modalities. For the ERC task,
there are various causes for desired data to be temporarily missing: sensors
can fail so that the corresponding signal is no longer available, or its
outcome may be corrupted by noise. In another situation, faces with obvious
expressions may be blocked or move drastically and then disappear from the
view of the sensor [39].

Few works focus on the modality-missing problem at utterance level. Typical
data imputation algorithm [37][24][40] are aimed at separate, non-sequential
data. [26] pad missing modalities by zeros or repeated data with advanced
training strategies. [43] learns robust joint multimodal representations by
bidirectional encoders with cascade-connected residual imagination module. [7]
performs infinite imputations by integrating all modalities, which was
followed by [25] which leverages several classifiers under all combinations of
different modalities. By building features reconstruction module, [20] employs
Bayesian neural network to employ data uncertainty as feature regularization.

This paper deals with the utterance-level modalities missing problem with
uncertain patterns on emotion recognition in conversation (ERC) task. We
propose a novel framework Missing-Modality Robust emotion Recognition (M2R2),
which trains emotion recognition model with iterative data augmentation by
learned common representation.

We first introduce Party Attentive Network (PANet), which tracks all the
speakers’ states and context. Attention mechanism between speaker with other
participants and dialogue topic is used to decentralize dependence on multi-
time and multi-party utterances instead of the possible incomplete one.
Moreover, we define the Common Representation Learning (CRL) problem for
modality-missing data augmentation. Data imputation methods improved by the
adversarial strategy are used here to construct extra features. Extensive
experiments and case studies validate the effectiveness of our methods over
baselines for modality-missing emotion recognition on two different datasets.
The contributions are summarized as follows:

  1. 1.

We propose a novel M2R2 framework with iterative data augmentation on missing-
modality datasets, which alleviate the performance decline caused by
modalities incompleteness greatly.

  2. 2.

To better alleviate the degradation on ERC task caused by modality missing at
utterance level, this paper designs PANet to utilize attention mechanism in
speaker with other participants and dialogue topic.

  3. 3.

A new CRL task is defined to learn extra features for data augmentation.
Adversarial strategy, WGAN, is employed to generate extra features better.

##  2 Problem Statements

Suppose a multimodal conversation scene with $T$ length, let
$\bm{\mathrm{x}}^{\mathrm{a}}=[x^{\mathrm{a}}_{1},\dots,x^{\mathrm{a}}_{T}],\bm{\mathrm{x}}^{\mathrm{t}}=[x^{\mathrm{t}}_{1},\dots,x^{\mathrm{t}}_{T}],\bm{\mathrm{x}}^{\mathrm{v}}=[x^{\mathrm{v}}_{1},\dots,x^{\mathrm{v}}_{T}]$
define audio, text and visual representations respectively. And there are $Q$
parties/speakers $q_{1},q_{2},\dots,q_{Q}$ in this conversation. ERC task is
to predict the emotion label $e_{t}^{q_{I\left(t\right)}}$ at time $t$, where
$I(t)\in{1,2,\dots,Q}$ indicates the mapping from time $t$ to index of current
party. At each time $t$, the constituent utterance
$\bm{\mathrm{u}}_{t}=\left(x^{\mathrm{a}}_{t}\oplus x^{\mathrm{t}}_{t}\oplus
x^{\mathrm{v}}_{t}\right)\in\mathbb{R}^{D_{\mathrm{a}}+D_{\mathrm{t}}+D_{\mathrm{v}}}$
is uttered by $q_{I(t)}$, where $\oplus$ means concatenation here, and this
multimodal representations $\bm{\mathrm{u}}_{t}$ is extracted as described in
section 4.2. Current systems use GRU cells [4] to update hidden state and
result representation. At time step $t$, the GRU cell computes a new hidden
state $h_{t}$ by $\mathrm{GRU}(x_{t},h_{t-1})$ as follows:

$\begin{split}z_{t}&=\sigma(W_{z}x_{t}+U_{z}h_{t-1}),\\\
r_{t}&=\sigma(W_{r}x_{t}+U_{r}h_{t-1}),\\\
\tilde{h}_{t}&=\mathrm{tanh}(Wx_{t}+U(r_{t}\circ h_{t-1})),\\\
h_{t}&=(1-z_{t})\circ h_{t-1}+z_{t}\circ\tilde{h}_{t},\end{split}$ (1)

where $h_{t-1}$ is previous hidden state and $x_{t}$ is current
input.$W_{z},U_{z},W_{r},U_{r},W,U$ server as trainable parameters which make
$\mathrm{GRU}$ efficient.

Given the lack of several modality at different time, a new data piece
$\\{\mathcal{S}_{t},e_{t}\\}_{t=1}^{T}$ can be obtain as the subset of the
full utterance $\mathcal{U}_{t}=\\{\bm{\mathrm{u}}_{t}^{m}\\}_{m=1}^{M}$ (i.e,
$\mathcal{S}\subseteq\mathcal{U}$). Here $M$ means the number of modalities
and equals to $3$ in ERC task. If a modality $m_{0}$ lacks during whole
conversation, the new data piece will be
$\\{\mathcal{U}_{t}-\bm{\mathrm{u}}_{t}^{m0},e_{t}\\}_{t=1}^{T},\forall
t\in\\{1,2,\dots,T\\}$. Consider that a conversation cannot continue if all
modalities lacks at the same time, there must be available one for all $t$.
Accordingly, we define missing rate as
$\eta={\sum_{t}\sum_{m}{S_{t}}}/{\sum_{t}\sum_{m}{\bm{\mathrm{u}}_{t}^{m}}}$.
Fig.1 describes the possibilities.

For the case of data incompleteness, we define Common Representation Learning
task as follows:

######  Definition 1.

Common Representation Learning (CRL) Given the training set
$\\{\mathcal{S}_{t},y_{t}\\}_{t=1}^{T}$ that can be obtain as the subset of
the full utterance $\mathcal{U}_{t}=\\{\bm{\mathrm{u}}_{t}^{m}\\}_{m=1}^{M}$
(i.e, $\mathcal{S}\subseteq\mathcal{U}$), CRL aims to learn common
representation $\\{\bm{\mathrm{h}}_{t}\\}_{t=1}^{T}$ of
$\\{\mathcal{S}_{t}\\}_{t=1}^{T}$ under arbitrary missing pattern.

The best common representation can be obtained by maximizing
$p(y,\mathcal{S}|\bm{\mathrm{h}})$ as

$\bm{\mathrm{h}}^{*}=\arg_{h\in\mathbb{R}^{D_{\mathcal{H}}}}\max{p(y,\mathcal{S}|\bm{\mathrm{h}})},$
(2)

where $D_{\mathcal{H}}$ is the dimension of common representation. To employ
log likelihood, we assume that

  1. (1)

Each modality is conditionally independent;

  2. (2)

Utterance are independent and identically distributed (i.i.d).

This assumption gives a probability aspect for CRL. Under the above assumption
$(1)$, the probability of label $y$ can be viewed as

$p(y,\mathcal{S}|\bm{\mathrm{h}})=p(y|\bm{\mathrm{h}})p(\mathcal{S}|\bm{\mathrm{h}}),$
(3)

where
$p(\mathcal{S}|\bm{\mathrm{h}})=p(\bm{\mathrm{u}}^{1}|\bm{\mathrm{h}})p(\bm{\mathrm{u}}^{2}|\bm{\mathrm{h}})\dots
p(\bm{\mathrm{u}}^{M}|\bm{\mathrm{h}})$ and $y$ is treated as another semantic
modality, with $M$ being the number of modalities. Here we omit the time token
for simplification.

##  3 Methodology

In this work, we propose a Missing-Modality Robust emotion Recognition
framework (M2R2) for the ERC task. Fig.2 illustrates the overview and detailed
module structures. M2R2 updates PANet and CRL with iterative data augmentation
by each other’s latent representations.

###  3.1 M2R2 Framework

Figure 2: Illustration of Missing-Modality Robust emotion Recognition
framework on ERC task. a) The procedure of M2R2. PANet and CRL are learnt
iteratively on extended dataset with each other’s latent features, where PANet
model predicts emotion label. b) CRL procedure. Given the modality-missing
data, the encoder maps common representation to align modalities. WGAN
regularization is used here to improve construction. c) PANet structure.
Global GRUs and party, emotional GRUs for all parties are employed to tract
and update states. Emotion recognition module exploits multi-layer perception
to predict emotion labels. Detailed attention mechanism module is shown in
right column.

At training stage, PANet model and CRL are updated by turns with other’s
output until the accuracy converges. We first initialize network parameters
$\bm{\Theta}_{e},\\{{\bm{\Theta}_{\bm{\mathrm{u}}}^{m}}\\}_{m=1}^{M}$ and
common representation $\\{\bm{\mathrm{b}}_{n}\\}_{n=1}^{N}$ with random
values. And initializing $\\{\bm{\mathrm{h}}_{n}\\}_{n=1}^{N}$ as zero vectors
to extend data set $\mathcal{D}$ to $\mathcal{D}_{+,h}$ primitively for PANet
model.

Training Stage

Input: Incomplete training conversation dataset:
$\mathcal{D}=\\{(\bm{\mathcal{S}}^{(n)},\bm{y}^{(n)})\\}_{n=1}^{N}$ Hyper-
parameters:
$\lambda_{\mathrm{R}},\lambda_{\mathrm{P}},\lambda_{\mathrm{A}},\lambda_{\mathrm{G}},\alpha_{\mathrm{E}},\alpha_{\mathrm{P}}$

Initialize:
$\bm{\Theta}_{e},\\{\bm{\mathrm{b}}_{n}\\}_{n=1}^{N},\\{{\bm{\Theta}_{\bm{\mathrm{u}}}^{m}}\\}_{m=1}^{M}$
with random values; $\\{\bm{\mathrm{h}}_{n}\\}_{n=1}^{N}$

with zeros.

while  _not converged_ do

Set
$\mathcal{D}_{+,h}=\\{\bm{\mathcal{S}}^{(n)}_{+,h}\\}_{n=1}^{N}=\\{\bm{\mathcal{S}}^{(n)}+\\{\bm{\mathrm{h}}_{n}\\}\\}_{n=1}^{N}$;

for  _$i_{\mathrm{E}}=1:n_{\mathrm{E}}$_ do

Update PANet parameters;

Get intermediate features:
$\\{\bm{\mathrm{b}}_{n}=\bm{\mathrm{e}}^{(n)}\\}_{n=1}^{N}$;

end for

Set
$\mathcal{D}_{+,b}=\\{\bm{\mathcal{S}}^{(n)}_{+,b}\\}_{n=1}^{N}=\\{\bm{\mathcal{S}}^{(n)}+\\{\bm{\mathrm{b}}_{n}\\}\\}_{n=1}^{N}$;

for  _$i_{\mathrm{P}}=1:n_{\mathrm{P}}$_ do

Update the reconstruction parameters and common representation;

end for

end while

Output: $\\{\bm{\Theta}_{\mathrm{E}}^{m}\\}_{m=1}^{M}$ and
$\\{\bm{\mathrm{h}}_{n_{t}}\\}_{n=1}^{N}$;

Testing Stage

Input: Incomplete testing conversation dataset:
$\mathcal{D}=\\{(\bm{\mathcal{T}}^{(n)},\bm{y}^{(n)})\\}_{n=1}^{N}$

Initialize: Initialize $\\{\bm{\Theta}_{ut}^{m}\\}_{m=1}^{M}$ with
$\\{\bm{\Theta}_{u}^{m}\\}_{m=1}^{M}$;

Fine-tune networks parameters;

Get common representations $\bm{\mathrm{h}}^{*}$ of testing data set;

Set
$\\{\bm{\mathcal{T}}^{(n)}_{+,h}\\}_{n=1}^{N}=\\{\bm{\mathcal{T}}^{(n)}+\\{\bm{\mathrm{h}}^{*}\\}\\}_{n=1}^{N}$;

Predict test emotion labels with extended dataset
$\mathcal{D}^{*}=\\{(\bm{\mathcal{T}}^{(n)}_{+,h},\bm{y}^{(n)})\\}_{n=1}^{N}$;

Output: Predicted emotion labels $\hat{y}$.

Algorithm 1 M2R2 Framework

Then the extended data set $\mathcal{D}_{+,h}$ is sent to PANet model to train
and update the ERC task parameters $\bm{\Theta}_{\mathrm{E}}$, getting
intermediate information $\bm{\mathrm{b}}$ at during the conversation
utterance at different time of the model. This intermediate features,
containing conversation topic related(spatial) and time context(temporal)
information, can reinforce the CRL by extending data set to extended one
$\mathcal{D}_{+,b}$ and produce more accurate, abundant common representation
$\\{\bm{\mathrm{h}}_{n}\\}_{n=1}^{N}$. Finally the common representation is
combined with input incomplete data set $\mathcal{D}$, producing new extended
data set $\mathcal{D}_{+,h}$ and using it to train PANet model.

A big challenge lies in narrowing the gap between training and testing in
multimodal representation learning, originating from their difference. We
introduce a fine-tuned strategy to address this problem. The consistency of
common representation from training to testing stage is ensured by optimizing
the objective function $\mathcal{L}^{\prime}_{\mathrm{P}}$ defined in Eq.(17),
getting a new set of parameters $\\{\bm{\Theta}_{ut}^{m}\\}_{m=1}^{M}$ that
applied in test stage. Then test common representation can be calculated by
CRL $\bm{\mathrm{h}}^{*}$. Finally, this common representation is sent to the
trained PANet model with test utterance (incomplete) to get predicted emotion
labels.

###  3.2 Party Attentive Network

Apparently the emotion $e_{t}$ is mainly determined by the utterance
$\bm{\mathrm{u}}_{t}$. In addition, we assume that (1) the conversation scene
has context by proceeding utterances; (2) every party(speaker) has its
emotional context by its utterances 3) the emotion of one party is affected by
the other party’s emotion context.

Our PANet model is shown in Fig.2(b). We model the conversation scene state by
global state, which is updated by the previous speaker’s party-states,
emotional states and current speaker’s utterance, to encode shared information
between all parties. It tracks party state, emotional state and utterance of
all parties, using all proceeding global state jointly with utterance to get
global context for party state.

Furthermore, each party is modeled by party state, which changes as a new
utterance flows in, tracking parties’ dynamic representation related to the
emotional state. Like global context, all proceeding party states with
utterance are exploited to get party context for emotion state. Finally, the
model infers the emotional states of different parties from the corresponding
party states along with other parties’. The emotional states are then sent to
the classification head to get the label.

Global State The purpose of global state
($g\in\mathbb{R}^{D_{\mathcal{G}}},D_{\mathcal{G}}$ represents dimension) is
to model and track conversation scene and shared information between its
parties. Current state $g_{t}$ focuses on current utterance’s impact while
proceeding global ones $g_{1},\dots,g_{t-1}$ reveals the change of
conversation representation, which provide meaningful context for party state.
At time $t$, the global state $g_{t}$ updated through
$\mathrm{GRU}_{\mathrm{G}}$ as follows:

$g_{t}=\mathrm{GRU}_{\mathrm{G}}(g_{t-1},\bm{\mathrm{u}}_{t}\oplus
p^{q_{I(t-1)}}_{t-1}\oplus e^{q_{I(t-1)}}_{t-1}),$ (4)

where $p^{q_{I(t-1)}}_{t-1}$ is the previous speaker’s party state,
$e^{q_{I(t-1)}}_{t-1}$ is the corresponding emotion state and $\oplus$ means
concatenation.

Party State PANet defines party state $p_{1},p_{2},\dots,$ $p_{Q}\in$
$\mathbb{R}^{D_{\mathcal{P}}}$ for each party in conversation scene, where
$D_{\mathcal{P}}$ is the dimension of party state. It represents the speakers’
role in a conversation and serves as bridge between global state with
corresponding emotion state. Proceeding party state can also be exploited as
party context attending to improve the accuracy of emotion state.

To fully take the global state into consideration, we employ Attention
Mechanism [38] to calculate global context
$c^{g}_{t}=\mathrm{ATTN}([g_{1},g_{2},\dots,g_{t}],\bm{\mathrm{u}}_{t})$
related to current utterance:

$\begin{split}\alpha_{t}&={\mathrm{softmax}}(\bm{\mathrm{u}}^{T}_{t}W_{\alpha}[g_{1},g_{2},\dots,g_{t}]),\\\
c_{t}^{g}&=\alpha_{t}^{T}[g_{1},g_{2},\dots,g_{t}],\end{split}$ (5)

where $\alpha_{t}\in\mathbb{R}^{t-1}$ is the attention weight which will be
higher for important factors assigned for all proceeding global state related
to current utterance, $c_{t}\in\mathbb{R}^{D_{\mathcal{G}}}$ is the weighted
sum of proceeding global state by $\alpha_{t}$. The scheme of attention module
for global is shown in Fig.2(b).

Then party state $p_{q}$ is updated respectively by employing global context,
current utterance and previous emotion state:

$p^{q}_{t}=\mathrm{GRU}_{\mathrm{P}}^{q}(p^{q}_{t-1},\bm{\mathrm{u}}_{t}\oplus
c_{t}\oplus e^{q}_{t-1}),\quad q\in\\{1,2,\dots,Q\\},$ (6)

where $p^{q}_{t-1}$ is previous party state, $e^{q}_{t-1}$ is previous
emotional state. Notice that party attention updates using its party and
emotion state instead of the previous speaker’s, which is different from
global states.

Emotion State We assume that every party in the conversation has its emotional
state behind party state and is affected by other ones, and it varies under
different global states. It shows how the conversation scene and other
parties’ state can affect party emotion. Thus we exploit all parties’ state
related to global state to represent party context:

$c^{p_{q}}_{t}=\mathrm{ATTN}([p^{q}_{1},p^{q}_{2},\dots,p^{q}_{t}],g_{t}),\quad
q\in\\{1,2,\dots,Q\\},$ (7)

where function $\mathrm{ATTN}$ is defined as Eq.(5), $p^{q}_{t}$ is the state
of party $q$ at time $t$ and $c^{p_{q}}_{t}\in\mathbb{R}^{D_{\mathcal{P}}}$ is
the party context of $p_{q}$. The scheme of attention module for party context
is shown in Fig.2(b).

All the party context $\\{c^{p_{1}}_{t},c^{p_{2}}_{t},\dots,c^{p_{Q}}_{t}\\}$
and current global state $g_{t}$ are then used to update emotion state by

$\begin{split}e^{q}_{t}=\mathrm{GRU}_{\mathrm{E}}(e^{q}_{t-1},c^{p_{1}}_{t}\oplus
c^{p_{2}}_{t}\oplus\dots c^{p_{Q}}_{t}\oplus g_{t}),\\\ \quad
q\in\\{1,2,\dots,Q\\},\end{split}$ (8)

where $e^{q}_{t}\in\mathbb{R}^{\mathcal{E}}$.

Emotion Classification Head We employ a simple perceptron $\mathcal{C}$ to
classify the speaker’s emotion state as final emotion label:

$\hat{y}_{t}=\arg\max\mathcal{C}(e^{I(t)}_{t}),$ (9)

where $\hat{y}_{t}$ is the predicted emotion label.

Finally, we employ categorical cross-entropy along with L2-regularization as
the objective function:

$\mathcal{L}_{\mathrm{E}}(\bm{\mathrm{x}},\bm{y})=-\frac{1}{T}\sum_{t=1}^{T}{\log{\mathcal{P}_{t}[y_{t}]+\lambda_{\mathrm{E}}\|\bm{\Theta}_{\mathrm{E}}\|_{2}}}$
(10)

where $\bm{\mathrm{x}},\bm{y}$ is the time-sequence data with labels, $T$ is
the total number of utterance in a conversation, $\mathcal{P}_{t}$ is the
distribution of emotion labels for utterance $\bm{\mathrm{u}}_{t}$, $y_{t}$ is
the ground truth, $\lambda_{\mathrm{E}}$ is the L-2 weight and
$\bm{\Theta}_{\mathrm{E}}$ represents the parameters of PANet model.

###  3.3 Common Representation Learning

Enlightened by [41] and take the assumptions in section 2, we use CPM-Nets to
learn common representation. Model $p(\bm{\mathrm{u}}_{i}|\bm{\mathrm{h}})$ as
follows:

$p(\bm{\mathrm{u}}^{m}|\bm{\mathrm{h}})\propto
e^{-l_{r}(\bm{\mathrm{u}}^{m},f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))},\quad
m\in\\{1,2,\dots,M\\},$ (11)

where $\bm{\Theta}_{u}^{m}$ are the parameters responsible for the mapping
$f_{i}(\cdot)$ from common representation $\bm{\mathrm{h}}$ to utterance
modality $\bm{\mathrm{u}}^{m}$, with $l_{r}(\cdot)$ being the reconstruction
loss function.

Thus, the log-likelihood function can be written as

$\begin{split}\mathcal{L}\left(\\{\bm{\mathrm{h}}_{n}\\}_{t=1}^{T},\bm{\Theta}_{u}^{1},\dots,\bm{\Theta}_{u}^{M},\bm{\Theta}_{y})\right)=\sum_{t=1}^{T}{\ln{p(y_{t},\mathcal{S}_{t}|\bm{\mathrm{h}}_{t})}}\\\
\propto-\left(l_{c}(y_{t},g(\bm{\mathrm{h}}_{t}|\bm{\Theta}_{y}))+\sum_{t=1}^{T}{\sum_{m=1}^{M}{l_{r}(\bm{\mathrm{u}}^{m},f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))}}\right).\end{split}$
(12)

The common representation $\bm{\mathrm{h}}_{t}$ is used to reconstruct data
$\bm{\mathrm{u}}^{m}$ by the loss
$l_{r}(\bm{\mathrm{u}}^{m},f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))$, which
is balanced by the classification loss
$l_{c}(y_{n},g(\bm{\mathrm{h}}_{t}|\bm{\Theta}_{y}))$ encourage the alignment
with the ground truth.

For classification task, we define reconstruction loss function as

$\begin{split}l_{r}(\bm{\mathrm{u}}^{m},f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))=\|\bm{\mathrm{u}}^{m}-f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))\|^{2},\\\
\quad m\in\\{1,2,\dots,M\\},\end{split}$ (13)

where the mapping direction is from common representation to different
modalities, making sure all the samples (regardless of their missing patterns)
comparable.

Accordingly, the classification loss is defined as

$\begin{split}l_{c}(y_{t},\hat{y}_{t})=\max(0,\Delta(y_{t},\hat{y})&+\mathbb{E}_{\bm{\mathrm{h}}\sim\mathcal{H}(y)}{F(\bm{\mathrm{h}},\bm{\mathrm{h}}_{t})}\\\
&-\mathbb{E}_{\bm{\mathrm{h}}\sim\mathcal{H}(y_{t})}{F(\bm{\mathrm{h}},\bm{\mathrm{h}}_{t})}),\end{split}$
(14)

where
$\hat{y}=\arg_{y\in\mathcal{Y}}\max{\mathbb{E}_{\bm{\mathrm{h}}\sim\mathcal{H}(y)}{F(\bm{\mathrm{h}},\bm{\mathrm{h}}_{t})}}$
is the predicted label and $\mathcal{H}(y)$ is the set of common
representation for $y$.The loss $\Delta(y_{t},\hat{y})$ is 0 if
$y_{t}=\hat{y}$ else 1.

To better employ observed data and to promote data imputation, we introduce
adversarial strategy in our algorithm. Learning common representation is
guided simultaneously by observed data, unobserved data and labels. The
adversarial loss is calculated by wasserstein GAN(WGAN) [11] as:

$\begin{split}\mathcal{L}_{\mathrm{A}}=\sum_{m=1}^{M}\sum_{i=1}^{I}(1-s_{m})[D_{m}(\hat{\bm{\mathrm{u}}}^{m};\bm{\Theta}_{d}^{m})\\\
-D_{m}(\bm{\mathrm{u}}_{i}^{m};\bm{\Theta}_{d}^{m})+\lambda_{\mathrm{G}}\mathcal{P}_{\mathrm{G}}],\end{split}$
(15)

where
$\hat{\bm{\mathrm{u}}}^{m}=f_{m}(\bm{\mathrm{h}}_{n};\bm{\Theta}_{\bm{\mathrm{u}}}^{m})$
is the generated missing view data,
$\mathcal{P}_{\mathrm{G}}=(\|\nabla_{\hat{\bm{\mathrm{u}}}^{m}}D_{m}(\hat{\bm{\mathrm{u}}}^{m})\|_{2}-1)^{2}$
is the gradient penalty in WGAN and $\lambda_{\mathrm{G}}$ is the balanced
parameter. $(1-s_{m})$ indicates whether or not the sample at scene $n$ and
modality $m$ should be computed. $\\{\bm{\mathrm{u}}_{i}^{m}\\}_{i}^{I}$ are
the observed data, with the total number of samples being $I$. The
discriminator $D_{m}$ of $m$-th view defined by parameter
$\bm{\Theta}_{d}^{m}$ determinate the imputation data between the observed
ones.

Finally, the overall loss function is shown as:

$\begin{split}\mathcal{L}_{\mathrm{P}}(\bm{\mathcal{S}},\bm{y})&=\lambda_{\mathrm{R}}\mathcal{L}_{\mathrm{R}}(\bm{\mathcal{S}},\bm{\mathrm{h}})+\lambda_{\mathrm{C}}\mathcal{L}_{\mathrm{C}}(\bm{y},\bm{\mathrm{h}})+\lambda_{\mathrm{A}}\mathcal{L}_{\mathrm{A}}(\bm{\mathcal{S}})\\\
\mathcal{L}_{\mathrm{R}}(\bm{\mathcal{S}},\bm{\mathrm{h}})&=\frac{1}{T}\sum_{t=1}^{T}\sum_{m=1}^{M}{l_{r}(\bm{\mathrm{u}}^{m},f_{m}(\bm{\mathrm{h}};\bm{\Theta}_{u}^{m}))}\\\
\mathcal{L}_{\mathrm{C}}(\bm{y},\bm{\mathrm{h}})&=\frac{1}{T}\sum_{t=1}^{T}{l_{c}(y_{t},y,\bm{\mathrm{h}}_{t})},\\\
\end{split}$ (16)

where $\bm{\mathcal{S}},\bm{y}$ is the incomplete input data with labels and
$\bm{\mathrm{h}}$ is the common representation learned by reconstruction
function $f_{m}$. The hyper-parameters
$\lambda_{\mathrm{R}},\lambda_{\mathrm{C}},\lambda_{\mathrm{A}}$ are used to
adjust reconstruction loss, classification loss and adversarial loss
respectively.

At testing stage, we employ fine-tune strategy to ensure the consistency by
adjusting the loss function defined as Eq.(13) to narrow the gap between
training set and testing set:

$\mathcal{L}^{\prime}_{\mathrm{P}}(\bm{\mathcal{S}}^{\prime})={l_{r}(\bm{\mathcal{S}}^{\prime},f^{\prime}_{m}(\bm{\mathrm{h}};\bm{\Theta}_{ut}^{m}))},\quad
m\in\\{1,2,\dots M\\},$ (17)

where $\bm{\Theta}_{ut}^{m}$ is the parameters that control the mapping from
test input to its common representation, and $\bm{\mathcal{S}}$ represents the
incomplete testing dataset.

##  4 Experiment Setup

###  4.1 Datasets

We test our model and algorithm on the ERC datasets: IEMOCAP [2], and MELD
[30]. IEMOCAP is a two-party dataset, while MELD is a multiple-party one. We
split both the datasets into train/val and test samples approximately by the
ratio 80/20, guaranteeing that they do not share any party. The detailed
conversations’ and utterances’ distribution of train/val/test samples are
shown in Table 1.

Dataset | partition | #conversations | #utterances | #classes
---|---|---|---|---
IEMOCAP | train/val | 132 | 5810 | 7
test | 31 | 1623
MELD | train/val | 1153 | 11098 | 3&7
test | 280 | 2610

Table 1: Statistics of two datasets.In MELD, the annotation contains both
sentiments(#3) and emotions(#7).

IEMOCAP This dataset is acted, multimodal and multi-speaker containing
approximately 12 hours of dyadic sessions with markers on the face, head, and
hands. The actors performed selected emotional scripts and also improvised
hypothetical scenarios designed to elicit specific types of emotions
(happiness, anger, sadness, frustration, neutral state).

MELD Multimodal EmotionLines Dataset has been created by enhancing and
extending the EmotionLines dataset. MELD contains the same conversation
instances available in EmotionLines, but it also encompasses audio and visual
modality along with the text. MELD has more than 1400 conversations and 13000
utterances from the Friends TV series. Multiple speakers participated in the
conversations. Each utterance in the conversation has been labeled by any of
these seven emotions (happiness, anger, sadness, disgust, surprise, fear,
neutral).

###  4.2 Feature Extraction

Textual Feature We employ convolutional neural networks (CNN) for textual
feature extraction. Following [17], we obtain n-gram features from each
utterance using three distinct convolution filters of sizes 3,4 and 5
respectively, each having 50 feature maps. Outputs are then subjected to max-
pooling and rectified linear unit (ReLU) activation. These activations are
concatenated and fed to a 100-dimensional dense layer, regarded as the textual
utterance representation. Then the whole network is trained with the emotion
labels.

Audio and Visual Feature We follow the identical procedure with [21],
employing 3D-CNN and openSMILE [8] to extract audio and visual feature [16],
respectively.

###  4.3 Baselines

bc-LSTM Biredectional LSTM [29] is used to capture thecontext from the
surrounding utterances to generate context-aware utterance representation.
However, this model does not differentiate among the speakers.

DialogueRNN models the emotion of utterances in a conversation with speaker,
context and emotion information from neighbour utterances[21]. These factors
are modeled using three separate GRU networks to keep track of the individual
speaker states.

ICON uses two GRU networks to learn the utterance representations for
conversations between two participants. The output of the two speakers’ GRUs
is then connected using another GRU that helps perform explicit inter-speaker
modeling.

KET or Knowledge enriched transformers dynamically leverages external
commonsense knowledge using hierarchical self-attention and context-aware
graph attention.

MMER is composed of several sub-modules: three feature extractors, three
unimodal classifiers, three bimodal classifiers and one trimodal
classifier[25]. It leverages a multimodal multitask approach to tackle the
problem of missing modality at test time and training time.

SMIL uses building features reconstruction network to solve the problem of
missing modality. It uses a Bayesian neural network to assess the data
uncertainty as feature regularization to overcome model and data bias produced
by the feature reconstruction network[20].

SMVAE essentially performs infinite imputations by integrating out the missing
data[7]. It first develops a multi-view variational autoencoder model for
fusing multi-modality emotional data and builds a semi-supervised emotion
recognition algorithm.

###  4.4 Network Architectures

We employ bi-directional GRU [33] cells in our PANet model, where global state
dimension $D_{\mathcal{G}}$, party state dimension $D_{\mathcal{P}}$,
emotional state dimension $D_{\mathcal{E}}$ being $512,512,256$ respectively.
For emotion classification head, a simple perceptron composed of 2-layer
linear layers attached with ReLU activation is used. For CPM-Nets, we employ
the fully connected networks with batch normalization and LeakyReLU for all
generators $f_{m}$. Similarly, fully connected networks with LeakyReLU
activation structures are employed for discriminators $D_{m}$. L2
regularization is used with the value of the trade-off parameter being 0.001
for $f_{m}$ and $D_{m}$.

###  4.5 Parameter Settings

At training state, we use Adam optimizer [18] for all parameter update, where
$\beta_{1}=0.5,\beta_{2}=0.999$. The learning rate for PANet model
$\alpha_{\mathrm{E}}$ is $1\text{e-}4$ with the L2 regularization being
$1\text{e-}5$. And learning rates for the generators and discriminators
$\alpha_{\mathrm{E}}$ in CPM-Nets are all $1\text{e-}3$. The trade-off
parameters $\lambda_{\mathrm{R}},\lambda_{\mathrm{P}},\lambda_{\mathrm{A}}$ of
reconstruction, classification and adversarial losses in CRL problem are
$1,10,10$ respectively. Specifically, the parameter for gradient penalty
$\lambda_{\mathrm{G}}$ in WGAN is default as $1$. And we train generators
after discriminators update 2 times in order to guarantee the stability in GAN
structure [22]. In each M2R2 epoch, we train PANet model $n_{\mathrm{E}}$
times and CPM-Nets for $n_{\mathrm{P}}$ times.

##  5 Results and Analysis

###  5.1 PANet Compared with Baselines

(a) IEMOCAP

(b) MELD

Figure 3: Results of different emotion recognition models with missing rate.

Firstly we test our PANet model and baselines at different missing rate
defined as
$\eta={\sum_{t}\sum_{m}{S_{t}}}/{\sum_{t}\sum_{m}{\bm{\mathrm{u}}_{t}^{m}}}$
in section 2 ranging from $0.0$ to $0.6$ by step $0.1$ repeatedly 5 times. To
guarantee that no blank utterance is sent to the model, every row of the
missing matrix must have one element, where zeros vector is used here for the
absent modalities. Additional hidden representations are not used here.

Comparison results of IEMOCAP&MELD datasets are shown in Fig 3, where there
are the following observations: 1) Our PANet model performs better at both
accuracy and F1-score than baselines when the dataset is complete; 2) As
missing rate $\eta$ increases, the degradation of our model is much faint than
baselines, especially on MELD dataset. For example, the relative dropping
accuracy of PANet at MELD dataset from $\eta=0$ to $\eta=0.3$ is $1.55\%$,
while the relative dropping accuracy of the second-best DialogueRNN is
$2.96\%$; 3) Our model is more robust to the modalities missing rate than
other baselines. For instance, the variance of our PANet model related to the
missing rate $\eta$ on the MELD dataset is $2.364$ compared with $3.54099$ of
DialogueRNN.

Methods | Happy | Sad | Neutral | Angry | Excited | Frustrated | Average(w)
---|---|---|---|---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1 Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |
bc-LSTM | 38.74 | 24.24 | 69.34 | 62.35 | 48.72 | 52.35 | 54.43 | 56.33 | 56.32 | 70.33 | 55.38 | 52.37 | 54.28 | 52.99
ICON | 51.16 | 20.09 | 73.73 | 60.80 | 40.97 | 46.05 | 67.87 | 56.87 | 70.48 | 66.11 | 50.07 | 56.07 | 54.49 | 53.73
KET | 56.82 | 23.12 | 76.18 | 65.44 | 42.44 | 48.66 | 65.90 | 58.76 | 67.65 | 62.20 | 51.62 | 57.41 | 55.12 | 54.53
DialogueRNN | 52.89 | 18.65 | 79.53 | 59.24 | 39.01 | 46.31 | 58.15 | 53.97 | 71.34 | 63.20 | 50.89 | 57.73 | 57.11 | 52.40
SMVAE | 39.34 | 25.27 | 71.42 | 65.63 | 49.24 | 53.24 | 55.24 | 56.33 | 58.42 | 73.24 | 54.24 | 53.22 | 55.33 | 54.49
MMER | 41.74 | 26.23 | 73.24 | 67.64 | 54.03 | 55.24 | 57.26 | 57.33 | 60.24 | 74.27 | 56.39 | 53.24 | 56.83 | 55.66
SMIL | 40.35 | 27.50 | 76.82 | 66.78 | 53.03 | 55.10 | 58.73 | 59.53 | 63.87 | 78.13 | 57.39 | 55.85 | 58.68 | 57.42
Ours | 41.35 | 32.84 | 77.82 | 71.42 | 54.72 | 56.72 | 59.73 | 59.49 | 64.87 | 70.80 | 57.46 | 58.51 | 59.64 | 59.17

(a) IEMOCAP

Methods | Happy | Sad | Neutral | Angry | Surprise | Fear | Disgust | Average(w)
---|---|---|---|---|---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
bc-LSTM | 43.65 | 35.27 | 24.24 | 11.24 | 62.38 | 74.27 | 35.42 | 39.46 | 36.35 | 36.35 | - | - | - | - | 51.24 | 49.32
ICON | 37.70 | 36.94 | 0.00 | 0.00 | 62.48 | 72.03 | 36.43 | 36.21 | 24.05 | 17.64 | - | - | - | - | 52.83 | 47.04
KET | 45.13 | 37.24 | 24.33 | 12.37 | 63.37 | 76.24 | 34.24 | 40.83 | 38.37 | 38.46 | - | - | - | - | 53.72 | 51.01
DialogueRNN | 45.68 | 39.43 | 29.16 | 13.01 | 62.25 | 71.37 | 36.52 | 38.67 | 40.02 | 34.92 | - | - | - | - | 54.48 | 50.04
SMVAE | 42.18 | 42.18 | 14.58 | 1.80 | 63.46 | 72.34 | 38.22 | 35.68 | 30.66 | 27.40 | - | - | - | - | 54.24 | 49.12
MMER | 46.24 | 38.24 | 25.38 | 13.24 | 66.24 | 78.32 | 34.18 | 40.24 | 38.37 | 38.37 | - | - | - | - | 54.24 | 51.68
SMIL | 45.68 | 39.60 | 27.86 | 12.88 | 63.72 | 73.09 | 37.25 | 41.74 | 40.16 | 37.22 | - | - | - | - | 55.23 | 52.32
Ours | 44.86 | 42.31 | 30.22 | 3.55 | 66.63 | 80.14 | 38.37 | 36.24 | 37.07 | 32.98 | - | - | - | - | 55.83 | 53.31

(b) MELD

Table 2: Detailed result of different emotion recognition models at missing
rate $\eta=0.5$.

We can also observe that our algorithm performs better on MELD dataset than
IEMOCAP regardless the accuracy or F1-score metric, the possible explanation
is that MELD is a film cut of TV-shows, where the visual and audio expression
is much more clear, exaggerative and severely changes than IEMOCAP as shown in
Fig.4.

Figure 4: Visual/Facial modality comparison of MELD and IEMOCAP, where MELD is
more expressive and vivid.

###  5.2 M2R2 Compared with Baselines

We also test the detailed emotion prediction results of all classes on both
datasets under the missing rate $\eta=0.5$. As Table.2(b)) shows, our model
performs better than baselines on both IEMOCAP and MELD datasets at missing
rate $\eta=0.5$, where metrics are accuracy and F1 score. Significantly, the
M2R2 framework can detect Neutral and Frustrated emotion more accurately,
gaining far more accuracy and F1 socre than the second-best DialogueRNN model.
We presumed that the exploit of party attention related to the global state
provides more hidden emotion representation for the speaker, which neutralizes
the potential opposite emotions from all parties in a conversation.

In further discussion, we test the accuracy and F1 score of both IEMOCAP and
MELD datasets with other multimodal data reconstruction algorithms. The
comparison is in Table 2 as shown that our M2R2 framework achieves the highest
accuracy and F1 score on all emotion classes of both datasets, where accuracy
is about 1% better than the second-best SMIL algorithm on both datasets.

###  5.3 Case Study

Context Attention We test our global and party attention mechanism on IEMOCAP
dataset when missing rate $\eta=0.2$, where the global and different parties’
attention weight
$\alpha_{\mathrm{G}},\alpha_{\mathrm{P_{1}}},\alpha_{\mathrm{P_{2}}}$ at scene
time turn 7 is shown in Fig.5. Observations can be summarized as follwing:

Figure 5: Global and different parties’ attention weight related to all
conversation utterances at scene turn 7. Note that the turn number starts from
0.

  1. (1)

Dependency on utterance at different distances. The attention weight of all
conversation turns related to chosen turns shows that neighboring and distant
utterances can influence the current emotional state. For example, the emotion
at turn 7, where the model correctly predicts ’Disgust’, is influenced by the
nearby utterance at turn 2 and the distant utterance at turn 43. A similar
dependency trend is also shared by party attention mechanism such as the
nearby utterance at turn 6 and the distant utterance at turn 39 both influence
the current emotion prediction at turn 7.

  2. (2)

Dependency on future utterances. The figure also reveals that future
utterances can influence the proceeding emotion greatly. In this discussion,
we can observe that the turn 7 contributes almost half to proceeding turn 7 of
global attention $\alpha_{\mathrm{G}}$ and similarly, turn 36, 37 and turn 39
have a significant influence on their proceeding turn 7 of
$\alpha_{\mathrm{P_{1}}},\alpha_{\mathrm{P_{2}}}$ respectively.

  3. (3)

Dependency on different parties. Like common sense in daily life, one
speaker’s emotional state changes from the speaker’s internal shift and other
speakers’ influence. For example, both turn 8 and 43 strongly attend to the
emotion prediction of turn 7 while turn 8 is uttered by party $q_{I(8)}$ and
turn 43 belongs to the other party.

To avoid the exception error in analysis, we test the global attention weight
at all turns in a conversation scene of the IEMOCAP dataset. The Fig.6 shows
the result. As can be observed, early utterances attend to early scene emotion
prediction more strongly than other times and accordingly, late utterances
attend more strongly to late scene emotion prediction. This trend can be
explained by sequential neural networks’ memory capacity and time.

Figure 6: Global attention weight of all turns in one test conversation scene
at IEMOCAP dataset.

CRL Convergence We show the loss curve of reconstruction and classification
networks at the training and testing stage as shown in Fig.7, where the
dataset used is MELD and repeated 5 times. Observations can be made that
reconstruction loss converges faster and more stable than classification loss,
and testing reconstruction loss has a slower convergence speed.

Figure 7: Reconstruction and classification loss at training and testing
stages with epoch.

Common Representation Clustering The common representation of IEMOCAP dataset
at missing rate $\eta=0.5$ is shown in 8, where t-SNE is exploited for
visualization. As can be observed, the common representation presents obvious
clustering features and margins, especially for sad emotions, which lie at the
lower right corner of the scatter map.

Figure 8: t-SNE result of commom representation of IEMOCAP dataset at missing
rate $\eta=0.5$. Clustering of emotions are observed in feature map.

###  5.4 Ablation Study

This paper’s main novelty is the introduction of party attention and M2R2
framework. We remove these two components once a time and test on the IEMOCAP
dataset at missing rate $\eta=0.5$ to study the influence on them, where the
results are shown in Table 3.

As expected, the proposed two components improve the accuracy and F1 score by
about 3% percent in total. In addition, the M2R2 is more influential than
party attention, where the accuracy falls about 2.20% than 0.9%.

Component | Metrics
---|---
Party Attention | M2R2 Framework | Accuracy | F1 score
✗ | ✗ | 57.82 | 57.43
✔ | ✗ | 58.68 | 58.16
✔ | ✔ | 60.01 | 59.63

Table 3: Ablation results on IEMOCAP at missing rate $\eta=0.5$.

##  6 Conclusions

We proposed the PANet model and M2R2 Framework for ERC task with missing
modalities at the utterance level. The proposed PANet model based on RNN can
handle incomplete utterances by spreading dependency on all conversation
parties. Our algorithm learns PANet and CPM-Nets of incomplete data jointly,
where extended datasets with hidden features for CPM-Nets and common
representation for PANet can provide more versatile information for ERC task.
We empirically validate that the proposed algorithm is relatively robust to
sequential data with missing modalities. More structures and techniques with
suitable common representation learning methods should be tested, which we
plan to explore in the future.

## References

  * [1] Sharmeen M Saleem Abdullah Abdullah, Siddeeq Y Ameen Ameen, Mohammed AM Sadeeq, and Subhi Zeebaree.  Multimodal emotion recognition using deep learning.  Journal of Applied Science and Technology Trends, 2(02):52–58, 2021\.
  * [2] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation, 42(4):335–359, 2008.
  * [3] E. Cambria, Soujanya Poria, Alexander Gelbukh, and Mike A Thelwall.  Sentiment analysis is a big suitcase.  IEEE Intelligent Systems, 32:74–80, 2017.
  * [4] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555, 2014.
  * [5] Nasrin Dehbozorgi, Mary Lou Maher, and Mohsen Dorodchi.  Sentiment analysis on conversations in collaborative active learning as an early predictor of performance.  In 2020 IEEE Frontiers in Education Conference (FIE), pages 1–9. IEEE, 2020.
  * [6] Laurence Devillers, Laurence Vidrascu, and Lori Lamel.  Challenges in real-life emotion annotation and machine learning based detection.  Neural Networks, 18(4):407–422, 2005.  Emotion and Brain.
  * [7] Changde Du, Changying Du, Hao Wang, Jinpeng Li, Wei-Long Zheng, Bao-Liang Lu, and Huiguang He.  Semi-supervised deep generative modelling of incomplete multi-modality emotional data.  In Proceedings of the 26th ACM International Conference on Multimedia, MM ’18, page 108–116, New York, NY, USA, 2018. Association for Computing Machinery.
  * [8] Florian Eyben, Martin Wöllmer, and Björn Schuller.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462, 2010.
  * [9] Jose Maria Garcia-Garcia, Victor MR Penichet, and Maria D Lozano.  Emotion detection: a technology review.  In Proceedings of the XVIII international conference on human computer interaction, pages 1–8, 2017.
  * [10] Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria.  Cosmic: Commonsense knowledge for emotion identification in conversations.  arXiv preprint arXiv:2010.02795, 2020.
  * [11] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of wasserstein gans.  arXiv preprint arXiv:1704.00028, 2017.
  * [12] Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann.  Icon: Interactive conversational memory network for multimodal emotion detection.  In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2594–2604, 2018.
  * [13] Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann.  Conversational memory network for emotion recognition in dyadic dialogue videos.  Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, 2018:2122—2132, June 2018.
  * [14] Chenyang Huang, Amine Trabelsi, and Osmar R Zaïane.  arXiv preprint arXiv:1904.00132, 2019.
  * [15] Xincheng Ju, Dong Zhang, Junhui Li, and Guodong Zhou.  Transformer-based label set generation for multi-modal multi-label emotion detection.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 512–520, 2020.
  * [16] Konstantinos Kamnitsas, Christian Ledig, Virginia FJ Newcombe, Joanna P Simpson, Andrew D Kane, David K Menon, Daniel Rueckert, and Ben Glocker.  Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation.  Medical image analysis, 36:61–78, 2017.
  * [17] Yoon Kim.  Convolutional neural networks for sentence classification.  arXiv: Computation and Language, 2014.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations, 2015.
  * [19] Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou.  Partial multi-view clustering.  In Proceedings of the AAAI conference on artificial intelligence, volume 28, 2014.
  * [20] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng.  Smil: Multimodal learning with severely missing modality.  arXiv preprint arXiv:2103.05677, 2021.
  * [21] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria.  Dialoguernn: An attentive rnn for emotion detection in conversations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818–6825, 2019.
  * [22] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In International conference on machine learning, pages 3481–3490. PMLR, 2018.
  * [23] Weizhi Nie, Rihao Chang, Minjie Ren, Yuting Su, and Anan Liu.  I-gcn: Incremental graph convolution network for conversation emotion detection.  IEEE Transactions on Multimedia, 2021.
  * [24] Shigeyuki Oba, Masa aki Sato, Ichiro Takemasa, Morito Monden, Kenichi Matsubara, and Shin Ishii.  A bayesian missing value estimation method for gene expression profile data.  Bioinformatics, 19:2088–2096, 2003.
  * [25] Mathieu Pagé Fortin and Brahim Chaib-draa.  Multimodal multitask emotion recognition using images, texts and tags.  In Proceedings of the ACM Workshop on Crossmodal Learning and Application, WCRML ’19, page 3–10, New York, NY, USA, 2019. Association for Computing Machinery.
  * [26] Srinivas Parthasarathy and Shiva Sundaram.  Training strategies to handle missing modalities for audio-visual expression recognition.  arXiv: Audio and Speech Processing, 2020.
  * [27] Duc-Anh Phan, Yuji Matsumoto, and Hiroyuki Shindo.  Autoencoder for semisupervised multiple emotion detection of conversation transcripts.  IEEE Transactions on Affective Computing, 12(3):682–691, 2018.
  * [28] Rosalind W Picard.  Affective computing.  MIT press, 2000.
  * [29] Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency.  Context-dependent sentiment analysis in user-generated videos.  In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 873–883, 2017.
  * [30] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea.  Meld: A multimodal multi-party dataset for emotion recognition in conversations.  arXiv preprint arXiv:1810.02508, 2018.
  * [31] Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy.  Emotion recognition in conversation: Research challenges, datasets, and recent advances.  IEEE Access, 7:100943–100953, 2019.
  * [32] Minjie Ren, Xiangdong Huang, Xiaoqi Shi, and Weizhi Nie.  Interactive multimodal attention network for emotion recognition in conversation.  IEEE Signal Processing Letters, 28:1046–1050, 2021.
  * [33] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  IEEE transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [34] Ch. Sekhar, M. Srinivasa Rao, A. S. Keerthi Nayani, and Debnath Bhattacharyya.  Emotion recognition through human conversation using machine learning techniques.  In Debnath Bhattacharyya and N. Thirupathi Rao, editors, Machine Intelligence and Soft Computing, pages 113–122, Singapore, 2021. Springer Singapore.
  * [35] Muhammad Yusril Helmi Setyawan, Rolly Maulana Awangga, and Safif Rafi Efendi.  Comparison of multinomial naive bayes algorithm and logistic regression for intent classification in chatbot.  In 2018 International Conference on Applied Engineering (ICAE), pages 1–5. IEEE, 2018.
  * [36] Catherine Soladié, Hanan Salam, Catherine Pelachaud, Nicolas Stoiber, and Renaud Séguier.  A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection.  In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 493–500, 2012.
  * [37] Olga G. Troyanskaya, Michael N. Cantor, Gavin Sherlock, Patrick O. Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B. Altman.  Missing value estimation methods for dna microarrays.  Bioinformatics, 17:520–525, 2001.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [39] Johannes Wagner, Florian Lingenfelser, and Elisabeth André.  Building a robust system for multimodal emotion recognition.  Emotion recognition: A pattern analysis approach, pages 379–410, 2015.
  * [40] Lei Yuan, Yalin Wang, Paul M. Thompson, Vaibhav A. Narayan, and Jieping Ye.  Multi-source learning for joint analysis of incomplete multi-modality neuroimaging data.  In Knowledge Discovery and Data Mining, 2012.
  * [41] Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu.  Deep partial multi-view learning.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [42] Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou.  Modeling both context-and speaker-sensitive dependence for emotion detection in multi-speaker conversations.  In IJCAI, pages 5415–5421, 2019.
  * [43] Jinming Zhao, Ruichen Li, and Qin Jin.  Missing modality imagination network for emotion recognition with uncertain missing modalities.  In Meeting of the Association for Computational Linguistics, 2021\.
  * [44] Peixiang Zhong, Di Wang, and Chunyan Miao.  Knowledge-enriched transformer for emotion detection in textual conversations.  arXiv preprint arXiv:1909.10681, 2019.