# Transfer Learning in Conversational Analysis through Reusing Preprocessing
Data as Supervisors

Joshua Yee Kim
University of Sydney
<EMAIL_ADDRESS>
&Tongliang Liu
University of Sydney
<EMAIL_ADDRESS>

Kalina Yacef
University of Sydney
<EMAIL_ADDRESS>

###### Abstract

Conversational analysis systems are trained using noisy human labels and often
require heavy preprocessing during multi-modal feature extraction. Using noisy
labels in single-task learning increases the risk of over-fitting. Auxiliary
tasks could improve the performance of the primary task learning during the
same training – this approach sits in the intersection of transfer learning
and multi-task learning (MTL). In this paper, we explore how the preprocessed
data used for feature engineering can be re-used as auxiliary tasks, thereby
promoting the productive use of data. Our main contributions are: (1) the
identification of sixteen beneficially auxiliary tasks, (2) studying the
method of distributing learning capacity between the primary and auxiliary
tasks, and (3) studying the relative supervision hierarchy between the primary
and auxiliary tasks. Extensive experiments on IEMOCAP and SEMAINE data
validate the improvements over single-task approaches, and suggest that it may
generalize across multiple primary tasks.

##  1 Introduction

The sharp increase in uses of video-conferencing creates both a need and an
opportunity to better understand these conversations (Kim et al., 2019a). In
post-event applications, analyzing conversations can give feedback to improve
communication skills (Hoque et al., 2013; Naim et al., 2015). In real-time
applications, such systems can be useful in legal trials, public speaking,
e-health services, and more (Poria et al., 2019; Tanveer et al., 2015).

Analyzing conversations requires both human expertise and a lot of time.
However, to build automated analysis systems, analysts often require a
training set annotated by humans (Poria et al., 2019). The annotation process
is costly, thereby limiting the amount of labeled data. Moreover, third-party
annotations on emotions are often noisy. Deep networks coupled with limited
noisy labeled data increases the chance of overfitting (James et al., 2013;
Zhang et al., 2016). Could data be used more productively?

From the perspective of feature engineering to analyze video-conferences,
analysts often employ pre-built libraries (Baltrušaitis et al., 2016;
Vokaturi, 2019) to extract multimodal features as inputs to training. This
preprocessing phase is often computationally heavy, and the resulting features
are only used as inputs. In this paper, we investigate how the preprocessed
data can be re-used as auxiliary tasks which provide inductive bias through
multiple noisy supervision (Caruana, 1997; Lipton et al., 2015; Ghosn and
Bengio, 1997) and consequently, promoting a more productive use of data.
Specifically, our main contributions are (1) the identification of
beneficially auxiliary tasks, (2) studying the method of distributing learning
capacity between the primary and auxiliary tasks, and (3) studying the
relative supervision hierarchy between the primary and auxiliary tasks. We
demonstrate the value of our approach through predicting emotions on two
publicly available datasets, IEMOCAP (Busso et al., 2008) and SEMAINE (McKeown
et al., 2011).

##  2 Related Works and Hypotheses

Multitask learning has a long history in machine learning (Caruana, 1997). In
this paper, we focus on transfer learning within MTL, a less commonly
discussed subfield within MTL (Mordan et al., 2018). We are concerned with the
performance on one (primary) task – the sole motivation of adding auxiliary
tasks is to improve the primary task performance.

In recent years, this approach has been gaining attention in computer vision
(Yoo et al., 2018; Fariha, 2016; Yang et al., 2018; Mordan et al., 2018;
Sadoughi and Busso, 2018), speech recognition (Krishna et al., 2018; Chen and
Mak, 2015; Tao and Busso, 2020; Bell et al., 2016; Chen et al., 2014), and
natural language processing (NLP) (Arora et al., 2019; Yousif et al., 2018;
Zalmout and Habash, 2019; Yang et al., 2019; Du et al., 2017). The drawback of
adding multiple tasks increases the risk of negative transfer (Torrey and
Shavlik, 2010; Lee et al., 2016, 2018; Liu et al., 2019; Simm et al., 2014),
which leads to many design considerations. Three of such considerations are,
identifying (a) what tasks are beneficial, (b) how much of the model
parameters to share between the primary and auxiliary tasks, and (c) whether
we should prioritize primary supervision by giving it a higher hierarchy than
the auxiliary supervision.

In contrast with previous MTL works, our approach (a) identifies sixteen
beneficially auxiliary targets, (b) dedicates a primary-specific branch within
the network, and (c) investigates the efficacy and generalization of
prioritizing primary supervision across eight primary tasks.

Since our input representation is fully text-based, we dive deeper into MTL
model architecture designs in NLP, Søgaard and Goldberg (2016) found that
lower-level tasks like part-of-speech tagging, are better kept at the lower
layers, enabling the higher-level tasks like Combinatory Categorical Grammar
tagging to use these lower-level representations. In our approach, our model
hierarchy is not based on the difficulty of the tasks, but more simply, we
prioritize the primary task. Regarding identifying auxiliary supervisors in
NLP, existing works have included tagging the input text (Zalmout and Habash,
2019; Yang et al., 2019; Søgaard and Goldberg, 2016). Text classification with
auxiliary supervisors have included research article classification (Du et
al., 2017; Yousif et al., 2018), and tweet classification (Arora et al.,
2019).

Multimodal analysis of conversations has been gaining attention in deep
learning research (Poria et al., 2019). The methods in the recent three years
have been intelligently fusing numeric vectors from the text, audio, and video
modalities before feeding it to downstream layers. This approach is seen in
MFN (Zadeh et al., 2018a), MARN (Zadeh et al., 2018b), CMN (Hazarika et al.,
2018b), ICON (Hazarika et al., 2018a), DialogueRNN (Majumder et al., 2019),
and M3ER (Mittal et al., 2020). Our approach is different in two ways. (1) Our
audio and video information is encoded within text before feeding only the
text as input. Having only text as input has the benefits of interpretability,
and the ability to present the conversational analysis on paper (Kim et al.,
2019b), similar to how the linguistics community performs manual
conversational analysis using the Jefferson transcription system (Jefferson,
2004), where the transcripts are marked up with symbols indicating how the
speech was articulated. (2) Instead of using the audio and video information
as only inputs, we demonstrate how to use multimodal information in both input
and as auxiliary supervisors.

Hypothesis H1: The introduced set of auxiliary supervision features improves
primary task performance. We introduce and motivate the full set of sixteen
auxiliary supervisions, all based on existing literature: these are grouped
into four families, each with four auxiliary targets. The four families are
(1) facial action units, (2) prosody, (3) historical labels, and (4) future
labels:
(1) Facial action units, from the facial action coding system identifies
universal facial expressions of emotions (Ekman, 1997). Particularly, AU 05
(upper lid raiser), 17 (chin raiser), 20 (lip stretcher), 25 (lips part) have
been shown to be useful in detecting depression (Yang et al., 2016a; Kim et
al., 2019b) and rapport-building (Kim et al., 2021).
(2) Prosody, the tone of voice – happiness, sadness, anger, and fear – can
project warmth and attitudes (Hall et al., 2009), and has been used as inputs
in emotions detection (Garcia-Garcia et al., 2017).
(3 and 4) Using features at different historical time-points is a common
practice in statistical learning, especially in time-series modelling (Christ
et al., 2018). Lastly, predicting future labels as auxiliary tasks can help in
learning (Caruana et al., 1996; Cooper et al., 2005; Trinh et al., 2018; Zhu
et al., 2020; Shen et al., 2020). We propose using historical and future (up
to four talkturns ago or later) target labels as auxiliary targets.

Figure 1: Reusing components (dotted lines) of the feature engineering process
as auxiliary targets (in blue). The MONAH framework (Kim et al., 2021) is
introduced in section 4.2

Given that we are extracting actions and prosody families as inputs, we
propose to explore whether they can be reused as supervisors (see Fig. 1). Our
hypothesis H1 is that re-using them as auxiliary supervision improves primary
task performance. This is related to using hints in the existing MTL
literature, where the auxiliary tasks promote the learning of the feature
(Cheng et al., 2015; Yu and Jiang, 2016).

Hypothesis H2: When the primary branch is given maximum learning capacity, it
would not be outperformed by models with primary branch having less than the
maximum learning capacity. Deeper models with higher learning capacity produce
better results (Huang et al., 2019; Nakkiran et al., 2019; Menegola et al.,
2017; Blumberg et al., 2018; Romero et al., 2014). Also, since the auxiliary
branch is shared with the primary supervision, the auxiliary capacity should
be limited to improve primary task performance (Wu et al., 2020) because
limiting the auxiliary capacity will force the branch to learn common
knowledge (instead of auxiliary specific knowledge) across the auxiliary tasks
(Arpit et al., 2017). Therefore, given a fixed learning capacity budget, our
hypothesis H2 implies that we should allocate the maximum learning capacity to
the primary branch because we care only about the primary task performance.

Hypothesis H3: Auxiliary supervision at the lower hierarchy yields better
primary task performance as compared to flat-MTL. Having the auxiliary tasks
at the same supervisory level as the primary task is inherently sub-optimal
because we care only about the performance of the primary task (Mordan et al.,
2018). Information at the lower hierarchy learns basic structures that are
easy to transfer, whilst upper hierarchy learns more semantic information that
is less transferable (Zeiler and Fergus, 2014). Therefore, we propose that the
auxiliary supervision to be at the lower hierarchy than the primary
supervision.

##  3 Model Architecture

###  3.1 Flat-MTL Hierarchical Attention Model

We start with an introduction of the Hierarchical Attention Model (HAN) (Yang
et al., 2016b). We chose HAN because of its easy interpretability as it only
uses single-head attention layers. There are four parts to the HAN model, (1)
text input, (2) word encoder, (3) talkturn encoder, and (4) the predictor. In
our application, we perform our predictions at the talkturn-level for both
IEMOCAP and SEMAINE. For notation, let $s_{i}$ represent the i-th talkturn and
$w_{it}$ represent the t-th word in the i-th talkturn. Each single talkturn
can contain up to T words, and each input talkturn can contain up to L past
talkturns to give content context (see section 4.2).

Given a talkturn of words, we first convert the words into vectors through an
embedding matrix $W_{e}$, and the word selection one-hot vector, $w_{it}$. The
word encoder comprises of bidirectional GRUs (Bahdanau et al., 2014) and a
single head attention to aggregate word embeddings into talkturn embeddings.
Given the vectors $x_{it}$, the bidirectional GRU reads the words from left to
right as well as from right to left (as indicated by the direction of the GRU
arrows) and concatenates the two hidden states together to form $h_{it}$. We
then aggregate the hidden states into one talkturn embedding through the
attention mechanism. $u_{it}$ is the hidden state from feeding $h_{it}$ into a
one-layer perceptron (with weights $W_{w}$ and biases $b_{w}$). The attention
weight ($\alpha_{it}$) given to $u_{it}$ is the softmax normalized weight of
the similarity between itself ($u_{it}$) and $u_{w}$, which are all randomly
initialized and learnt jointly.

Figure 2: Forward pass of the Flat-MTL HAN architecture. Auxiliary tasks
(yellow) are added at the same level of the primary task (orange). Figure 3:
Forward pass of the HAN-ROCK architecture. There is a primary branch (orange)
where auxiliary supervision (yellow) can not influence. The fusion module
(blue) aggregates the talkturn embeddings from all tasks into one.

$\displaystyle x_{it}=W_{e}w_{it},t\in[1,T].$
$\displaystyle\overrightarrow{h}_{it}=\overrightarrow{GRU}(x_{it}),t\in[1,T].$
$\displaystyle\overleftarrow{h}_{it}=\overleftarrow{GRU}(x_{it}),t\in[T,1].$
$\displaystyle{h}_{it}=(\overrightarrow{h}_{it},\overleftarrow{h}_{it}).$
$\displaystyle u_{it}=relu(W_{w}h_{it}+b_{w}).$ $\displaystyle
s_{i}=\Sigma_{t}\alpha_{it}u_{it}.$
$\displaystyle\alpha_{it}=\frac{exp(u_{it}^{\top}u_{w})}{\Sigma_{t}exp(u_{it}^{\top}u_{w})}.$

With the current and past talkturn embeddings (content context, to discuss in
section 4.2), the talkturn encoder aggregates them into a single talkturn
representation ($v$) in a similar fashion, as shown below.

$\displaystyle\overrightarrow{h}_{i}=\overrightarrow{GRU}(s_{i}),i\in[1,L].$
$\displaystyle\overleftarrow{h}_{i}=\overleftarrow{GRU}(s_{i}),i\in[L,1].$
$\displaystyle{h}_{i}=(\overrightarrow{h}_{i},\overleftarrow{h}_{i}).$
$\displaystyle u_{i}=relu(W_{s}h_{i}+b_{s}).$
$\displaystyle\alpha_{i}=\frac{exp(u_{i}^{\top}u_{s})}{\Sigma_{i}exp(u_{i}^{\top}u_{s})}.$
$\displaystyle v=\Sigma_{i}\alpha_{i}u_{i}.$

The simplest way of adding the sixteen auxiliary task predictors would be to
append them to where the primary task predictor is, as illustrated in Fig. 2.
That way, all predictors use the same representation $v$. We refer to this
architecture as flat-MTL, but we are unable to test H2 and H3 using this
architecture. Therefore, we introduce HAN-ROCK next.

###  3.2 HAN-ROCK

We adapted111Implementation will be available at GitHub; Please see attached
supplementary material during the review phase. the ROCK architecture (Mordan
et al., 2018) which was built for Convolutional Neural Networks (LeCun et al.,
1995) found in ResNet-SSD (He et al., 2016; Liu et al., 2016) to suit GRUs
(Bahdanau et al., 2014) found in HAN (Yang et al., 2016b) (see Fig. 3). To
study H3, we bring the auxiliary task predictors forward (see Fig. 3), so that
the back-propagation from the primary supervision is able to temper the back-
propagation from the auxiliary supervision but not vice-versa. This also sets
us up to study H2. Each of the auxiliary tasks has its own talkturn encoder
but shares one word encoder in the auxiliary branch (to keep the network
small). Subscript $a$ indicates whether the word encoder is for the primary or
auxiliary branch:

$\displaystyle x_{it}=W_{e}w_{it},t\in[1,T]$
$\displaystyle\overrightarrow{h}_{ait}=\overrightarrow{GRU_{a}}(x_{it}),t\in[1,T],a\in\\{pri,aux\\}$
$\displaystyle\overleftarrow{h}_{ait}=\overleftarrow{GRU_{a}}(x_{it}),t\in[T,1],a\in\\{pri,aux\\}$
$\displaystyle{h}_{ait}=(\overrightarrow{h}_{ait},\overleftarrow{h}_{ait})$
$\displaystyle u_{ait}=relu(W_{aw}h_{ait}+b_{aw})$
$\displaystyle\alpha_{ait}=\frac{exp(u_{ait}^{\top}u_{aw})}{\Sigma_{t}exp(u_{ait}^{\top}u_{aw})}$
$\displaystyle s_{ai}=\Sigma_{t}\alpha_{ait}u_{ait}$

Figure 4: Example of a MONAH transcript.

Each task has its own talkturn encoder. Subscript $b$ indicates which of the
seventeen tasks – the primary talkturn task or one of the sixteen auxiliary
tasks – is the talkturn encoder is dedicated to:

$\displaystyle\overrightarrow{h}_{abi}=\overrightarrow{GRU_{ab}}(s_{ai}),i\in[1,L],a\in\\{pri,aux\\},$
$\displaystyle b\in\\{pri,aux1,aux2,...,aux16\\}$
$\displaystyle\overleftarrow{h}_{abi}=\overleftarrow{GRU_{ab}}(s_{ai}),i\in[L,1],a\in\\{pri,aux\\},$
$\displaystyle b\in\\{pri,aux1,aux2,...,aux16\\}$
$\displaystyle{h}_{abi}=(\overrightarrow{h}_{abi},\overleftarrow{h}_{abi})$
$\displaystyle u_{abi}=relu(W_{b}h_{abi}+b_{b})$
$\displaystyle\alpha_{abi}=\frac{exp(u_{abi}^{\top}u_{b})}{\Sigma_{i}exp(u_{abi}^{\top}u_{b})}$
$\displaystyle v_{ab}=\Sigma_{i}\alpha_{abi}u_{abi}$

The seventeen talkturn embeddings ($v_{ab}$) goes through a concatenation,
then the single head attention, aggregating talkturn embeddings across
seventeen tasks into one talkturn embedding for the primary task predictor.
Subscript $c$ pertains to the fusion module.

$\displaystyle\text{concatenation: }v_{c}=(v_{ab}),a\in\\{pri,aux\\},$
$\displaystyle b\in\\{pri,aux1,aux2,...,aux16\\}$
$\displaystyle\text{attention:
}\alpha_{c}=\frac{exp(v_{c}^{\top}u_{c})}{\Sigma_{c}exp(v_{c}^{\top}u_{c})}$
$\displaystyle\text{overall primary talkturn vector:
}v=\Sigma_{c}\alpha_{c}v_{c}$

##  4 Experiments

###  4.1 Data and Primary Tasks

We validate our approach using two datasets with a total of eight primary
tasks: the IEMOCAP (Busso et al., 2008) and the SEMAINE (McKeown et al., 2011)
datasets. Both datasets are used in multimodal emotions detection research
(Poria et al., 2019). We divided the datasets into train, development, and
test sets in an approximate 60/20/20 ratio such that the sets do not share any
speaker (Appendix A.1 details the splits).

The target labels of the eight primary tasks are all at the talkturn-level.
The four primary tasks of IEMOCAP consists of the four-class emotions
classification (angry, happy, neutral, sad), and three regression problems –
valence (1-negative, 5-positive), activation (1-calm, 5-excited), and
dominance (1-weak, 5-strong). The four-class emotions classification target is
common for IEMOCAP (Latif et al., 2020; Xia and Liu, 2015; Li et al., 2019;
Hazarika et al., 2018b; Mittal et al., 2020). For SEMAINE, there are four
regression problems – activation, intensity, power, valence. We use two
standard evaluation metrics, mean absolute error (MAE), and 4-class weighted
mean classification accuracy, MA(4).

###  4.2 Input

Multimodal feature extraction is computed using the MONAH framework (Kim et
al., 2021). This framework uses a variety of pre-trained models to extract
nine multimodal features, associated with the prosody of the speech and the
actions of the speaker, and weaves them into a multimodal text narrative. We
refer the reader to Kim et al. (2021) for the details and efficacy of the
MONAH framework. The benefit of the created narrative is that it describes
what is said together with how it is said for each talkturn, giving richer
nonverbal context to the talkturn (see Fig. 4 for an example). Being fully
text-based means that the analysis product can be printed out on paper,
without the need for speakers nor monitors to replay the conversation on a
computer.

In addition to nonverbal context, we concatenated a variable number of
preceding talkturns to the current talkturn as content context. Content
context has been proven to be useful in CMN (Hazarika et al., 2018b) and ICON
(Hazarika et al., 2018a), DialogueRNN (Majumder et al., 2019). The content-
context size is tuned as a hyperparameter. The resulting multimodal text
narrative, consisting of both nonverbal and context context, is used as the
sole input to the model.

###  4.3 Auxiliary Targets

We first clarify the method of extraction for the auxiliary families. The
OpenFace algorithm (Baltrušaitis et al., 2016) is used to extract the four
continuous facial action units (AU) – AU 05, 17, 20, 25. The Vokaturi
algorithm (Vokaturi, 2019) is used to extract the four continuous dimensions
in the tone of voice – happiness, sadness, anger, and fear. As for historical
and future features, we simply look up the target label for the past four
talkturns and future four talkturns. Any label that is not available (for
example, the label four talkturns ago is not available for the third talkturn)
is substituted with the next nearest non-missing label.

All auxiliary targets that reused the input features (actions and prosody) are
converted into a percentile rank that has the range [0,1] using the values
from the train partition. This is a subtle but note-worthy transformation.
When reusing an input as an auxiliary target, it would be trivial if the input
can easily predict the target. For example, given the following MONAH
transcript as input, “The woman sadly and slowly said no." It would be trivial
to use a binary (quantized) auxiliary target of “was the tone sad?" because we
would only be training the model to look for the word “sadly". However, if the
auxiliary target is a percentile rank (less quantized) of the sadness in tone,
then the presence of the word “sadly" increases the predicted rank, but the
model could still use the rest of the nonverbal cues (“slowly") and what is
being said (“no") to predict the degree of sadness. That way, representations
learnt for the auxiliary tasks uses more of the input.

Percentile rank also has the convenient property of having the range [0,1]. We
scaled the percentile ranks so that they all have the same range as the
primary task (see appendix A.2 for transformation details). This ensures that
if we assigned equal loss weights to all tasks, the contribution of every task
is of the same order of magnitude (Gong et al., 2019; Hassani and Haley, 2019;
Sener and Koltun, 2018).

Table 1: H1 Results. *: the model performance has a statistically significant difference with the baseline model. a: action, p: prosody, h: historical labels, f: future labels. | IEMOCAP | SEMAINE
---|---|---
Aux. Target |  Classif.
MA(4) |  Val.
MAE |  Act.
MAE |  Dom.
MAE |  Act.
MAE |  Int.
MAE |  Pow.
MAE |  Val.
MAE
None (Baseline) |  0.625 |  0.527 |  0.518 |  0.667 |  0.194 |  0.238 |  0.170 |  0.177
ap |  0.715* |  0.538 |  0.507 |  0.600* |  0.184* |  0.218* |  0.167* |  0.178
aphf |  0.706* |  0.497* |  0.504 |  0.587* |  0.187 |  0.231 |  0.165* |  0.169
Table 2: H2 Results. *: the model performance has a statistically significant difference with the baseline model (P = 256). ^: Assigning 1 GRU to the auxiliary task talkturn encoder yields a statistically significant difference with assigning 0 GRU. | IEMOCAP | SEMAINE
---|---|---
Pri. GRU |  Classif.
MA(4) |  Val.
MAE |  Act.
MAE |  Dom.
MAE |  Act.
MAE |  Int.
MAE |  Pow.
MAE |  Val.
MAE
256 (Baseline) |  0.715^ |  0.538 |  0.507 |  0.587^ |  0.187 |  0.231 |  0.165^ |  0.169^
192 |  0.736 |  0.509 |  0.518 |  0.604 |  0.187 |  0.228 |  0.165 |  0.184*
128 |  0.711 |  0.537 |  0.512 |  0.597 |  0.189 |  0.216 |  0.176* |  0.196*
64 |  0.687 |  0.540 |  0.507 |  0.593 |  0.191 |  0.234 |  0.167 |  0.192*
1 |  0.656 |  0.554 |  0.509 |  0.599 |  0.190 |  0.229 |  0.168* |  0.191*

###  4.4 Models, training, and hyperparameters tuning

The overall loss is calculated as the weighted average across all seventeen
tasks: (1) we picked a random weight for the primary task from the range
[0.50, 0.99]; this ensures that the primary task has the majority weight. (2)
For the remaining weights (1 - primary weight), we allocated them to the
sixteen auxiliary tasks by: (a) random, (b) linearly-normalized mutual
information, or (c) softmax-normalized mutual information. (a) is self-
explanatory. As for (b) and (c), mutual information has been shown that it is
the best predictor – compared to entropy and conditional entropy – of whether
the auxiliary task would be helpful (Bjerva, 2017). We computed the mutual
information (vector $m$) of each auxiliary variable with the primary target
variable (Kraskov et al., 2004; Ross, 2014) using scikit-learn (Pedregosa et
al., 2011). Then, we linearly-normalized or softmax-normalized $m$ to sum up
to 1. Finally, we multiplied the normalized $m$ with the remaining weights
from (2); this ensures that the primary weight and the sixteen auxiliary
weight sum up to one. (a), (b), and (c) have ten trials each during hyper-
parameters tuning.

Two variants of the HAN architectures are used (Fig 2 and 3). For hypotheses
testing, we bootstrapped confidence intervals (appendix A.4).

##  5 Results and Discussion

The key takeaways are: (H1) The introduced set of auxiliary supervision
improves primary task performance significantly in six of the eight primary
tasks. (H2) Maximum learning capacity should be given to the primary branch as
a default. (H3) HAN-ROCK is unlikely (in one of the eight tasks) to degrade
primary task performance significantly, and sometimes significantly improves
it (in four of the eight tasks).

(H1): To test H1 (whether the introduced set of auxiliary supervision improves
primary task performance), we first train the model with all sixteen auxiliary
targets (from families: actions, prosody, historical, and future). Then, to
differentiate the effect from the historical and future supervision, we set
the loss weights from historical and future targets to be zero; effectively,
there is only supervision from eight auxiliary targets (actions and prosody).
Lastly, for the baseline model (no auxiliary supervision), we set the loss
weights from all sixteen auxiliary targets to zero.

Table 3: H3 Results. *: the model performance has a significant difference with the baseline. | IEMOCAP | SEMAINE
---|---|---
Hierarchy |  Classif.
MA(4) |  Val.
MAE |  Act.
MAE |  Dom.
MAE |  Act.
MAE |  Int.
MAE |  Pow.
MAE |  Val.
MAE
Flat (Baseline) |  0.699 |  0.520 |  0.526 |  0.606 |  0.183 |  0.230 |  0.164 |  0.185
HAN-ROCK |  0.715 |  0.538 |  0.507* |  0.600* |  0.184 |  0.218* |  0.167* |  0.178*
Table 4: Class-wise classification F1 score on IEMOCAP. Baseline (challenger) refers to HAN-Rock architecture under the three hypotheses. *: the challenger performance has a statistically significant difference with the baseline model. Distribution | H1 | H2 | H3 |  SoTA
---|---|---|---|---
Label |  Count |  Baseline
Aux target: None |  Challenger
Aux target: aphf |  Baseline
Pri GRU: 256 |  Challenger
Pri GRU: 1 |  Baseline
Hier-archy: Flat |  Challenger
Hier-archy: HAN ROCK |  M3-ER
Sad |  1,084 |  0.573 |  0.689* |  0.699 |  0.591* |  0.674 |  0.704* |  0.775
Anger |  1,103 |  0.531 |  0.683 |  0.752 |  0.672* |  0.657 |  0.720* |  0.862
Happy |  1,636 |  0.772 |  0.784 |  0.776 |  0.754* |  0.804 |  0.806 |  0.862
Neutral |  1,708 |  0.664 |  0.636 |  0.688 |  0.627* |  0.645 |  0.631 |  0.745

Given auxiliary supervision, the model significantly outperforms the baseline
of not having auxiliary supervision in six out of the eight primary tasks
(Table 1). Comparing the baseline model with the model with two auxiliary
target families, they significantly outperformed the baseline model in five
out of eight primary tasks. The addition of two auxiliary target families
(historical and future labels) sometimes significantly improved primary task
performance (valence in IEMOCAP), but it also sometimes significantly made it
worse (activation and intensity in SEMAINE). This shows that the value of
auxiliary tasks, and the associated risk of negative transfer, depends on the
auxiliary task.

(H2): To test H2 (whether maximum learning capacity should be given to the
primary branch), we let P represent the number of GRU assigned to the primary
talkturn encoder, and A represent the number of GRU assigned to each of the
sixteen auxiliary talkturn encoder. We constrained P \+ A to be equal to 257.
During our experiments, we set P to 1, 64, 128, 192, and 256. We set 256 as
the baseline model because it is the maximum learning capacity we can give to
the primary branch while giving 1 GRU ($=257-256$) to each of the sixteen
auxiliary talkturn encoders.

In all primary tasks, the baseline model of assigning 256 GRUs to the primary
branch is not significantly outperformed by models that assigned 1, 64, 128,
192 GRUs (Table 2). Generally, the performance decreased as the number of GRUs
assigned to the primary talkturn encoder decreased from 256 to 1. We observed
significantly worse performance in two out of eight tasks – in power and
valence in SEMAINE. Also, assigning 256 GRU to the primary talkturn encoders
and 1 to each of the sixteen auxiliary talkturn encoders yields the smallest
model222As opposed to assigning 1 GRU to the primary talkturn encoder and 256
GRU to each of the sixteen auxiliary encoder., and thus trains the fastest.
Therefore, we recommend that the maximum capacity be given to the primary
branch as a default.

That said, the presence of an auxiliary branch is still important. The
baseline of H1 (no auxiliary supervision, Table 1) can be approximated333Same
model architecture except that the loss weights of all auxiliary tasks are
zero as P=$256+16\times 1$, A=0 . We compared the former to the baseline in
Table 2, and found that four out of eight primary tasks have significant
improvements by changing the number of talkturn encoders assigned to each
auxiliary task from zero to one.

(H3): To test H3 (whether auxiliary supervision should be given a lower
hierarchy), we compare the results from the flat-MTL HAN architecture
(baseline) against the HAN-ROCK architecture (Table 3). Placing auxiliary
supervision at the lower hierarchy significantly improves primary task
performance in four out of eight tasks. In only one out of eight tasks (power
in SEMAINE), auxiliary supervision significantly degrades primary task
performance. Further improvements are possible through the fusion module with
future research.

###  5.1 Class-wise Performance and SoTA

Generally, we found that all hypotheses effects are stronger in lower resource
labels (sad and anger, Table 4). We also present the performance of M3ER
(Mittal et al., 2020), a previous state-of-the-art (SoTA) approach. We do not
expect the performance of our text-only input to match the SoTA approach,
which is confirmed in Table 4. By fusing numerical vectors from the three
modalities prevalent in SoTA approaches (Zadeh et al., 2018a, b; Hazarika et
al., 2018b, a; Majumder et al., 2019; Mittal et al., 2020), the inputs are of
a much higher granularity as compared to our approach of describing the
multimodal cues using discrete words. Although the text-based input is likely
to constrain model performance, the multimodal transcription could be helpful
for a human to analyze the conversation, we could also overlay the model
perspective on the multimodal transcription to augment human analysis (see
Appendix A.5).

##  6 Conclusion

We proposed to re-use feature engineering pre-processing data as auxiliary
tasks to improve performance and transfer learning. Three hypotheses were
tested. The experimental results confirm H1 – Introducing our set of sixteen
auxiliary supervisors resulted in better performance in most primary tasks.
For H2, maximum learning capacity should be given to the primary branch.
Lastly, for H3, placing the auxiliary supervision in a lower hierarchy is
unlikely to hurt performance significantly, and it sometimes significantly
improves performance. This is encouraging news for multi-modal conversational
analysis systems as we have demonstrated how pre-processed data can be used
twice to improve performance, once as inputs, and again as auxiliary tasks.

The first limitation of our paper is that the solutions are evaluated on eight
tasks in the conversational analysis domain, and it is not clear if these
would generalize outside of this domain. The second limitation is that we have
evaluated on HAN, but not on other network architectures.

A challenge to be addressed is the apriori selection of the auxiliary targets.
Future research could investigate targets selection, including how to use a
much larger range of auxiliary targets, how to decide the optimum number of
auxiliary targets, and whether it is possible to perform these automatically.

## References

  * Arora et al. (2019) Udit Arora, William Scott Paka, and Tanmoy Chakraborty. 2019.  Multitask learning for blackmarket tweet detection.  In _Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_ , pages 127–130.
  * Arpit et al. (2017) Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017.  A closer look at memorization in deep networks.  In _International Conference on Machine Learning_ , pages 233–242. PMLR.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Baltrušaitis et al. (2016) Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016.  Openface: an open source facial behavior analysis toolkit.  In _Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on_ , pages 1–10. IEEE.
  * Bell et al. (2016) Peter Bell, Pawel Swietojanski, and Steve Renals. 2016.  Multitask learning of context-dependent targets in deep neural network acoustic models.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 25(2):238–247.
  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio. 2012.  Random search for hyper-parameter optimization.  _The Journal of Machine Learning Research_ , 13(1):281–305.
  * Bjerva (2017) Johannes Bjerva. 2017.  Will my auxiliary tagging task help? estimating auxiliary tasks effectivity in multi-task learning.  In _Proceedings of the 21st Nordic Conference on Computational Linguistics_ , pages 216–220.
  * Blumberg et al. (2018) Stefano B Blumberg, Ryutaro Tanno, Iasonas Kokkinos, and Daniel C Alexander. 2018\.  Deeper image quality transfer: Training low-memory neural networks for 3d images.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 118–125. Springer.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42(4):335.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine learning_ , 28(1):41–75.
  * Caruana et al. (1996) Rich Caruana, Shumeet Baluja, and Tom Mitchell. 1996.  Using the future to “sort out" the present: Rankprop and multitask learning for medical risk evaluation.  In _Advances in neural information processing systems_ , pages 959–965.
  * Chen et al. (2014) Dongpeng Chen, Brian Mak, Cheung-Chi Leung, and Sunil Sivadas. 2014.  Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition.  In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5592–5596. IEEE.
  * Chen and Mak (2015) Dongpeng Chen and Brian Kan-Wing Mak. 2015.  Multitask learning of deep neural networks for low-resource speech recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 23(7):1172–1183.
  * Cheng et al. (2015) Hao Cheng, Hao Fang, and Mari Ostendorf. 2015.  Open-domain name error detection using a multi-task rnn.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 737–746.
  * Christ et al. (2018) Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. 2018.  Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package).  _Neurocomputing_ , 307:72–77.
  * Cooper et al. (2005) Gregory F Cooper, Vijoy Abraham, Constantin F Aliferis, John M Aronis, Bruce G Buchanan, Richard Caruana, Michael J Fine, Janine E Janosky, Gary Livingston, Tom Mitchell, et al. 2005.  Predicting dire outcomes of patients with community acquired pneumonia.  _Journal of biomedical informatics_ , 38(5):347–366.
  * Du et al. (2017) Yongping Du, Yunpeng Pan, and Junzhong Ji. 2017.  A novel serial deep multi-task learning model for large scale biomedical semantic indexing.  In _2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 533–537. IEEE.
  * Ekman (1997) Rosenberg Ekman. 1997.  _What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS)_.  Oxford University Press, USA.
  * Fariha (2016) Anna Fariha. 2016.  Automatic image captioning using multitask learning.  In _In the Proceedings of Neural Information Processing Systems_ , volume 20, pages 11–20.
  * Garcia-Garcia et al. (2017) Jose Maria Garcia-Garcia, Victor MR Penichet, and Maria D Lozano. 2017.  Emotion detection: a technology review.  In _Proceedings of the XVIII international conference on human computer interaction_ , pages 1–8.
  * Ghosn and Bengio (1997) Joumana Ghosn and Yoshua Bengio. 1997.  Multi-task learning for stock selection.  In _Advances in neural information processing systems_ , pages 946–952.
  * Golovin et al. (2017) Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. 2017.  Google vizier: A service for black-box optimization.  In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1487–1495.
  * Gong et al. (2019) Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndirango, Gokce Keskin, and Oguz H Elibol. 2019.  A comparison of loss weighting strategies for multi task learning in deep neural networks.  _IEEE Access_ , 7:141627–141632.
  * Hall et al. (2009) Judith A Hall, Debra L Roter, Danielle C Blanch, and Richard M Frankel. 2009.  Observer-rated rapport in interactions between medical students and standardized patients.  _Patient Education and Counseling_ , 76(3):323–327.
  * Hassani and Haley (2019) Kaveh Hassani and Mike Haley. 2019.  Unsupervised multi-task feature learning on point clouds.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 8160–8171.
  * Hazarika et al. (2018a) Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018a.  Icon: Interactive conversational memory network for multimodal emotion detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2594–2604.
  * Hazarika et al. (2018b) Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018b.  Conversational memory network for emotion recognition in dyadic dialogue videos.  In _Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting_ , volume 2018, page 2122. NIH Public Access.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Hoque et al. (2013) Mohammed Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W Picard. 2013.  Mach: My automated conversation coach.  In _Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing_ , pages 697–706.
  * Huang et al. (2019) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  In _Advances in neural information processing systems_ , pages 103–112.
  * James et al. (2013) Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013.  _An introduction to statistical learning_ , volume 112.  Springer.
  * Jefferson (2004) Gail Jefferson. 2004.  Glossary of transcript symbols with an introduction.  _Pragmatics and Beyond New Series_ , 125:13–34.
  * Kim et al. (2019a) Joshua Y Kim, Rafael A Calvo, Kalina Yacef, and NJ Enfield. 2019a.  A review on dyadic conversation visualizations-purposes, data, lens of analysis.  _arXiv preprint arXiv:1905.00653_.
  * Kim et al. (2019b) Joshua Y Kim, Greyson Y Kim, and Kalina Yacef. 2019b.  Detecting depression in dyadic conversations with multimodal narratives and visualizations.  In _Australasian Joint Conference on Artificial Intelligence_ , pages 303–314. Springer.
  * Kim et al. (2021) Joshua Y Kim, Kalina Yacef, Greyson Y Kim, Chunfeng Liu, Rafael Calvo, and Silas CR Taylor. 2021.  Monah: Multi-modal narratives for humans to analyze conversations.  _arXiv preprint arXiv:2101.07339_.
  * Kraskov et al. (2004) Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004.  Estimating mutual information.  _Physical review E_ , 69(6):066138.
  * Krishna et al. (2018) Kalpesh Krishna, Shubham Toshniwal, and Karen Livescu. 2018.  Hierarchical multitask learning for ctc-based speech recognition.  _arXiv preprint arXiv:1807.06234_.
  * Latif et al. (2020) Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Julien Epps, and Bjórn Wolfgang Schuller. 2020.  Multi-task semi-supervised adversarial autoencoding for speech emotion recognition.  _IEEE Transactions on Affective Computing_.
  * LeCun et al. (1995) Yann LeCun, Yoshua Bengio, et al. 1995.  Convolutional networks for images, speech, and time series.  _The handbook of brain theory and neural networks_ , 3361(10):1995.
  * Lee et al. (2016) Giwoong Lee, Eunho Yang, and Sung Hwang. 2016.  Asymmetric multi-task learning based on task relatedness and loss.  In _International Conference on Machine Learning_ , pages 230–238.
  * Lee et al. (2018) Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. 2018.  Deep asymmetric multi-task feature learning.  In _International Conference on Machine Learning_ , pages 2956–2964. PMLR.
  * Li et al. (2019) Yuanchao Li, Tianyu Zhao, and Tatsuya Kawahara. 2019.  Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning.  In _Interspeech_ , pages 2803–2807.
  * Lipton et al. (2015) Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. 2015.  Learning to diagnose with lstm recurrent neural networks.  _arXiv preprint arXiv:1511.03677_.
  * Liu et al. (2019) Shengchao Liu, Yingyu Liang, and Anthony Gitter. 2019.  Loss-balanced task weighting to reduce negative transfer in multi-task learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 9977–9978.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , pages 21–37. Springer.
  * Majumder et al. (2019) Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019.  Dialoguernn: An attentive rnn for emotion detection in conversations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6818–6825.
  * McKeown et al. (2011) Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2011\.  The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent.  _IEEE transactions on affective computing_ , 3(1):5–17.
  * Menegola et al. (2017) Afonso Menegola, Michel Fornaciali, Ramon Pires, Flávia Vasques Bittencourt, Sandra Avila, and Eduardo Valle. 2017.  Knowledge transfer for melanoma screening with deep learning.  In _2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)_ , pages 297–300. IEEE.
  * Mittal et al. (2020) Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.  In _AAAI_ , pages 1359–1367.
  * Mordan et al. (2018) Taylor Mordan, Nicolas Thome, Gilles Henaff, and Matthieu Cord. 2018.  Revisiting multi-task learning with rock: a deep residual auxiliary block for visual detection.  In _Advances in Neural Information Processing Systems_ , pages 1310–1322.
  * Naim et al. (2015) Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015\.  Automated prediction and analysis of job interview performance: The role of what you say and how you say it.  In _2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG)_ , volume 1, pages 1–6. IEEE.
  * Nakkiran et al. (2019) Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2019.  Deep double descent: Where bigger models and more data hurt.  _arXiv preprint arXiv:1912.02292_.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Poria et al. (2019) Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019.  Emotion recognition in conversation: Research challenges, datasets, and recent advances.  _IEEE Access_ , 7:100943–100953.
  * Romero et al. (2014) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014.  Fitnets: Hints for thin deep nets.  _arXiv preprint arXiv:1412.6550_.
  * Ross (2014) Brian C Ross. 2014.  Mutual information between discrete and continuous data sets.  _PloS one_ , 9(2):e87357.
  * Sadoughi and Busso (2018) Najmeh Sadoughi and Carlos Busso. 2018.  Expressive speech-driven lip movements with multitask learning.  In _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_, pages 409–415. IEEE.
  * Sener and Koltun (2018) Ozan Sener and Vladlen Koltun. 2018.  Multi-task learning as multi-objective optimization.  In _Advances in Neural Information Processing Systems_ , pages 527–538.
  * Shen et al. (2020) Wei Shen, Xiaonan He, Chuheng Zhang, Qiang Ni, Wanchun Dou, and Yan Wang. 2020.  Auxiliary-task based deep reinforcement learning for participant selection problem in mobile crowdsourcing.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 1355–1364.
  * Simm et al. (2014) Jaak Simm, Ildefons Magrans de Abril, and Masashi Sugiyama. 2014.  Tree-based ensemble multi-task learning method for classification and regression.  _IEICE TRANSACTIONS on Information and Systems_ , 97(6):1677–1681.
  * Søgaard and Goldberg (2016) Anders Søgaard and Yoav Goldberg. 2016.  Deep multi-task learning with low level tasks supervised at lower layers.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 231–235.
  * Tanveer et al. (2015) M Iftekhar Tanveer, Emy Lin, and Mohammed Hoque. 2015.  Rhema: A real-time in-situ intelligent interface to help people with public speaking.  In _Proceedings of the 20th International Conference on Intelligent User Interfaces_ , pages 286–295.
  * Tao and Busso (2020) Fei Tao and Carlos Busso. 2020.  End-to-end audiovisual speech recognition system with multitask learning.  _IEEE Transactions on Multimedia_.
  * Torrey and Shavlik (2010) Lisa Torrey and Jude Shavlik. 2010.  Transfer learning.  In _Handbook of research on machine learning applications and trends: algorithms, methods, and techniques_ , pages 242–264. IGI global.
  * Trinh et al. (2018) Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le. 2018.  Learning longer-term dependencies in rnns with auxiliary losses.  _arXiv preprint arXiv:1803.00144_.
  * Vokaturi (2019) Vokaturi. 2019.  Vokaturi Overview.
  * Wu et al. (2020) Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020.  Understanding and improving information transfer in multi-task learning.  _arXiv preprint arXiv:2005.00944_.
  * Xia and Liu (2015) Rui Xia and Yang Liu. 2015.  A multi-task learning framework for emotion recognition using 2d continuous space.  _IEEE Transactions on affective computing_ , 8(1):3–14.
  * Yang et al. (2019) Jianliang Yang, Yuenan Liu, Minghui Qian, Chenghua Guan, and Xiangfei Yuan. 2019\.  Information extraction from electronic medical records using multitask recurrent neural network with contextual word embedding.  _Applied Sciences_ , 9(18):3658.
  * Yang et al. (2016a) Le Yang, Dongmei Jiang, Lang He, Ercheng Pei, Meshia Cédric Oveneke, and Hichem Sahli. 2016a.  Decision tree based depression classification from audio video and language information.  In _Proceedings of the 6th international workshop on audio/visual emotion challenge_ , pages 89–96.
  * Yang et al. (2018) Min Yang, Wei Zhao, Wei Xu, Yabing Feng, Zhou Zhao, Xiaojun Chen, and Kai Lei. 2018\.  Multitask learning for cross-domain image captioning.  _IEEE Transactions on Multimedia_ , 21(4):1047–1061.
  * Yang et al. (2016b) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016b.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1480–1489.
  * Yoo et al. (2018) ByungIn Yoo, Youngjun Kwak, Youngsung Kim, Changkyu Choi, and Junmo Kim. 2018.  Deep facial age estimation using conditional multitask learning with weak label expansion.  _IEEE Signal Processing Letters_ , 25(6):808–812.
  * Yousif et al. (2018) Abdallah Yousif, Zhendong Niu, and Ally S Nyamawe. 2018.  Citation classification using multitask convolutional neural network model.  In _International Conference on Knowledge Science, Engineering and Management_ , pages 232–243. Springer.
  * Yu and Jiang (2016) Jianfei Yu and Jing Jiang. 2016.  Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification.  In _Association for Computational Linguistics_.
  * Zadeh et al. (2018a) Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a.  Memory fusion network for multi-view sequential learning.  _arXiv preprint arXiv:1802.00927_.
  * Zadeh et al. (2018b) Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018b.  Multi-attention recurrent network for human communication comprehension.  In _Proceedings of the… AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence_ , volume 2018, page 5642\. NIH Public Access.
  * Zalmout and Habash (2019) Nasser Zalmout and Nizar Habash. 2019.  Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling.  _arXiv preprint arXiv:1910.12702_.
  * Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus. 2014.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , pages 818–833. Springer.
  * Zhang et al. (2016) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016\.  Understanding deep learning requires rethinking generalization.  _arXiv preprint arXiv:1611.03530_.
  * Zhu et al. (2020) Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020.  Vision-language navigation with self-supervised auxiliary reasoning tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10012–10022.

##  Appendix A Appendix

###  A.1 Dataset partitions

We detail the dataset partition in Table 5 for aid reproducibility.

Table 5: Dataset partitions IEMOCAP | SEMAINE
---|---
Partition |  Session |  Count |  Partition |  Session |  Count
Train |  Ses01F |  861 |  Train |  2008.12.05.16.03.15 |  424
Train |  Ses01M |  958 |  Train |  2009.01.30.12.00.35 |  404
Train |  Ses02F |  889 |  Train |  2009.02.12.10.49.45 |  686
Train |  Ses02M |  922 |  Train |  2009.05.15.15.04.29 |  436
Train |  Ses03F |  958 |  Train |  2009.05.29.14.30.05 |  760
Train |  Ses03M |  1178 |  Train |  2009.05.22.15.17.45 |  668
Dev |  Ses04F |  1105 |  Train |  2009.05.25.11.23.09 |  928
Dev |  Ses04M |  998 |  Train |  2009.05.26.10.19.53 |  790
Test |  Ses05F |  1128 |  Train |  2009.06.05.10.14.28 |  732
Test |  Ses05M |  1042 |  Train |  2009.06.15.12.13.06 |  958
|  |  |  Train |  2009.06.19.14.01.24 |  586
|  |  |  Train |  2009.10.27.16.17.38 |  440
|  |  |  Dev |  2008.12.19.11.03.11 |  188
|  |  |  Dev |  2009.01.06.14.53.49 |  472
|  |  |  Dev |  2009.05.12.15.02.01 |  448
|  |  |  Dev |  2009.05.08.11.28.48 |  752
|  |  |  Dev |  2009.06.26.14.38.17 |  404
|  |  |  Test |  2008.12.14.14.47.07 |  372
|  |  |  Test |  2009.01.06.12.41.42 |  264
|  |  |  Test |  2009.01.28.15.35.20 |  364
|  |  |  Test |  2009.06.26.14.38.17 |  440
|  |  |  Test |  2009.06.26.14.09.45 |  448

###  A.2 Scaling the auxiliary targets

We detail the operations in scaling the percentile scores that range [0,1] to
various primary tasks. For IEMOCAP Primary Tasks that are regression problems,
we multiply the percentile score by 4 and add 1 to obtain the range [1,5]. For
IEMOCAP classification task, we leave the auxiliary targets in the range of
[0,1]. As for SEMAINE tasks, which are all regression problems, we multiply
the percentile score by 2 and minus 1 to obtain the range [-1,1].

###  A.3 Hyperparameters tuning process

Glove word embeddings (300-dimensions) are used to represent the words
(Pennington et al., 2014). Hyper-parameters tuning is crucial because
different combinations of primary and auxiliary tasks require different sets
of hyperparameters. For hyperparameters tuning, we used random search
(Bergstra and Bengio, 2012) with thirty trials. We tuned the learning rate,
batch size, L2 regularization, the number of GRUs assigned to the primary and
auxiliary branches, the auxiliary weights assignment, the content-context
size, and lastly the GRU dropout and recurrent dropout (as detailed in Table
6).

Training is done on a RTX2070 or a V100 GPU, for up to 350 epochs. Early
stopping is possible via the median-stopping rule (Golovin et al., 2017) after
the fifth epoch and after every two epochs (i.e., at epoch number 5, 7, 9,
$...$, 349). Table 7 details the hyperparameters of models that performed the
best on the development set.

Table 6: Range of Hyperparameters tuned. U: Uniform sampling, LU: Uniform sampling on the logarithmic scale. Name |  Min. |  Max |  Sampling
---|---|---|---
Learning rate |  $2^{-10}$ |  $2^{-5}$ |  LU
Batch-Size |  32 |  256 |  U
Pri GRU | 1, 64, 128, 192, 256 |  U
Aux GRU | 257 - (minus) Pri GRU
Loss-weights Assignment | Random, Linear-Normalized, Softmax-Normalized |  U
L2 Regularization |  0.0 |  0.50 |  LU
Content-Size |  1 |  30 |  U
GRU dropout |  0.01 |  0.50 |  U
Recurrent dropout |  0.01 |  0.50 |  U
Table 7: Best Hyperparameters settings for development set. R: Random, L: Linear-Normalized, S: Softmax-Normalized. | IEMOCAP | SEMAINE
---|---|---
|  Classif.
MA(4) |  Val.
MAE |  Act.
MAE |  Dom.
MAE |  Act.
MAE |  Int.
MAE |  Pow.
MAE |  Val.
MAE
Dev. set |  0.714 |  0.482 |  0.492 |  0.570 |  0.118 |  0.164 |  0.135 |  0.138
Test set |  0.747 |  0.497 |  0.499 |  0.587 |  0.184 |  0.215 |  0.164 |  0.169
Learning rate |  6.21 e-03 |  2.21 e-02 |  1.44 e-02 |  3.13 e-02 |  2.43 e-02 |  1.58 e-02 |  3.04 e-03 |  2.94 e-02
Batch size |  43 |  43 |  62 |  77 |  41 |  43 |  33 |  62
Pri GRU |  256 |  256 |  256 |  256 |  256 |  256 |  192 |  256
Aux GRU |  1 |  1 |  1 |  1 |  1 |  1 |  65 |  1
L2 regularization |  2.25 e-05 |  0 |  1.43 e-05 |  2.41 e-04 |  0 |  0 |  0 |  0
Content-Size |  18 |  4 |  3 |  3 |  17 |  21 |  19 |  14
GRU dropout |  0.27 |  0.49 |  0.33 |  0.30 |  0.06 |  0.17 |  0.05 |  0.07
Recurrent dropout |  0.04 |  0.03 |  0.02 |  0.32 |  0.25 |  0.28 |  0.08 |  0.26
Epoch No. |  232 |  161 |  74 |  14 |  48 |  116 |  110 |  110
Aux. weights assignment |  S |  R |  L |  L |  L |  R |  S |  R
Main loss |  0.840 |  0.560 |  0.810 |  0.610 |  0.870 |  0.940 |  0.950 |  0.920
AU05 loss |  0.009 |  0.013 |  0.014 |  0.000 |  0.000 |  0.002 |  0.002 |  0.008
AU17 loss |  0.009 |  0.027 |  0.009 |  0.003 |  0.002 |  0.004 |  0.002 |  0.003
AU20 loss |  0.009 |  0.028 |  0.0 |  0.001 |  0.000 |  0.007 |  0.002 |  0.006
AU25 loss |  0.009 |  0.031 |  0.006 |  0.002 |  0.015 |  0.008 |  0.002 |  0.009
Happy tone loss |  0.010 |  0.023 |  0.038 |  0.006 |  0.021 |  0.013 |  0.002 |  0.003
Sad tone loss |  0.010 |  0.028 |  0.053 |  0.008 |  0.037 |  0.010 |  0.002 |  0.003
Angry tone loss |  0.010 |  0.016 |  0.034 |  0.006 |  0.055 |  0.011 |  0.002 |  0.000
Fear tone loss |  0.010 |  0.043 |  0.036 |  0.004 |  0.000 |  0.004 |  0.002 |  0.003
$Y_{t-1}$ loss |  0.010 |  0.030 |  0.0 |  0.040 |  0.000 |  0.000 |  0.005 |  0.006
$Y_{t-2}$ loss |  0.011 |  0.022 |  0.0 |  0.049 |  0.000 |  0.000 |  0.004 |  0.004
$Y_{t-3}$ loss |  0.010 |  0.006 |  0.0 |  0.046 |  0.000 |  0.000 |  0.003 |  0.006
$Y_{t-4}$ loss |  0.010 |  0.040 |  0.0 |  0.044 |  0.000 |  0.000 |  0.003 |  0.007
$Y_{t+1}$ loss |  0.010 |  0.042 |  0.0 |  0.042 |  0.000 |  0.000 |  0.005 |  0.005
$Y_{t+2}$ loss |  0.011 |  0.0 |  0.0 |  0.052 |  0.000 |  0.000 |  0.004 |  0.006
$Y_{t+3}$ loss |  0.010 |  0.050 |  0.0 |  0.043 |  0.000 |  0.000 |  0.003 |  0.008
$Y_{t+4}$ loss |  0.010 |  0.038 |  0.0 |  0.044 |  0.000 |  0.000 |  0.003 |  0.003

###  A.4 Details of computing the bootstrap confidence interval

Baseline models for each hypothesis are detailed in section 2. All non-
baseline models are referred to as challenger models. We created 1000
bootstrap samples of the test set performance by (1) resampling the
development set performance, then (2) selecting the set of hyperparameters
that resulted in the best development set performance, and (3) looking up the
test set performance given the set of best-performing hyperparameters for the
development set. To judge whether the challenger outperforms the baseline, we
computed the 95 percent confidence interval by (1) performing element-wise
subtraction between the resampled test set performance of the baseline against
the challenger, (2) removing the top and bottom 2.5 percent from the
differences, and (3) observing whether the remaining 95 percent confidence
interval includes zero. If it does not include zero, then the difference is
statistically significant.

###  A.5 Visualization from HAN-ROCK

We demonstrate how the HAN-ROCK model could be used to support humans analyze
conversations using only text-based inputs. We visualized the attention
weights from two models, (1) MTL refers to the classification model with
auxiliary supervisors, whilst (2) STL refers to the same model architecture,
but its auxiliary supervisors’ loss weights are set to zero. In principle, the
MTL model should exhibit attention weights that are less likely to overfit
because the weights are tempered by auxiliary supervisors. We observe that
across the two models, both use the historical talkturns more so than the
current talkturn; secondly, both assign high attention to the second word of
the talkturns, which is interesting because the second word is where the
multimodal annotations are inserted.

Figure 5: Conversation analysis example. Both models predicted the class label
(anger) correctly. The left-most column denotes the talkturn context – $i$
refers to the current talkturn where the target emotion class is predicted.
The square boxes indicate the level of attention (N: None, L: Low, M: Medium,
H: High) assigned to the talkturn. Within the talkturn, we also enlarge and
darken the font color ti visualize higher attention weights.

As explained in Section 3.2, there are three levels of attention over the
internal representations, word ($\alpha_{it}$), talkturn ($\alpha_{abi}$), and
task ($\alpha_{c}$). To compute the overall word and talkturn attention, we
compute the weighted average of $\alpha_{it}$ and $\alpha_{abi}$ using the
$\alpha_{c}$ (task attention) as weights. Once we have the overall word and
talkturn attention, we standardize the weights by computing the z-score.
Depending on the z-score, we bucket the attention in none (z $<$ 0), low (0
$<$ z $<$ 1), medium (1 $<$ z $<$ 2), or high (2 $<$ z). We plan to validate
the efficacy of the attention weights with human users in future research.