# Transfer-based adaptive tree for multimodal sentiment analysis based on user
latent aspects

Sana Rahmani1, Saeid Hosseini2∗, Raziyeh Zall1∗, Mohammad Reza Kangavari1,
Sara Kamran1, Wen Hua3 1 S. Rahmani, R. Zall, M. Kangavari, S. Kamran are with
School of computer engineering, Iran University of Science and Technology,
Iran.
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and<EMAIL_ADDRESS>2 S. Hosseini is with the
Faculty of Computing and Information Technology, Sohar University, Sohar,
Oman. Email<EMAIL_ADDRESS>3 W. Hua is with the School of Information
Technology and Electrical Engineering, University of Queensland, Australia.
E-mail<EMAIL_ADDRESS>∗ S. Hosseini and R. Zall contributed equally

###### Abstract

Multimodal sentiment analysis benefits various applications such as human-
computer interaction and recommendation systems. It aims to infer the users’
bipolar ideas using visual, textual, and acoustic signals. Although
researchers affirm the association between cognitive cues and emotional
manifestations, most of the current multimodal approaches in sentiment
analysis disregard user-specific aspects. To tackle this issue, we devise a
novel method to perform multimodal sentiment prediction using cognitive cues,
such as personality. Our framework constructs an adaptive tree by
hierarchically dividing users and trains the LSTM-based submodels, utilizing
an attention-based fusion to transfer cognitive-oriented knowledge within the
tree. Subsequently, the framework consumes the conclusive agglomerative
knowledge from the adaptive tree to predict final sentiments. We also devise a
dynamic dropout method to facilitate data sharing between neighboring nodes,
reducing data sparsity. The empirical results on real-world datasets determine
that our proposed model for sentiment prediction can surpass trending rivals.
Moreover, compared to other ensemble approaches, the proposed transfer-based
algorithm can better utilize the latent cognitive cues and foster the
prediction outcomes. Based on the given extrinsic and intrinsic analysis
results, we note that compared to other theoretical-based techniques, the
proposed hierarchical clustering approach can better group the users within
the adaptive tree.

###### Index Terms:

Transfer-based adaptive tree, Cognitive cues, Dynamic dropout, Hierarchical
training, Multimodal sentiment analysis, Attention-based fusion

##  1 Introduction

As a branch of affective computing, sentiment analysis can classify user-
generated content as positive or negative [1]. This research area gains
growing interest in various domains: (i) In Human-Computer Interaction, to
expand the interchange between the agents and individuals [2]. (ii) In
political forecasting, to study political sentiments [3]. (iii) In
recommendation systems, to analyze the reviews of consumers [4]. (IV) In
market forecasting, to promote financial prediction outcome [5]. With the
rapid rise of social media, individuals mostly tend to convey their opinions
in videos, creating the opportunity to analyze multiple input channels known
as Acoustic, Visual, and Textual modals. Such multimodal input carries surplus
information on the individual’s latent cues, including personality,
motivations, and other ignored parameters in current sentiment analysis
methods. Given a video clip $v_{i}$ for user $l_{k}$, we aim to employ a
cognitive-based ensemble approach to improve sentiment analysis for $v_{i}$
through considering the cognitive cues of the user $l_{k}$, denoted by
$C_{k}$. However, certain challenges abound:
Challenge 1 (Missing Cognitive Annotations)
The sentiment analysis task needs the annotations to represent the cognitive
cues of the user in the input video. However, the majority of available
datasets on multimodal affective interpretation, including MOSI [6], MOUD [7],
and IEMOCAP [8], are not enriched by cognitive annotations. Aiming to augment
the datasets using cognitive cues, one can leverage the self and full-
supervised methods. Additionally, utilizing the pre-trained models may seem
partially inaccurate because they include the perturbations from the initial
trainee dataset. To address this issue, one may opt for devising a framework
to reduce the cross dataset effects or choose a labeled dataset with the
minimum track of skews.
Challenge 2 (Assorted Categories Affected by Various Cues)
The resultant categories constructed by the cognitive-oriented subset-based
ensembles can have substantial similarities or overlook information-
theoretical parameters including, entropy. Moreover, the unbiased categories
must adapt to the deep learning approaches to avoid starving or overfitting.
In retrospect, choosing the correct values for adjustment parameters in
categorization can be challenging. Dimension, cut-point, and the number of
clusters or hierarchical levels are samples of such parameters.

(a) Data categorization on MOSI[6]

(b) modals accuracy comparison

Figure 1: Base Models Accuracy Comparison

Observation To demonstrate how cognitive cues can partition a set of data
instances, we set up an observation that performs clustering on the videos in
the MOSI affective dataset. As depicted in Fig. 1(a), we utilize the Euclidean
distance to classify the videos into four groups. The data points and
individual colors respectively signify videos and pertinent clusters.
Furthermore, to investigate the effect of categorization on emotional
manifestation, Fig. 1(b) shows the accuracy heatmap to compare the modalities
of constructed clusters. The performance of the sentiment analytics on video
transcripts is higher than other input channels, reaching up to %87 on
accuracy. Moreover, the accuracy deviation in various clusters of the visual
channel implies that the level of cognition can directly affect the sentiment
understanding. Hence, besides the effectiveness of the categorization
algorithms, the availability of sufficient cognitive cues can foster the
sentiment analysis procedure.
Challenge 3 (Data Scarcity in Subset Categories)
Intuitively, we perform top-bottom hierarchical categorization on the adaptive
tree to better exploit the cognitive correlation between subsets. However,
dividing the data in each node into portions to form subsets can cause data
scarcity, where the data enclosed with some nodes may not suffice to train the
model, affecting the learning capability of the deep architecture. Relying
merely on the pruning procedure to control the expansion of the adaptive tree
based on the correlation criteria can negatively affect the performance of the
cognitive-based categorization. To address this issue, we urge that besides
the pruning supervision, the subsets must also be enriched to preserve the
original subset characteristics and avoid unintended influence on models.
Contributions. While most of the current works [9, 10, 11, 12] neglect the
indispensable role of the cognitive cues in multimodal sentiment analysis, our
proposed framework in this paper enhances analytical results by leveraging
user-specific latent characteristics. To the best of our knowledge, we reveal
the utopian spirit of the hidden parameters and devise the first approach
based on an adaptive tree that utilizes an attention-based fusion to
facilitate cognitive-oriented knowledge transfer within the tree, forming an
ensemble of submodels. In retrospect, where our previous works [13, 14, 15]
focus on short-text understanding, we propose an approach that not only takes
advantage of the textual analytics but also infers the cognitive cues from
acoustic and visual mediums to leverage the contrast between individuals and
effectively predict the sentiments. Our contributions are fourfold:

  * •

We develop a novel transfer-based adaptive tree to facilitate ensembles on
multimodal submodels.

  * •

We analyze multiple categorization strategies to classify individuals by
leveraging pertinent latent cues.

  * •

We design a dynamic dropout approach to regularize data of relevant classes
during hierarchical training.

  * •

We propose a unified cognitive-based framework for sentiment analysis on
multimodal datasets.

The rest of the paper is as follows: in Sec. 2, we study the literature; in
Sec. 3, we provide the problem and our framework; in Sec. 4 and 5 we explain
our model and experiments. Lastly, we conclude the paper in Sec. 6.

##  2 Related Work

As Table I briefs, the literature is threefold: multimodal sentiment analysis,
transfer learning, and leveraging individual characteristics.

###  2.1 Multimodal Sentiment Analysis

Sentiment analysis as a branch of affective computing aims to exploit the
polarity expressed by an individual in three possible categories of positive,
negative, or neutral [1]. Accordingly, numerous research works analyze the
sentiments from each of the modals (unimodal), including text [16], audio
[17], and video [18]. Moreover, multimodal affective analysis has adapted
various input modals [19], including acoustic-textual [20], acoustic-visual
[18, 21], and the tri-modal unified frameworks [9, 10]. The main challenge in
the multimodal analysis is the fusion method [1] that comprises two literature
domains of decision and feature level, facilitating a variety of modals
combinations. Weighted voting [22] and Bayes belief integration [23] are two
instances of decision-level fusion, fusing independent predictions of each
modal into a unified output. The feature-level approaches such as Tensor [9]
and attention-based [10] generate the modal-specific embeddings to feed into
another final analysis module. Nevertheless, various architectures adapt
models, including Multiple Kernel Learning (MKL) on Support Vector Machine
(SVM) classifiers [24] and Recurrent Neural Networks (RNN) classes of Gated-
Recurrent Unit (GRU)[11] and Long Short-Term Memory (LSTM) [21, 10], to target
the context-dependent analysis of inputs. Ensemble Learning approaches utilize
various modules to improve the performance by combining the results [25, 26].
Some train the same model on distinct datasets, as [27] utilizes various SVM
classifiers trained on different dataset categories distinguished by emotion
labels. Others train divergent models on the same dataset, as [12] applies
SVMs together with logistic regression, decision tree, random forest, and
clustering modules on the same corpus. In retrospect, we propose an ensemble
on multimodal LSTM models equipped with an attention-based fusion trained on
data divisions based on the composer’s latent cues.

###  2.2 Transfer Learning

Transfer Learning (TL) approaches benefit from pre-trained models to improve
learning by transferring knowledge between domains or extracting new
representations of input data as feature sets [28]. TL module adopts into
unimodal analysis approaches of textual, visual, and acoustic. For textual,
[29] utilizes TL to perform textual analysis on emojis, and [30] applies
transformer encoding to detect emotions in conversations, passing multi-turn
conversation parameters to an emotion classifier. For visual, [31] applies TL
on large-scale image classification to analyze visual cues in affective
analysis. Finally, [32] applies TL in sparse auto-encoding to accomplish
acoustic emotion prediction.
Furthermore, multimodal affective analysis adopts TL in various applications:
For instance, [33] detects audio-visual emotions by obtaining CNN features of
facial expressions through the TL module. In retrospect, aiming to reduce the
effect of catastrophic forgetting in TL, the multi-task learner in [34] adapts
into multimodal affective analysis by learning progressive networks on
paralinguistic tasks, recognizing emotions, speakers, and gender. Similarly,
[35] forms the multi-task learner by appending a classifier to each emotion,
where the work extends to [36] by integrating topical classification.
Accordingly, we perform multimodal analysis with a novel ensemble through a
hierarchical knowledge transfer in an adaptive tree based on users’ latent
characteristics.

###  2.3 Individual Characteristics

In this section, we review the personal characteristics that can influence
users’ expressions. We get inspired by cognitive studies that affirm how
significantly personality factors can affect human behavior [37]. Accordingly,
we survey the literature assuming five personality traits as the primary
latent cues of humans. The research on uni-modal personality analysis
comprises trilateral input modals of acoustic [38], textual [39], and visual
[40]. Likewise, there is comprehensive research to investigate multimodal
personalities, [41] leverages a deep residual network that can predict
personality on the first impression for audio-visual input.
Similarly, Batrinca et al. [42] apply audio-visual predictions on a Human-
Computer Interaction (HCI) task, and [43] collectively uses quadrilateral
modals of audio, text, face, and video background. Other works [44][45]
examine personality in the context of synthetic applications, simulating the
crowd through constructing artificial agents designated by personality weights
that replicate emotion contagion in incidents. From another perspective, [46,
47, 48] exploit applications on personality perception. While [46] devises a
user-centric recommendation system where the personality factor determines the
similarity weights between users and items, [47] leverages personality scores
from social networks to improve the user interaction with information systems.
Comparably, [48] utilizes the user’s personality to improve the textual
sentiment evaluation in short texts. Nevertheless, the majority of practices
in affective evaluations that consider user characteristics are solely
unimodal. In response, we utilize user-specific characteristics in a
multimodal analytical system to improve sentiment prediction results.

TABLE I: Literature Review Category | Approaches | References
---|---|---
Multimodal Sentiment Analysis | Traditional Learning |  [27][12]
Deep Learning |  [21][9][10][24]
Transfer Learning | Knowledge Transfer |  [30][34][35][36]
Feature Extraction |  [29][31][32][33]
Individual Characteristics | Prediction |  [38][39][40][41][42][43]
Perception |  [44][45][46][47][48]
|  |

##  3 Problem Statement

In this section, we elucidate primary definitions, the problem statements, and
the proposed framework.

###  3.1 Preliminary Concepts

######  Definition 1.

(Sentiment Value) The Sentiment value denoted by $s\in\\{0,1\\}$ is a binary
value, representing a positive or negative opinion regarding an entity.

######  Definition 2.

(Video) A video denoted by $v_{i}\in\mathbb{V}$ reflects the participating
user, $l_{k}$ reactions and is associated with a sentiment value, $v_{i}.s$,
start time, $v_{i}.t_{s}$, end time, $v_{i}.t_{f}$, and tri-modal data,
including, Acoustics ($v^{a}_{i}$), Visuals ($v^{v}_{i}$), and Textual
($v^{t}_{i}$) contents.

######  Definition 3.

(Utterance) Utterance, $u_{j,i}\in\mathbb{U}$, represents the partition $j$ of
video $v_{i}$, bounded by pauses. Each utterance $u_{j,i}$ contains a
sentiment value $u_{j,i}.s$, start time $u_{j,i}.t_{s}$, and the end time
$u_{j,i}.t_{f}$, such that:

  * •

The bounding rule applies as
$[u_{j,i}.t_{s},u_{j,i}.t_{f}]\subseteq[v_{i}.t_{s},v_{i}.t_{f}]$.

  * •

Utterances shall not overlap with each other in a video, $\forall m,n$, if
$m\neq n$ then
$[u_{m,i}.t_{s},u_{m,i}.t_{f}]\cap[u_{n,i}.t_{s},u_{n,i}.t_{f}]=\O$.

Furthermore, any $u_{j,i}$ is associated with Acoustics ($u^{a}_{j,i}$),
Visuals ($u^{v}_{j,i}$), and Textual contents ($u^{t}_{j,i}$). Also, each
video $v_{i}$ is composed of a set of utterances, $v_{i}=\\{u_{j,i}|1\leq
j\leq\digamma_{v}^{len}(v_{i})\\}$, where
$\digamma_{v}^{len}:\mathbb{V}\rightarrow\mathbb{Z}$ is the function
retrieving the number of utterances in the given video and $v_{i}.s$
represents the majority aggregation on sentiment values of the pertinent
utterances.

######  Definition 4.

(User) Each video $v_{i}$ is associated with a user $l_{k}\in\mathbb{L}$ that
expresses a sentiment value for every contained utterance $u_{j,i}$. Given one
or more videos composed by user $l_{k}$, we can utilize the function
$\digamma_{l}^{v}:\mathbb{L}\rightarrow\mathbb{V}$ to retrieve pertinent
videos.

######  Definition 5.

(Cognitive Cue) Each cognitive cue, denoted by $c^{y}\in\mathbb{R}$, can
constitute any latent cue, like personality, that can distinguish individuals.

###  3.2 Problem Definition

######  Problem 1.

(multimodal sentiment analysis) Given a video $v_{i}$ of the user $l_{k}$, our
goal is to improve the estimation of sentiments through utilizing the latent
cues of $l_{k}$.

######  Problem 2.

(enhancing intra-ensemble awareness) Given a set of submodels $M$, where each
inference submodel $m_{i}\in M$ partially contributes to the prediction
process, our goal is to enrich the knowledge submission between submodels in
the ensemble.

######  Problem 3.

(reducing data scarcity) Given a set of videos $V$, our goal is to enrich any
subset $\tilde{V}\subseteq V$ with data insufficiency.

###  3.3 Framework Overview

Fig. 2 illustrates our proposed unified framework that employs cognitive cues
to improve the sentiment prediction. Through the offline phase, we firstly
feed tri-modal visual, textual, and acoustic data into feature extraction
modules to construct an aligned, normalized set for each of the Modals. For
data augmentation, we develop an inference algorithm that extracts latent
aspects of individuals. To follow, we devise a novel adaptive tree with each
node comprising a user set, pertinent collective knowledge core, and a
transfer-based submodel. On the one hand, we employ clustering and
theoretical-based approaches in a top-bottom procedure to perform hierarchical
categorization of users based on their cognitive cues, and on the other hand,
we train the transfer-based submodel and form the knowledge core in parallel
through a bottom-up process. Accordingly, the LSTM submodel consumes the
attention-based fusion that comprises node-specific user data and the
knowledge core, aggregated by the submodels of the child nodes. Moreover, a
nontrivial module adopts users’ data from similar neighbors to tackle data
sparsity in training submodels within immature nodes through a dynamic dropout
approach. Finally, the prediction module utilizes the adaptive tree to
evaluate the sentiments based on user-specific latent cues.

Figure 2: Framework

In the online phase, we pass each input video into feature extraction and
cognitive assessment modules adopted from the offline section. Given the
cognitive cues of the video composer, we then find a top-bottom route that
leads to the most relevant leaf node. Conversely, we take the route to
transfer the knowledge by executing the submodels in a bottom-up manner.
Finally, given the collective knowledge of the tree, the prediction module
performs the final cognitive-based sentiment prediction of the input data.

##  4 Methodology

###  4.1 Offline Phase

####  4.1.1 Extracting Feature Vectors

As elucidated in Sec. 3.1, every input utterance, $u_{j,i}$, is associated
with acoustic ($u^{a}_{j,i}$), visual ($u^{v}_{j,i}$), and textual
($u^{t}_{j,i}$) modals. We designate a modal-specific embedding to extract the
feature vectors, denoted by $\vec{x}_{j,i}$, for each modal
($\vec{x}_{j,i}^{m}$, $m\in(a,v,t)$). The steps after feature extraction
include normalization and alignment, followed by an assignment process which
links features to respective utterance. Accordingly, we explain the feature
extraction modules as follows:
Acoustic Feature Extraction.  Given the significance of acoustic expression in
affective computation, we need a legitimate feature representation process.
Hence, for each utterance, $u_{j,i}$ with respective prevalent textual and
acoustic pre-alignments as $u^{t}_{j,i}$ and $u^{a}_{j,i}$, we partition the
monophonic audio signal into textually coherent partitions. Subsequently, we
adopt the acoustic means, including COVAREP [49] and OpenSmile [50], to
exploit the audio feature vector for each partition, denoted by
$\vec{x}^{a}_{j,i}$, including the low-level and diverse granular descriptors
(LLD). To continue, we devise an acoustic-LSTM module to generate the embedded
representation for each input segment, $\vec{x}^{a}_{j,i}$.
Visual Feature Extraction.  To infer the facial cues, we fragment the visual
contents $u^{v}_{j,i}$ into the frames, isolate the metaphorical facial region
from the background for each frame, and exclude negative frames where the
human face is unobserved. To adjust the visual feature vector
$\vec{x}^{v}_{j,i}$, we successively utilize Openface2 [51] to detect facial
landmarks, estimate head pose and gaze direction, and recognize facial action
units. In contrast, we apply the Facet algorithm [52] to analyze the feature
value distribution on various granular frame representations. The proposed
Visual-LSTM generates an embedded representation for each vector
$\vec{x}^{v}_{j,i}$.
Textual Feature Extraction. The textual contents, enclosed as transcripts to
utterance $u^{t}_{j,i}$, are not infallible but tractable to exploit the
textual features. A naïve solution would be to notoriously train the exact
textual contents [53][54], yielding incorrect correlation weights [13][14].
Alternatively, we utilize semantic embedding models [55] to obtain word
vectors. We then utilize the uni-dimensional convolution networks besides the
order-sensitive LSTM. Here, each convnet serves the neural network as a
preprocessing tool. In other words, the convnet can transform the long input
sequence into shorter sequences of higher-level features $\vec{x}^{t}_{j,i}$.
Finally, the Textual-LSTM can feed the feature vector $\vec{x}^{t}_{j,i}$ to
generate the associated embedded representations.

####  4.1.2 Cognitive Inference

As a complement to feature extraction, it is through mapping to cognitive
space that we can hierarchically categorize the users. Here, the cognitive
space ($\Omega,\digamma_{\Omega}^{d}$) adapts to a pseudo-metric
$\mathbb{R}^{|C|}$ space [56] with $\Omega$ as the set of points, $C$ as the
set of cognitive cues, and $\digamma_{\Omega}^{d}$ as the pseudo-metric
distance. While each cognitive cue $c^{y}$ represents a dimension in the
$\Omega$ space, the surrogate-point
$\varphi_{k}=(c^{1}_{k},\dots,c^{|C|}_{k})$ delineates the user $l_{k}$ into
$\Omega$. In addition, we denote the set of surrogate-points as $\Phi$ and
clarify the connotations within $\Omega$ by defining the following functions:
Mapping ($\digamma_{\Omega}^{m}$: $\mathbb{L}\rightarrow\Omega$): the function
mapping the user $l_{k}$ to a surrogate-point $\varphi_{k}$ in $\Omega$ using
each cognitive cue $c^{y}_{k}$.
Distance ($\digamma_{\Omega}^{d}$:
$\Omega\times\Omega\rightarrow\mathbb{R}^{+}_{0}$): a pseudo-metric function
to measure the dissimilarity ratio between each pair of points in $\Omega$,
denoted by ($\varphi_{i}$,$\varphi_{j}$). Only those distance metric models
that can sufficiently satisfy the pseudo-metric axioms, as formulated in eq.
1, are eligible to estimate the similarity.

$\small\left\\{\begin{array}[]{l}\begin{multlined}Positive\;Definiteness:\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})\geq
0;\end{multlined}Positive\;Definiteness:\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})\geq
0;\vspace{1mm}\\\
\begin{multlined}Symmetry:\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})=\digamma_{\Omega}^{d}(\varphi_{j},\varphi_{i});\end{multlined}Symmetry:\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})=\digamma_{\Omega}^{d}(\varphi_{j},\varphi_{i});\vspace{1mm}\\\
\begin{multlined}Triangle\;Inequality:\vspace{-5mm}\\\
\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})\leq\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{k})+\digamma_{\Omega}^{d}(\varphi_{k},\varphi_{j});\end{multlined}Triangle\;Inequality:\vspace{-5mm}\\\
\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})\leq\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{k})+\digamma_{\Omega}^{d}(\varphi_{k},\varphi_{j});\end{array}\right.$
(1)

We adopt the effective vector-based Minkowski measure as the distance
$\digamma_{\Omega}^{d}$ (Eq.2) that is generalizable and capable of
accommodating an infinite number of cognitive cues. $\varphi_{i}$ and
$\varphi_{j}$ instantiate a pair of points in cognitive space $\Omega$, and
$r\in\mathbb{N}$ denotes the adjustment parameter.

$\small\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})=\displaystyle(\sum_{d=1}^{|C|}(|c_{i}^{d}-c_{j}^{d}|)^{r})^{\dfrac{1}{r}}\vspace{-2mm}$
(2)

Successively, we propose the mapping functionality $\digamma_{\Omega}^{m}$ by
an effective model [41] that consumes the multimodal users’ data to predict
multi-aspect character attributes. To continue, we propose an observation on
user mapping using three cues of agreeableness, consciousness, and
extroversion. As Fig. 3 depicts, we appoint each user in the dataset [6] to a
surrogate point in $\Omega$ using a three-dimensional weight. Subsequently, we
can utilize the K-means method for clustering, illustrated in various colors.

Figure 3: User distribution in a three-dimensional $\Omega$ on MOSI[6]

######  Lemma 1.

The cognitive space is a pseudo-metric space.

###### Proof.

By a contradiction, we assume $(\Omega,\digamma^{d}_{\Omega})$ is a metric
space. In response, the distance module $\digamma^{d}_{\Omega}$ must preserve
the metric axioms, including the non-zero criterion for the distance between
any pair of specific points (Eq. 3).

$\small\forall\varphi_{i},\varphi_{j}\in\Omega:\;\varphi_{i}\neq\varphi_{j}\Longleftrightarrow\digamma^{d}_{\Omega}(\varphi_{i},\varphi_{j})>0\vspace{-2mm}$
(3)

Let $l_{i}$ and $l_{j}$ be a pair of users that can have identical cognitive
cues. Inherently, due to the high cognitive correlation as formalized in Eq.
4, $\digamma_{\Omega}^{m}$ can map the pair to the surrogate-points of the
same coordination in $\Omega$, reflecting the zero distance between users.
Such an outcome can contradict the primary assumption, proving the
proposition.

$\small\begin{multlined}\exists l_{i},l_{j},l_{i}\neq
l_{j}\;and\;\underset{c_{y}\in
C}{\forall}(\digamma_{\Omega}^{m}(l_{i}).c_{y}=\digamma_{\Omega}^{m}(l_{j}).c_{y})\vspace{-2mm}\\\
\Longrightarrow\digamma_{\Omega}^{d}(\digamma_{\Omega}^{m}(l_{i}),\digamma_{\Omega}^{m}(l_{j}))=0\end{multlined}\exists
l_{i},l_{j},l_{i}\neq l_{j}\;and\;\underset{c_{y}\in
C}{\forall}(\digamma_{\Omega}^{m}(l_{i}).c_{y}=\digamma_{\Omega}^{m}(l_{j}).c_{y})\vspace{-2mm}\\\
\Longrightarrow\digamma_{\Omega}^{d}(\digamma_{\Omega}^{m}(l_{i}),\digamma_{\Omega}^{m}(l_{j}))=0\vspace{-5mm}$
(4)

∎

######  Lemma 2.

Minkowski measure satisfies pseudo-metric axioms.

$\small\left\\{\begin{array}[]{l}\digamma^{d}_{\Omega}(\varphi_{i},\varphi_{k})=||u+v||\\\
\digamma^{d}_{\Omega}(\varphi_{i},\varphi_{j})=||u||\\\
\digamma^{d}_{\Omega}(\varphi_{j},\varphi_{k})=||v||\end{array}\right.\vspace{-2mm}$
(5)

###### Proof.

With axioms in Eq. 1, the proof to preserve the Triangle Inequality can leave
out the naïve proofs about both trivial features of the Minkowski metric,
positive definiteness and the symmetry criterion. Accordingly, let
$\varphi_{i},\>\varphi_{j},\>\varphi_{k}\in\Omega$ be a triple surrogate-
points and define the vectors $u$ and $v$ as $u=\varphi_{i}-\varphi_{j}$ and
$v=\varphi_{j}-\varphi_{k}$, with the sum vector as
$u+v=\varphi_{i}-\varphi_{k}$. Hence, the vector norms in Eq. 5 equate to the
distance between corresponding points. Given $u.v$ as the dot product of the
pair of vectors $u$ and $v$, we can adopt the norm convention to affirm the
symmetry in Eq. 6 .

$\small||u+v||^{2}=||u||^{2}+||v||^{2}+2u.v\vspace{-2mm}$ (6)

Given the triangle axiom, we must prove Eq. 7 inequality:

$\small\digamma^{d}_{\Omega}(\varphi_{i},\varphi_{k})\leq\digamma^{d}_{\Omega}(\varphi_{i},\varphi_{j})+\digamma^{d}_{\Omega}(\varphi_{j},\varphi_{k})\vspace{-2mm}$
(7)

To commence, we can attain the sequential order in Eq. 8 by the criteria
verbalized in Eq. 5:

$\small\begin{multlined}||u+v||\leq||u||+||v||\overset{Square}{\Longrightarrow}\\\
||u+v||^{2}\leq||u||^{2}+||v||^{2}+2||u||\times||v||\end{multlined}||u+v||\leq||u||+||v||\overset{Square}{\Longrightarrow}\\\
||u+v||^{2}\leq||u||^{2}+||v||^{2}+2||u||\times||v||\vspace{-2mm}$ (8)

By replacing the Eq. 6 in Eq. 8, we can retrieve Eq. 9.

$\small\begin{multlined}||u||^{2}+||v||^{2}+2u.v\leq||u||^{2}+||v||^{2}+2||u||\times||v||\\\
\Longrightarrow
u.v\leq||u||\times||v||\end{multlined}||u||^{2}+||v||^{2}+2u.v\leq||u||^{2}+||v||^{2}+2||u||\times||v||\\\
\Longrightarrow u.v\leq||u||\times||v||\vspace{-2mm}$ (9)

Here the Eq. 9 results in Cauchy-Schwarz inequality[57]
($|u.v|\leq||u||\times||v||$) that logically proves the lemma. ∎

####  4.1.3 Adaptive Ensemble Tree Definition

The necessity to devise a novel tree structure to categorize users through
performing sharding on the $\Omega$ space sets off before determining the
sentiment prediction models. Aiming to carry out the user grouping procedure,
we rely on the verified association between the individuals’ latent cues and
affective manifestation [58].
Inspired by VP-Tree [59], which searches more efficiently compared to R*-tree
[60], we develop a non-trivial adaptive tree $T=(N,E)$ with $N$ and $E$ as the
set of nodes and edges. In response, the tree utilizes surrogate points to
divide $\Omega$ space based on the dimensions. Accordingly, the sharding
procedure hierarchically constructs a collection of subspaces from $\Omega$,
facilitating parallel processing in simultaneous training of multiple
subspaces.
The properties of each node $n$ can be threefold: (1) The Subspace $n^{\Phi}$
comprising a set of surrogate points created by sharding on the parent node,
(2) The Submodel $n^{m}$, a transfer-based LSTM model that consumes the videos
in $n^{\Phi}$, denoted by $\\{\digamma^{v}_{l}(l_{k})|l_{k}\in n^{\Phi}\\}$,
and (3) The Knowledge Core $n^{k}$ to gain an aggregated perception from the
child nodes. Furthermore, the adaptive tree must hold two functions:

  * •

Fragmentation ($\digamma_{T}^{f}:N\rightarrow\Omega$) divides the subspace
pertinent to the given node $n^{\Phi}$ into $k$ tractable clusters, adjusted
by parameter $\ddot{\rho}$ where the cluster set should attain specific
criteria denoted by $\gamma^{\Phi}$:

$\small\gamma^{\Phi}=\left\\{\begin{array}[]{l}\begin{multlined}\Phi_{i}\neq\emptyset,\;i\leq
k;\end{multlined}\Phi_{i}\neq\emptyset,\;i\leq k;\\\
\begin{multlined}\cup_{i=1}^{k}\Phi_{i}=n_{i}^{\Phi};\end{multlined}\cup_{i=1}^{k}\Phi_{i}=n_{i}^{\Phi};\\\
\begin{multlined}\Phi_{i}\cap\Phi_{j}=\varnothing,\;i,j\leq k\;\&\;i\neq
j;\end{multlined}\Phi_{i}\cap\Phi_{j}=\varnothing,\;i,j\leq k\;\&\;i\neq
j;\vspace{-1mm}\\\ \end{array}\right.\vspace{-1mm}$ (10)

Here $\Phi_{i}$ and $\Phi_{j}$ are the splits from fragmenting the node
subspace $n^{\Phi}$. Union ($\cup$) ensures the subspace $n^{\Phi}$ will be a
portmanteau of resultant splits and intersection ($\cap$) affirms the
exclusivity of the splits.

  * •

Disjunction ($\digamma_{T}^{d}:\Omega\rightarrow\mathbb{R}^{+}_{0}$) specifies
the relative distance between every pair of node subspaces in the same level
of the adaptive tree (Eq.11). For ($n_{i}$,$n_{j}$) as a pair, the disjunction
will be the cognitive distance $\digamma_{\Omega}^{d}$ between the medoids,
denoted by $\tilde{\varphi}$:

$\small\digamma_{T}^{d}(n_{i}^{\Phi},n_{j}^{\Phi})=\digamma_{\Omega}^{d}(n_{i}^{\Phi}.\tilde{\varphi},n_{j}^{\Phi}.\tilde{\varphi})\vspace{-2mm}$
(11)

####  4.1.4 Cognitive space categorization

In this step, we devise a modified top-down breadth-first traversing method to
construct the adaptive tree (Sec. 4.1.3) by iteratively partitioning the
$\Omega$ space. Given node $n_{i}$, we can terminate the node branching by
controlling the triple data attributes of cohesion, sparsity, and amplitude.
Given $H$ as the set of fragments on the node $n_{i}$, we define trilateral
termination criteria for each split $\Phi\;\in H$ as follows:
Criterion1. Subject to cohesiveness attribute for each split $\Phi$, Eq. 12
determines the unity measure $\digamma^{uni}$ to compute the cognitive
distance $\digamma_{\Omega}^{d}$ between each pair of points in $\Phi$.

$\small\digamma^{uni}(\Phi)=\left(\begin{array}[]{c}|\Phi|\\\
2\end{array}\right)^{-1}\underset{\varphi_{x},\varphi_{z}\in\Phi}{\sum}\digamma_{\Omega}^{d}(\varphi_{x},\varphi_{z})\vspace{-2mm}$
(12)

Eq. 13 obtains the branching tendency on the node subspace $n_{i}^{\Phi}$ for
each split by evaluating the quality of fragmentation using unity and
disjunction trade-offs. Hence, we collectively analyze the mean-variance (MV)
to compute the recuperation capacity $\digamma^{rcp}$ using the acquired
measures.

$\small\digamma^{rcp}(H)=\underset{\Phi\in
H}{MV}(\dfrac{(|H|-1)\times\digamma^{uni}(\Phi)}{\underset{\Phi_{j}\in
H,\Phi_{j}\neq\Phi}{\sum}\digamma_{T}^{d}(\Phi,\Phi_{j})})\vspace{-2mm}$ (13)

Where lower values of recuperation ensure better cohesion and separation,
higher values than $\theta_{p}$ can stop branching.
Criterion2. Besides the recuperation ratio, we also need to ensure the
training capability of independent splits. Hence, we propose an entropy-based
criterion that consumes the sentiment labels within split data to control the
bias. Correspondingly, Eq. 14 formalizes the set of utterances $U_{j}$
pertinent to users of a subspace $\Phi_{j}$.

$U_{j}=\\{u_{z,x}|u_{z,x}\subseteq
v_{x},v_{x}\in\digamma_{l}^{v}(l_{k}),l_{k}\in\Phi_{j}\\}\vspace{-2mm}$ (14)

Eq. 15 can estimate the portion of utterances $U_{j}$ convey the sentiment
$\tilde{s}$ in each split $\Phi_{j}$.

$\small\digamma^{pr}(\tilde{s},\Phi_{j})=\\{\dfrac{|u|}{|U_{j}|}|u\in
U_{j},u.s=\tilde{s}\\}\vspace{-2mm}$ (15)

Eq. 16 adopts Shannon entropy [61] to measure data impurity in the split
$\Phi_{j}$.

$\small\digamma^{imp}(\Phi_{j})=-\underset{\tilde{s}\in\\{0,1\\}}{\sum}\digamma^{pr}(\tilde{s},\Phi_{j})\log(\digamma^{pr}(\tilde{s},\Phi_{j}))\vspace{-2mm}$
(16)

We can utilize the impurity ratio to evaluate the quality of the set $H$ of
splits by ${\biguplus}_{\Phi_{j}\in H}\digamma^{imp}(\Phi_{j})$, where
$\biguplus$ denotes the weighted average. Accordingly, we can compare the
result with the threshold $\theta_{e}$ to terminate nodes with lower rates.
Criterion3. Relying on data amplitude, this restrain asserts under what verge
the training can infer meaningful information from each of the splits. Given
the set of pertinent utterances $U_{j}$ in Eq. 14 for a split $\Phi_{j}$, we
ensure the amplitude capacity of the splits using threshold $\theta_{a}$, a
prerequisite to terminate the cases with lower values than ${min}_{\Phi_{j}\in
H}\\{|U_{j}|\\}$.
Algorithm 1 initializes the adaptive tree $T=(N,E)$, the cognitive aggregate
learning machine, with $N$ and $E$ as the set of nodes and edges. Accordingly,
the fragmentation module can extend the tree by adopting clustering and
theoretical-based categorization techniques. We utilize a breadth-first
iterative fragmentation procedure to generate the subspaces for each node.
Thus, we insert the subspaces in-row with parents into the allocated queue,
$Q$, where the genuine $\Omega$ space represents the root node of no parent.
In a FIFO manner, we pop the item $x$ in each iteration from the queue with
pertinent parent and subspace. We then initialize $x$ as node $n$ and append
it to the intrinsic tree. Additionally, we fragment the node $n$ subspace by
adjusting an optimized $\ddot{\rho}$ to retrieve $H$ as the set of child
splits. To continue, we apply trilateral termination criteria on $H$ to
evaluate branching and either append the node $n$ as a leaf or insert each
split of $H$ with $n$ as a parent into the queue, resulting in the growth of
the subspaces in $Q$.

Algorithm 1 Adaptive Tree Generation

Input: $\Omega$
Output: $T,Leaf$

1:$Leaf=\emptyset,N=\emptyset,E=\emptyset,Q=\emptyset$

2:$Q.Enque(\\{\Omega,\emptyset\\})$

3:while $Q\neq\emptyset$ do

4: $\;x=Q.Deque()$

5: $n=Node(x[0],\emptyset,\emptyset)$

6: $N.append(n)$

7: $E.append(\\{n,x[1]\\})$

8: $\ddot{\rho}=AdaptParameters(x[0])$

9: $H=NodeFragmentation(n,\ddot{\rho})$

10: if $Terminate(H)$ then

11: $Leaf.append(n)$

12: else

13: for all h in H do

14: $Q.Enque(\\{h,n\\})$

15: end for

16: end if

17:end while

18:$T=(N,E)$

19:return $T$, Leaf

Prior to the tree construction, the fragmentation procedure divides the
subspace of each vertex $n$ into a set of tractable clusters, to be handled in
two ways as follows:
Theoretical-based Subspace Categorization. In this approach, we employ the
fragmentation $\digamma_{T}^{f}$ to build the tree $T$ within $\Omega$ space
using the cut-points as $\Omega$’s dimensions. Accordingly, $\digamma_{T}^{f}$
can consume the parameter $\ddot{\rho}$ comprising the pair
{$c^{*},c^{*}_{cut}$} to fragment given node on the $c^{*}$ dimension to
maximize the heterogeneity with $c^{*}_{cut}$ as the optimal cut-point. Having
$C$ as the set of dimensions, Eq.17 verbalizes the dimension $c^{*}$ selection
for the fragmentation of node $n$.

$\small c^{*}=\underset{c\in
C}{\arg\max}\\{\digamma^{het}(n^{\Phi},c)\\}\vspace{-3mm}$ (17)

Inspired by [62], the ratio $\digamma^{het}$ retrieves the heterogeneity by
consuming the subspace $\Phi$ and the dimension $c$ (Eq.18).

$\small\digamma^{het}(\Phi,c)=\underset{\varphi_{x},\varphi_{z}\in\Phi}{-\sum}\widetilde{d}_{x,z}(1-\widetilde{d}_{x,z}^{c})+\widetilde{d}_{x,z}^{c}(1-\widetilde{d}_{x,z})\vspace{-3mm}$
(18)

Here $\widetilde{d}_{x,z}$ equates the local normalized cognitive distance
$\digamma^{d}_{\Omega}(\varphi_{x},\varphi_{z})\times d_{max}^{-1}$ with
$d_{max}$ as the maximum local distance (i.e.,
${Max}_{\varphi_{x},\varphi_{z}\in\Phi}\\{\digamma^{d}_{\Omega}(\varphi_{x},\varphi_{z})\\}$),
and $\widetilde{d}^{c}_{x,z}$ is calculated in the shrunk $\Omega$ space by
the dimension $c$. To continue, we can use Eq.19 to assert fragmentation with
the best cut-point $c^{*}_{cut}$ that, on the one hand, maintains cohesiveness
within each of the splits, and on the other hand, ensures unbiased labels in
each resultant subspaces.

$\small c^{*}_{cut}=\underset{k\in c^{*}}{\arg\max}\\{\lambda\underset{\Phi\in
H_{k}}{\biguplus}\digamma^{het}(\Phi,c^{*})+(1-\lambda)\underset{\Phi\in
H_{k}}{\biguplus}\digamma^{imp}(\Phi)\\}\vspace{-2mm}$ (19)

Here $\lambda$ is the adjustment parameter and $H_{k}$ denotes the split pair
formed by cutting the node $n^{\Phi}$ using the cut-point $k$.
Cluster-based Subspace Categorization. In this approach, we adopt the simple
but effective K-means clustering as $\digamma_{T}^{f}$ to fragment the
subspaces. Unlike the theoretical-based model, K-means comprises all
dimensions in partitioning, resulting in a more flexible sub-spacing. As
verbalized in Eq.20, we consume the parameter setting $\ddot{\rho}$ of
$\\{d^{*}\\}$ on node $n$ to categorize the node subspace $n^{\Phi}$ into
$d^{*}$ clusters.

$\small\digamma_{T}^{f}(n,\\{d^{*}\\})=\underset{R}{\arg\min}\\{\overset{|n^{\Phi}|}{\underset{x=1}{\sum}}\overset{d^{*}}{\underset{y=1}{\sum}}r_{xy}\times\digamma_{\Omega}^{d}(\varphi_{x},\Phi_{y}.\tilde{\varphi})\\}\vspace{-2mm}$
(20)

Here R=($r_{xy}$) is an $|n^{\Phi}|\times d^{*}$ matrix, $r_{xy}\in\\{0,1\\}$
estimates if user $x$ participates in cluster $\Phi_{y}$ with
$\Phi_{y}.\tilde{\varphi}$ as medoid, where the matrix assures exclusivity in
fragmentation conditioned by $\sum^{d^{*}}_{y=1}r_{xy}=1$ and
$0<\sum^{|n^{\Phi}|}_{x=1}r_{xy}<|n^{\Phi}|$. Finally, Eq. 21 computes the
ideal number of clusters, denoted by $d^{*}$.

$\small d^{*}=\underset{d}{\arg\max}\\{\underset{\Phi\in
H_{d}}{\biguplus}\digamma^{slh}(\Phi)+\underset{\Phi\in
H_{d}}{\biguplus}\digamma^{imp}(\Phi)-\digamma^{rcp}(H_{d})\\}\vspace{-2mm}$
(21)

Here $H_{d}$ is the set of splits generated by
$\digamma_{T}^{f}(n_{i},\\{d\\})$ and $\digamma^{slh}$ is the silhouette
measure [63].

(a) Theoretical-based

(b) Clustering-based

Figure 4: Proposed Categorizations

Fig. 4 compares the theoretical-based categorization versus clustering. Given
extroversion and neuroticism cognitive cues, the $\Omega$ space can be of two
dimensions. While the theoretical-based approach initially divides the space
into two splits using respective cut-point on neuroticism, the clustering-
based approach groups the surrogate-points into three clusters.

####  4.1.5 Transfer-Based Submodels and Knowledge Core

In this section, we utilize a bottom-up traversing approach to concurrently
construct the transfer-based submodels and the knowledge cores. As visualized
in Fig. 5, for each node $n$, on the one hand, we form the knowledge core
$n^{k}$ by consuming the submodel output of the child nodes, and on the other
hand, we collectively feed the submodel $n^{m}$ with the pertinent video
contents of the users belonging to the node subspace and the knowledge core
$n^{k}$. Hence, the tree can meticulously transfer the knowledge from child
submodels to the corresponding parent submodels.

Figure 5: The Proposed Adaptive Tree Architecture

As a prerequisite to generating the node-specific submodels, we pad the
utterance set of each video to a maximum length $g$ as formulated in Eq. 22:

$\small
g=\underset{v_{i}\in\mathbb{V}}{\max}\digamma_{v}^{len}(v_{i})\vspace{-2mm}$
(22)

Given node $n$, each submodel $n^{m}$ includes three unimodal LSTMs for each
input modal and a multimodal LSTM. The training of $n^{m}$ conjointly conveys
both the node pertinent videos
$\\{v_{i}|v_{i}\in\digamma_{l}^{v}(l_{k}),l_{k}\in n^{\Phi}\\}$ and the
knowledge core $n^{k}$. Given an input video $v_{i}$ with the set of
utterances $[u_{1,i},u_{2,i},\dots,u_{g,i}]$, we define the context criterion
for $u_{i,j}$ as $\\{u_{i,z}|\forall z<g,z\neq j\\}$. As Fig. 6 visualizes the
submodel architecture for $n^{m}$, given feature vectors
$X^{m}=\\{x^{m}_{t}|x^{m}_{t}\in\mathbb{R}^{d_{m}},t\leq g\\}$ with a set of
modals as $m\in(A,V,T)$ and $d_{m}$ as the length of the corresponding feature
vector, the unimodal models can generate modal-specific embeddings,
represented by $H^{m}=\\{h^{m}_{t}|h^{m}_{t}\in\mathbb{R}^{\tilde{d}},t\leq
g\\}$. The resultant embeddings $H^{m}$ and the node knowledge core
$K=\\{k_{t}|k_{t}\in\mathbb{R}^{\tilde{d}},i\leq g\\}$ collectively feed into
the attention-based fusion module to provide the fused input of
$H^{F}=\\{h^{F}_{t}|h^{F}_{t}\in\mathbb{R}^{\tilde{d}},t\leq g\\}$ for the
final multimodal model. To conclude, the multimodal model utilizes $H^{F}$ to
transfer an enhanced knowledge, denoted by
$H^{k}=\\{h^{k}_{t}|h^{k}_{t}\in\mathbb{R}^{\tilde{d}},t\leq g\\}$, to the
parent knowledge core.

Figure 6: Submodels architecture

For both unimodal and multimodal models, INLSTM: $\mathbb{R}^{d_{m}\times
g}\rightarrow\mathbb{R}^{\tilde{d}\times g}$ as an LSTM procedure can convey
sentiment analysis. As formalized in Eq. 23, besides the internal state
$s_{t}^{m}$, and the output $h_{t}^{m}$, the INLSTM comprises three gates of
forget $f_{t}^{m}$, external-input $q_{t}^{m}$, and output $o_{t}^{m}$.

$\small
f_{t}^{m}=\S(U_{f}^{m}.X^{m}+W_{f}^{m}.h_{t-1}^{m}+b_{f}^{m})\vspace{-1mm}$
$\small
q_{t}^{m}=\S(U_{e}^{m}.X^{m}+W_{e}^{m}.h_{t-1}^{m}+b_{e}^{m})\vspace{-1mm}$
$\small
o_{t}^{m}=\S(U_{o}^{m}.X^{m}+W_{o}^{m}.h_{t-1}^{m}+b_{o}^{m})\vspace{-1mm}$
$\small s_{t}^{m}=f_{t}^{m}\odot s_{t-1}^{m}+q_{t}^{m}\odot
tanh(U_{s}^{m}.X^{m}+W_{s}^{m}.h_{t-1}^{m}+b_{s}^{m})\vspace{-1mm}$ $\small
h_{t}^{m}=o_{t}^{m}\odot tanh(s_{t}^{m})\vspace{-1mm}$ (23)

Here, $W^{m}_{f}$, $W^{m}_{e}$, $W^{m}_{o}$, $W^{m}_{s}$, $U^{m}_{f}$,
$U^{m}_{e}$, $U^{m}_{o}$, $U^{m}_{s}\in\mathbb{R}^{d_{m}\times
2d_{m}},b_{f}^{m}$, $b_{c}^{m}$, $b_{o}^{m}$, $b_{s}^{m}\in\mathbb{R}^{d_{m}}$
are training parameters where $\S$ is the sigmoid function and $\odot$ denotes
the element-wise multiplication. Given $m\in(A,V,T)$, the INLSTM can generate
the pertinent embedding output for the given channel as $H^{m}$. Aiming to
carry out the fusion procedure (Eq. 24), we concatenate INLSTM outputs with
the knowledge core $K$ to form $H=\\{H^{A}||H^{V}||H^{T}||K\\}$ that feeds the
multimodal input for the attention-based module.

$\small b_{t}=tanh(W_{b}.H)\vspace{-1mm}$
$\small\alpha_{t}=\sigma(\omega_{b}^{T}.b_{t})\vspace{-1mm}$ $\small
c_{t}=H.\alpha_{t}^{T}\vspace{-1mm}$ $\small
h_{t}^{F}=tanh(W_{h}.c_{t}+U_{h}.H)\vspace{-2mm}$ (24)

Having $\sigma$ as the softmax function, we can retrieve $H^{F}$ as an
attention-based fusion for the output of unimodal layers and the knowledge
core to train the multimodal layer within INLSTM, resulting in $H^{K}$.
Finally, we can transfer the knowledge from $H^{K}$ to feed the knowledge core
of the parent node, continuing the bottom-up learning procedure toward the
root in the hierarchy.

####  4.1.6 Constructing Prediction Module

In general, our proposed framework maps the dataset users to the $\Omega$
space, resulting in an adaptive tree that can hierarchically categorize the
users and learn the ensemble of submodels. The prediction module can estimate
the sentiments by consuming the final improved representation of the data in
the root. As formalized in Eq. 25, the prediction module can utilize a softmax
function on $H_{0}^{K}=[h_{0,1}^{K},h_{0,2}^{K},\dots,h_{0,g}^{K}]$ with
$h_{0,t}^{K}\in\mathbb{R}^{\tilde{d}}$ that is the output of the root node
submodel.

$\small g_{t}=\sigma(W_{g}.(h_{0,t}^{K})^{T}+b_{g})\vspace{-2mm}$ (25) $\small
y_{t}^{\prime}=\underset{m\in[0,1]}{\arg\max}(g_{t}[m])\vspace{-2mm}$

Here, $W_{g},b_{g}$ are parameters we learn during training the prediction
layer, and $Y^{\prime}=\\{y_{t}^{\prime}|y_{t}^{\prime}\in\\{0,1\\},t\leq
g\\}$ is the final set of sentiment predictions for each utterance where
$H_{0}^{K}$ is normalized to a probability distribution over binary sentiments
using cognitive-based perceptions.

####  4.1.7 Kind Neighbors

Given the node submodels with inadequate users, our goal in this section is to
reduce data sparsity. Hence, we employ a non-trivial approach to compensate
insufficiency by borrowing similar users from neighboring nodes. Each user
within a node can then either be original or marked as adopted from adjacent
nodes. Consequently, we propose the dynamic dropout to optimize submodel
training.
The sparsity phenomenon occurs when $|n_{o}^{\Phi}|$, as the number of users
in the subspace of the node $n_{o}$, deems to be less than threshold
$\theta_{b}$. Accordingly, we can locate the nearest neighbors using the least
distance between pairs (Eq. 11) and continue adopting the users from neighbors
as long as the amplitude satisfies the $\theta_{b}$ constraint. Hence, the
higher the dissimilarity between the nodes, the lower the node adaptation
chance, and the higher the dropout ratio will be, resulting in a more
generalized model in training. Given the set of data enclosed with the adopted
users from a neighboring node $n_{c}$, Eq. 26 calculates the dropout ratio
$p^{d}_{c,o}$ for the given $n_{c}$ to facilitate training of the submodel of
$n_{o}$.

$\small p^{d}_{c,o}=\dfrac{\underset{\varphi_{i},\varphi_{j}\in
n_{o}^{\Phi}}{Max}\\{\digamma_{\Omega}^{d}(\varphi_{i},\varphi_{j})\\}}{\digamma_{T}^{d}(n_{c},n_{o})}\times\lambda\vspace{-2mm}$
(26)

Here, $\lambda$ is the adjusting parameter to scale the dropout probability.
Accordingly, the generalization of the model will be proportionate to the
conditional likelihood of the adoption cases for each given original node. We
compute the probability by dividing the maximum distance between the surrogate
points within $n_{o}$ sub-space by the distance between $n_{o}$ and any given
neighboring node $n_{c}$.

###  4.2 Online Phase

In the online phase, we evaluate the output sentiment for the given video
using a cognitive-based ensemble implemented within an adaptive tree. After
leveraging the features of all data channels, we accomplish three procedures
to complete the inference process: Cognitive Assessment, Cognitive Tree
Routing, and Cognitive-Based Prediction.

####  4.2.1 Cognitive Assessment

In this section, given the input video and the participating user,
respectively denoted by $v$ and $v.l$, we utilize the function
$\digamma_{\Omega}^{m}$ to map the user $v.l$ to a corresponding surrogate-
point $\varphi_{\bullet}$ within the $\Omega$ space (Eq. 27).

$\varphi_{\bullet}=\digamma_{\Omega}^{m}(v.l)\vspace{-2mm}$ (27)

This approach is beneficial for cold-start users, where we exploit the
cognitive cues and estimate sentiments simultaneously. Moreover, we can
benefit from mapping to track a user-specific route within the adaptive tree.

####  4.2.2 Adaptive Tree Routing

In this section, we find the most relevant leaf node to the corresponding
surrogate point. To this end, we traverse a route from the root toward the
objective leaf node in the adaptive tree. Following the categorization
approach (Sec. 4.1.4) considered in the construction process of the tree, for
each node $n$ within the traverse procedure, we obtain the similar child node
$\ddot{n}$ to the surrogate-point $\varphi_{\bullet}$.
Theoretical-Based Categorization In this approach, as a prerequisite of
fragmentation, for each node $n$ in the adaptive tree, we hold both dimension
$c^{*}$ and corresponding cut-point $c^{*}_{cut}$ to assign the data to the
left or right child node. We then compare the value of $c^{*}$ at the point
$\varphi_{\bullet}$ versus $c^{*}_{cut}$ to choose the corresponding child
node $\ddot{n}$.
Clustering-Based Categorization Given $N^{c}$ as the set of child nodes, the
clustering-based method computes the distance between $\varphi_{\bullet}$ and
the medoid of the subspace comprising candidate nodes to determine $\ddot{n}$
(Eq. 28).

$\small\ddot{n}=\underset{n_{\kappa}\in{N^{c}}}{\arg\min}\\{\digamma_{\Omega}^{d}(n_{\kappa}^{\Phi}.\tilde{\varphi},\varphi_{\bullet})\\}\vspace{-2mm}$
(28)

Here $n_{\kappa}^{\Phi}.\tilde{\varphi}$ is the subspace medoid of
$n_{\kappa}$. In a nutshell, we firstly traverse the adaptive tree from the
root toward the specific leaf node in an iterative top-down process and
subsequently span the retrieved path in a bottom-up manner to execute the
transfer-based submodels.

####  4.2.3 Cognitive-Based Prediction

Algorithm 2 estimates cognitive-based sentiments for the input video $v$,
comprising four steps: feature extraction, cognitive assessment, adaptive tree
routing, and prediction.

Algorithm 2 Prediction In Online Phase

Input: v
Output: s

1:$X=\emptyset,H=\emptyset,S=\emptyset,P=\emptyset$

2:for all $m\>in\>\\{A,V,T\\}$ do

3: for all $u_{j}\in v$ do

4: $x_{j}^{m}=ExtractFeatures(u_{j}^{m}$)

5: end for

6:end for

7:$\varphi_{\bullet}=\digamma_{\Omega}^{m}(v.l)$

8:$\ddot{n}=n_{0}$

9:while $\ddot{n}\neq\>Null$ do

10: $S.Push(\ddot{n})$

11: $\ddot{n}=SelectChildNode(\ddot{n},\varphi_{\bullet})$

12:end while

13:while $S\neq\emptyset$ do

14: $\ddot{n}=S.Pop()$

15: $\ddot{n}^{k}.append(H)$

16: $H=\ddot{n}^{m}(\\{X^{A},X^{V},X^{T}\\},\ddot{n}^{k})$

17:end while

18:$Y^{\prime}=Predict(H)$ // Eq. 4.1.6

19:$s=MajVote(Y^{\prime})$

20:return s

Given video $v$ with the set of utterances $v=[u_{1},\dots,u_{g}]$ padded to
the length $g$ (Eq. 22), we first extract the modal-specific feature vectors
$X^{m}\in\mathbb{R}^{d_{m}\times g}$ for all utterances, with $m\in(A,V,T)$ as
the input modality and $d_{m}$ as the vector length. We then put forward the
function $\digamma_{\Omega}^{m}$ to map the participating user $v.l$ to a
point $\varphi_{\bullet}$ in the $\Omega$.
Consequently, we devise a novel stack-wise approach to convey the routing
process. Aiming to build the path for the given user, we can trace a hierarchy
from the root toward the user-designated leaf node by obtaining similar child
nodes and pushing them into stack $S$.
Conversely, we process the set of feature vectors ($X^{A}$, $X^{V}$, and
$X^{T}$) via a bottom-up approach using the pertinent route from $S$ to
predict the sentiments of the input $v$. To this end, we iteratively pop a
node $\ddot{n}$ from $S$ and append the submodel output from the previous
iteration to the knowledge core $\ddot{n}^{k}$, following by executing the
node submodel $\ddot{n}^{m}$ on the input feature vectors and the enclosed
$\ddot{n}^{k}$.
Finally, by executing the root submodel in the last iteration, the predict
module utilizes the submodel output of each utterance as
$H\in\mathbb{R}^{\tilde{d}\times g}$ to obtain a set of predictions for each
utterance $Y^{\prime}$. The majority voting on $Y^{\prime}$ can attain the
final cognitive-based sentiment estimation $s$ for $v$.

##  5 Experiment

We conducted extensive experiments on real-world datasets [11] to evaluate our
proposed model in multimodal sentiment analysis. We developed our algorithms
using Tensorflow, Scikit-learn, and Seaborn. Additionally, we performed the
experiments on a server with 2.60GHz Intel Core i7-6700HQ CPU, NVidia GeForce
1080 GPU, and 64GB of RAM (available to download
111https://sites.google.com/view/multi-modalsentimentanalysis).

###  5.1 Configuration

####  5.1.1 Data

For the experiments, we used two publicly available datasets [11] from online
video-sharing platforms. Table II elucidates the corresponding dataset
features.

| # of Users | # of Videos | Avg Utterance | Density
---|---|---|---|---
CMU-MOSI | 93 | 93 | 24 | 96.4%
MOUD | 55 | 79 | 6 | 97.6%
TABLE II: Datasets Statistical Description

CMU-MOSI [6] includes the multimodal reviews on movie contents in English,
where each video comprises multiple segments as utterances. Multiple
annotators assign labels between -3 and +3 to utterances. We make the labels
binary (positive and negative) as required in experiments.
MOUD [7] includes video product reviews in the Spanish language. The
utterances labeled in neural, positive, and negative classes similarly project
into binary values.

####  5.1.2 Benchmark

Aiming to assess the multimodal sentiment prediction, we tested the
statistical hypothesis on the distinctive train and test users for both
datasets. We define the statistical parameters as follows: For each utterance
$u_{j}$ composed by a user $c_{i}$, we determined True Positive when $c_{i}$
expressed positive sentiment on $u_{j}$ while the model predicted the same
positive sentiment, False Positive when $c_{i}$ expressed negative sentiment
on $u_{j}$ while the model predicted positive, True Negative when $u_{j}$
gained negative sentiment while the model predicted the same sentiment, and
False Negative when $u_{j}$ gained positive sentiment while the model
predicted the opposite. Finally, we compute the performance metrics of
Precision, Recall, and F-measure to compare competitors in the multimodal
sentiment analysis.

####  5.1.3 Baselines

This section lists the baselines in multimodal sentiment analysis in both
cognitive and non-cognitive categories.
CHF This hierarchical framework[11] handles three input modals of acoustics,
visual, and textual to compute the sentiments in a hierarchy of uni, bi, and
tri-modal layers.
CAF This model[10] performs the context-dependent sentiment analysis on
utterances using an LSTM network beside an attention-based mechanism to handle
fusion.
SEP Similar to [64], this ensemble model utilizes majority voting on base
models. The median divides cognitive cues to train submodels on high and low
splits.
CTP Our proposed cognitive-based sentiment analysis approach that leverages an
attention-based fusion to transfer the agglomerative knowledge within the
adaptive tree. As explained in Sec. 4.1.4, this method constructs the adaptive
tree through cluster-based categorization of users.
ITP As elucidated in Sec. 4.1.4, this approach replicates CTP where it
conversely builds the adaptive tree via a theoretical-based categorization.

####  5.1.4 Data Cognitive Representation

One of the most notable points of our framework is that the changes in
cognitive features can substantially affect the sentiment analysis results. As
elucidated in Sec. 4.1.2, we map each individual to the $\Omega$ space by
$\digamma_{\Omega}^{m}$ function, resulting in unprecedented cognitive cues.
The standard density function of underlying distributions can compare the
cognitive space of various cues to include OPN, CON, EXT, AGR, and NEU,
representing openness, consciousness, extroversion, agreeableness, and
neuroticism.

Figure 7: Cognitive cue annotation distribution on MOSI dataset

As Fig. 7 depicts, the distribution of cognitive cues, reflected by the users,
follows the Gaussian distribution in both datasets, an indication of high
impurity that is more centralized for AGR property. Such distribution can
constitute a significant contrast between samples that can further facilitate
the categorization of users. Compared to MOSI, the cognitive values pertinent
to OPN, EXT, and NEU are slightly higher in MOUD. Such traits concerning the
sentiment intensity and duration can directly affect emotion manifestation
[37]. Nevertheless, we will further need to investigate if categorizing the
users can result in meaningful differences between datasets.

###  5.2 Effectiveness

####  5.2.1 Categorization Performance

In this section, we perform an intrinsic evaluation of the clustering-based
(CTP) and information-based (ITP) categorization in the adaptive tree (Sec.
4.1.4).
To examine the cluster quality, we utilize both Silhouette and Davies-Bouldin
metrics [13], reporting the average scores for each level in the hierarchical
categorization. Initially, we observe that the more profound the hierarchy
grows toward the leaves (level 4), the lower the silhouette score (Fig. 8). By
contrast, the Davis-Bouldin index, an evaluation based on the cluster
centroids, does not pursue a consistent behavior for ITP, reflecting an
immense loss in the second level and highlighting a sparse division due to the
limitedness of the ITP in generating the split pairs.

(a) MOSI

(b) MOUD

Figure 8: Evaluating clusters by Silhouette and Davies Bouldin

Finally, given higher values for Silhouette (solid lines) and lower numbers
for Davis-Bouldin (dashed lines) in Fig. 8 proves that compared to ITP, the
divisions conveyed by the CTP model can better obtain clusters.

####  5.2.2 Impact Of Trilateral Criteria

To attain optimal growth in the adaptive tree, as elucidated in Sec. 4.1.4, we
employ the trilateral criteria, proposing three exclusive constraints denoted
by $\theta_{p}$, $\theta_{e}$, and $\theta_{a}$. Nevertheless, the more
significant the tree growth, the more inevitable efficiency loss will be.
Hence, we are to seek thresholds that, on the one hand, can improve the
effectiveness of the model, increasing the accuracy, and on the other hand,
can preserve the efficiency with a legitimate growth in the adaptive tree.

(a) Accuracy

(b) Tree Volume

Figure 9: Trilateral Thresholds Comparison

Fig. 9 collectively examines the impact of recuperation ($\theta_{p}$) and
impurity ($\theta_{e}$) on the accuracy and the tree volume. The accuracy
measures in Fig. 9(a) show where $\theta_{p}$ ranges between [0.725, 0.75],
$\theta_{e}$ reflects a positive spike, with better performance for the values
of 0.55 and 0.65 where the latter demonstrates more robustness compared to the
former.
More specifically, as depicted in Fig. 9(b), we notice that compared to 0.55,
setting $\theta_{e}$ to 0.65 can better control the size of the tree,
nominating $\theta_{e}=0.65$ as the best fit threshold. In this way, we
elegantly sacrifice the negligible change in effectiveness to gain a
significant improvement in efficiency.
Fig. 10 further analyzes the selected intervals on the accuracy and tree
volume in a more fine-grained view. Finally, if maximizing the performance of
the proposed framework in effectiveness and efficiency is of indication, the
optimal values of 0.74 and 0.6 for $\theta_{p}$ and $\theta_{e}$ can meet the
preferences.

(a) Accuracy

(b) Tree Volume

Figure 10: Trilateral Thresholds Comparison

####  5.2.3 Impact Of Transferring In The Tree

One notable point about our framework is that it can investigate how the
hierarchical transferring of the knowledge from child submodels to the
corresponding parent nodes can improve the performance of the sentiment
prediction procedure. To empirically track the evidence, we conducted an
experiment to examine statistical measures during a trace from leaf nodes
toward the root, performed on the cognitive tree. More specifically, given the
test set associated with node $n$ (Sec. 4.2.2), we executed sentiment
prediction on the submodel $n^{m}$ using CTP. Finally, we used the average
F1-score for all nodes on the same level of the cognitive tree, reported by
Fig. 11 for MOSI and MOUD datasets of the respective 5 and 3 levels. The
higher the level in the hierarchy, the more comprehensive the knowledge core
will be from transferring the submodel outputs of the lower levels, resulting
in an overall improvement in the tree.

(a) MOSI

(b) MOUD

Figure 11: Level-based comparison of F1 in the cognitive tree

Furthermore, the expected improvement ratio in the MOUD dataset is more than
MOSI, affirming the inference of Sec. 5.1.4, where the general cognitive
differences imply that the sentiment specifics, including density and
duration, vary more for users in the MOUD dataset. Hence, due to a better
contrast between users in the MOUD dataset, the cognitive categorization and
subsequent transferring modules have substantially improved the initial
predictions.

####  5.2.4 Effectiveness of multimodal sentiment analysis

Given the benchmark in Sec. 5.1.2, our goal is to examine the competitors’
performance (Sec. 5.1.3) in multimodal sentiment prediction. Unlike CHF and
CAF, the other three, SEP, ITP, and CTP, are cognitive-aware in sentiment
inference procedure. Table III displays the statistics of an average 5-fold
test. While $rc^{+}$ and $rc^{-}$ report the recall for the respective
positive and negative data classes, the pc and $F_{1}$ represent the precision
and F1-score metrics.

TABLE III: Baseline Comparison | MOSI | MOUD
---|---|---
| pc | $rc^{+}$ | $rc^{-}$ | $F_{1}$ | pc | $rc^{+}$ | $rc^{-}$ | $F_{1}$
CHF | 0.865 | 0.862 | 0.869 | 0.865 | 0.694 | 0.534 | 0.793 | 0.686
CAF | 0.882 | 0.875 | 0.893 | 0.887 | 0.699 | 0.504 | 0.807 | 0.644
SEP | 0.889 | 0.879 | 0.897 | 0.888 | 0.702 | 0.506 | 0.813 | 0.661
ITP | 0.895 | 0.882 | 0.9 | 0.893 | 0.717 | 0.505 | 0.88 | 0.683
CTP | 0.898 | 0.885 | 0.904 | 0.896 | 0.727 | 0.542 | 0.844 | 0.695

For the MOSI dataset, the cognitive-oriented approaches, including ITP and
CTP, significantly improve the performance of non-cognitive methods.
Nevertheless, while CTP unanimously gains the best performance in all metrics,
the rate of improvement among cognitive-aware models (SEP, ITP, and CTP) is
less than 1%.
More specifically, the MOUD dataset reveals some latent aspects in performance
concerning cognitive methods. Given the F1 measure as the main metric to
signify the best performance, CTP overpasses other competitors, including ITP.
We note that while the number of positive data instances dominates negative
data samples, the second-best method, ITP, plunges up to 4% less than other
rivals (CHF and CTP). Therefore, based on the nature of the dataset, we can
rely on 0.695 instead of 0.683 to gain up to 4% of improvement for positive
recall, sacrificing the negligible improvement for the limited negative data
samples. It worths mentioning that using ITP in a converse manner can attain
up to 4% of improvement for negative recall when the positive samples may
flatten compared to negative data instances.
Since the recommended approaches of ITP and CTP are the primary attempts in
utilizing cognitive cues in the multimodal sentiment prediction, we compare
the results with non-cognitive baselines. Inherently, we observe the advantage
of considering cognitive cues in improving the overall performance, where
compared to non-cognitive-based counterparts, the cognitive-based approaches
show an average of 2% improvement in the $F_{1}$ measure. In a nutshell, the
results signify that using cognitive cues can maximize the performance on the
negative data class, reflecting an average of 3.6% increase in the recall.
More specifically, we probe the role of agglomerative ensembles in cognitive-
aware methods (ITP, CTP, and SEP). While benefitting the MOUD dataset more,
compared to SEP, the $F_{1}$-measures on ITP and CTP get enhanced
significantly in both datasets. The following affirms the empirical results in
Sec. 5.2.3 that compared to MOSI, the ratio of improvement in each step of
agglomeration is considerably higher for MOUD. On the one hand, our proposed
CTP framework is capable of creating cohesive splits with higher separation in
the adaptive tree (Sec. 5.2.1), and on the other hand, by maintaining a
relatively higher performance compared to ITP, it is capable of effectively
distinguishing the users by pertinent cognitive cues in their brief contents.
To conclude, the CTP model overpasses ITP both in the intrinsic and extrinsic
analysis, proving that the clustering-based adaptive trees can better track
cognitive-aware sentiments.

####  5.2.5 Multimodal Sentiment Analysis Complexity

In this section, we evaluate the time complexity of our framework from a
theoretical computer science perspective. The proposed framework comprises
offline and online components, where the former supplies the output of the
overall training to the latter, justifying more time consumption by the
offline section. Henceforth, we first report the time complexity of the
offline components for both CTP and ITP approaches, followed by complexity
intuitions about the online section. In retrospect, Eq.29 verbalizes the
overall complexity through aggregating the comprising times in the framework.

$\small T=T_{f}+T_{c}+T^{t}_{b}+T^{t}_{m}+T_{p}\vspace{-1mm}$ (29)

Here, $T_{f}$, $T_{c}$, $T^{t}_{b}$, $T^{t}_{m}$, and $T_{p}$ respectively
denote the time complexities of the procedures, including feature extraction,
cognitive cue annotation, top-down tree generation, bottom-up model
construction, and the final prediction training. Due to the insignificance
effect of the feature extraction and cognitive cue annotation modules, the
relative time complexities of $T_{f}$ and $T_{c}$ fade in contrast with the
summation of the other efficiency metrics, Eq. 29. To this end, we further
examine the complexity of $T^{t}_{b}$, $T^{t}_{m}$, and $T_{p}$.
Tree construction: $T^{t}_{b}$ denotes the complexity for top-down
construction of the adaptive tree. Given each node $n$ with $m$ instances, a
three-fold set of actions are involved in construction: fragmentation
parameter setting $T^{t}_{par}$, subspace partitioning $T^{t}_{frag}$, and
trilateral termination criteria $T^{t}_{stop}$.
$T^{t}_{stop}$ is the time requirements to control all the parameters in the
worst-case. Given $H$ as the set of splits for the node $n$, we can compute
$T^{t}_{stop}$ by the summation of three complexities, including amplitude
$O(|H|\times 1)$, impurity $O(|H|\times 2\times 1)$, and recuperation
$O(2\times|H|\times m)$ where the overall complexity for $T^{t}_{stop}$ can
simplify as $O(m)$.
Since the complexity $T^{t}_{par}$ referring to the parameter setting differs
for ITP and CTP models, on the one hand, we select the dimension and the
respective cut-point of the node subspace for ITP, and on the other hand, we
select the number of clusters in each node for CTP. Correspondingly, the
computation complexity of ITP equates to the summation of selection
complexities for dimension $O({|C|}^{2}\times m^{2})$ and the cut-point
$O(k\times m^{2})$, where $|C|$ and $k$ are respectively the number of
dimensions in $\Omega$ space and candidate points in the selection process.
Given 5 and 10 as the respective values for $|C|$ and $k$, we can simplify
$T^{t}_{par}$ to $O(m^{2})$ in the ITP model. Similarly, given Eq. 21, we can
obtain the computation complexity for CTP by selecting an optimum number of
children as $O(k\times m^{2})$, where $k$ counts the constant number of
candidates, ending with $O(m^{2})$ as the time requirements of $T^{t}_{par}$.
Finally, we can respectively attain $T^{t}_{frag}$ for ITP and CTP models
using $O(m)$ and $O(k\times m)$, where $k$ denotes the constant number of
clusters, resulting in the same complexity of $O(m)$ for both. For ease of
computation, the construction complexity in each node can be calculated by the
aggregation of the three-fold actions, equating to $O(m+m^{2}+m)$, where
approximately is further simplified to $O(m^{2})$. Eq. 30 formalizes the
complexity of the tree construction ($T^{t}_{b}$).

$\small T^{t}_{b}\approx
O(m_{01}^{2})+O(m_{11}^{2}+m_{12}^{2}+\dots)+\dots\vspace{-1mm}$ (30)

Here, we aggregate the computational complexity for the nodes in each level.
We further notice that the number of instances in each node is less than the
total count in the root $m_{01}$. Where we suppose the criterion
$\sum_{j}m_{ij}\leq m_{01}$ for each level $i$, we can approximately conclude
Eq. 30 as $T^{t}_{b}\approx O(l\times m_{01}^{2})$. Accordingly, because of
limited capacity for the growth of the adaptive tree, we can disregard $l$,
the number of levels, in the final complexity.
Tree submodel construction: Given the training procedure in LSTM models, we
assume that the time required for each weight in a time step is $O(1)$. Hence,
concerning $W$ as the total number of training parameters, the complexity can
be designated as $O(W)$ in each time step. As verbalized in Eq. 31, $W$ can
disregard the biases and include three final parameters of memory cells,
input, and output units, respectively denoted by $d_{mc}$, $d_{i}$, and
$d_{o}$.

$\small W=4\times d_{mc}+4\times d_{mc}\times d_{i}+d_{mc}\times d_{o}+3\times
d_{mc}\vspace{-1mm}$ (31)

To this end, for a node $n$ with $m$ instances, $O(W\times e\times m\times t)$
can represent the time for training an LSTM, where $e$ is the number of epochs
and $t$ is the time steps, equating to the number of utterances within each
video (Eq. 22). Both $e$ and $t$ are constant values and can be dismissed
[65].
Concerning the same input length, the complexity of the four LSTMs will be
identical to $O(W\times m)$. As Eq. 32 elucidates the total summation of the
complexity for the proposed adaptive tree $T^{t}_{m}$.

$\small T^{t}_{m}\approx O(W\times m_{01})+O(W\times m_{11}+W\times
m_{12}+\dots)+\dots\vspace{-2mm}$ (32)

Likewise, we can put a proposition forward to simplify the complexity as
$O(l\times W\times m_{01})$.
Prediction module: converting an R-dimensional input to an S-dimensional
output demands a time requirement of $O(R\times S)$. Here, the training time
$T_{p}$ will be linear and equal to $O(m_{01})$ with $m_{01}$ as the total
number of instances.
Finally, following Eq. 29, the overall running-time complexity for the offline
section will be the same for both CTP and ITP approaches, formalized by Eq.
33:

$\small T\approx O(l\times m_{01}^{2})+O(l\times W\times
m_{01})+O(m_{01})\vspace{-2mm}$ (33)

By employing simplification on the above equation, the computational
complexity of the offline section in the framework can be related to $T\approx
O(m_{01}^{2})$.
Likewise, the time complexity for the online section will be linear with the
order of $O(\acute{m})$, where $\acute{m}$ counts the number of input
instances. Since the time-sensitive scenarios, including human-computer
interaction, can adopt multimodal affective analysis, the running time of the
online module deems quite crucial. Therefore, the efficient polynomial
approach proposed in this paper can be significantly useful in the practical
requirements of real-time frameworks.

##  6 Conclusion

In this work, we propose a novel unified framework to predict multimodal
sentiments by consuming the cognitive cues in user contents. In summary, to
leverage individual characteristics, we firstly map each individual to a
cognitive space based on exploited latent cues and subsequently perform an
agglomerative ensemble to carry out prediction. Aiming to form the ensemble,
we generate the novel adaptive tree by a hierarchical categorization of
individuals in the given cognitive space that is further followed by
developing transferring-based submodels. Correspondingly, we employ both
clustering and theoretical-based models to classify individuals. Aiming to
avoid data sparsity caused by the construction of submodels, we devise a non-
trivial deep learning approach based on a dynamic dropout strategy to borrow
data from neighboring nodes. Finally, the prediction module leverages the
ensemble outcome of the adaptive tree to analyze sentiments, deciding whether
the multi-media contents reflect a positive or a negative attitude.
The experimental results on two real-world datasets reveal that leveraging
latent cognitive cues can enhance multimodal sentiment analysis, the reason
why the proposed solution in this paper outperforms other trending approaches.
From another perspective, the proposed agglomerative ensemble can better
foster the impact of cognitive cues in sentiment analysis via altering the
theoretical-based module by a clustering-based categorization technique. To
continue, we can integrate adversarial networks into the adaptive tree to
further compensate for data incompleteness and incorporate the personality
effects into the fusion-based module to support the attention function. We
leave these tasks to tackle in the future.

## References

  * [1] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” _Information Fusion_ , vol. 37, pp. 98–125, 2017.
  * [2] L.-A. Perez-Gaspar, S.-O. Caballero-Morales, and F. Trujillo-Romero, “Multimodal emotion recognition with evolutionary computation for human-robot interaction,” _Expert Systems with Applications_ , vol. 66, pp. 42–61, 2016.
  * [3] M. Ebrahimi, A. H. Yazdavar, and A. Sheth, “Challenges of sentiment analysis for dynamic events,” _IEEE Intelligent Systems_ , vol. 32, no. 5, pp. 70–75, 2017.
  * [4] H. Li, J. Cui, B. Shen, and J. Ma, “An intelligent movie recommendation system through group-level sentiment analysis in microblogs,” _Neurocomputing_ , vol. 210, pp. 164–173, 2016.
  * [5] R. Ren, D. D. Wu, and T. Liu, “Forecasting stock market movement direction using sentiment analysis and support vector machine,” _IEEE Systems Journal_ , vol. 13, no. 1, pp. 760–770, 2018.
  * [6] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,” _IEEE Intelligent Systems_ , vol. 31, no. 6, pp. 82–88, 2016.
  * [7] V. Pérez-Rosas, R. Mihalcea, and L.-P. Morency, “Utterance-level multimodal sentiment analysis,” in _Proc. of the 51st Annual Meet. of the Association for Computational Linguistics (Vol. 1)_ , 2013, pp. 973–982.
  * [8] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, no. 4, p. 335, 2008.
  * [9] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” in _Proc. of the Conf. on Empirical Methods on Natural Language Processing (EMNLP)_ , 2017.
  * [10] S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P. Morency, “Context-dependent sentiment analysis in user-generated videos,” in _Proc. of the 55th annual Meet. of the association for computational linguistics (Vol. 1)_ , 2017, pp. 873–883.
  * [11] N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria, “Multimodal sentiment analysis using hierarchical fusion with context modeling,” _Knowledge-based systems_ , vol. 161, pp. 124–133, 2018.
  * [12] M. G. Huddar, S. S. Sannakki, and V. S. Rajpurohit, “An ensemble approach to utterance level multimodal sentiment analysis,” in _2018 Inter. Conf. on Computational Techniques, Electronics and Mechanical Systems (CTEMS)_. IEEE, 2018, pp. 145–150.
  * [13] S. Najafipour, S. Hosseini, W. Hua, M. R. Kangavari, and X. Zhou, “Soulmate: Short-text author linking through multi-aspect temporal-textual embedding,” _IEEE Trans. on Knowledge and Data Engineering_ , 2020.
  * [14] S. Hosseini, S. Najafipour, N.-M. Cheung, H. Yin, M. R. Kangavari, and X. Zhou, “Teags: time-aware text embedding approach to generate subgraphs,” _Data Mining and Knowledge Discovery_ , vol. 34, pp. 1136–1174, 2020.
  * [15] W. Hua, Z. Wang, H. Wang, K. Zheng, and X. Zhou, “Understand short texts by harvesting and analyzing semantic knowledge,” _IEEE transactions on Knowledge and data Engineering_ , vol. 29, no. 3, pp. 499–512, 2016.
  * [16] C. Song, X.-K. Wang, P.-f. Cheng, J.-q. Wang, and L. Li, “Sacpc: A framework based on probabilistic linguistic terms for short text sentiment analysis,” _Knowledge-Based Systems_ , vol. 194, p. 105572, 2020.
  * [17] Y. Sun and G. Wen, “Ensemble softmax regression model for speech emotion recognition,” _Multimedia Tools and Applications_ , vol. 76, no. 6, pp. 8305–8328, 2017.
  * [18] L. Kessous, G. Castellano, and G. Caridakis, “Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis,” _Journal on Multimodal User Interfaces_ , vol. 3, no. 1-2, pp. 33–48, 2010.
  * [19] C. A. Corneanu, M. O. Simón, J. F. Cohn, and S. E. Guerrero, “Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications,” _IEEE Trans. on pattern analysis and machine intelligence_ , vol. 38, no. 8, pp. 1548–1568, 2016\.
  * [20] F. Eyben, M. Wöllmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie, “On-line emotion recognition in a 3-d activation-valence-time continuum using acoustic and linguistic cues,” _Journal on Multimodal User Interfaces_ , vol. 3, no. 1-2, pp. 7–19, 2010.
  * [21] M. Wöllmer, F. Weninger, T. Knaup, B. Schuller, C. Sun, K. Sagae, and L.-P. Morency, “Youtube movie reviews: Sentiment analysis in an audio-visual context,” _IEEE Intelligent Systems_ , vol. 28, no. 3, pp. 46–53, 2013.
  * [22] G. Evangelopoulos, A. Zlatintsi, A. Potamianos, P. Maragos, K. Rapantzikos, G. Skoumas, and Y. Avrithis, “Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention,” _IEEE Trans. on Multimedia_ , vol. 15, no. 7, pp. 1553–1568, 2013.
  * [23] G. Chanel, C. Rebetez, M. Bétrancourt, and T. Pun, “Emotion assessment from physiological signals for adaptation of game difficulty,” _IEEE Trans. on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , vol. 41, no. 6, pp. 1052–1063, 2011.
  * [24] S. Poria, H. Peng, A. Hussain, N. Howard, and E. Cambria, “Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis,” _Neurocomputing_ , vol. 261, pp. 217–230, 2017.
  * [25] R. Zall and M. R. Keyvanpour, “Semi-supervised multi-view ensemble learning based on extracting cross-view correlation,” _Advances in Electrical and Computer Engineering_ , vol. 16, no. 2, pp. 111–125, 2016.
  * [26] R. Zall and M. R. Kangavari, “On the construction of multi-relational classifier based on canonical correlation analysis,” _Inter. Journal of Artificial Intelligence_ , vol. 17, no. 2, pp. 23–43, 2019.
  * [27] V. Rozgić, S. Ananthakrishnan, S. Saleem, R. Kumar, and R. Prasad, “Ensemble of svm trees for multimodal emotion recognition,” in _Proc. of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conf._ IEEE, 2012, pp. 1–4.
  * [28] R. Liu, Y. Shi, C. Ji, and M. Jia, “A survey of sentiment analysis based on transfer learning,” _IEEE Access_ , vol. 7, pp. 85 401–85 412, 2019.
  * [29] B. Felbo, A. Mislove, A. Søgaard, I. Rahwan, and S. Lehmann, “Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm,” in _Proc. of the Conf. on Empirical Methods on Natural Language Processing (EMNLP)_ , 2017.
  * [30] D. Hazarika, S. Poria, R. Zimmermann, and R. Mihalcea, “Conversational transfer learning for emotion recognition,” _Information Fusion_ , vol. 65, pp. 1–12, 2021.
  * [31] D. Tamil Priya and J. Divya Udayan, “Transfer learning techniques for emotion classification on visual features of images in the deep learning network,” _Inter. Journal of Speech Technology_ , vol. 23, pp. 361–372, 2020.
  * [32] J. Deng, S. Frühholz, Z. Zhang, and B. Schuller, “Recognizing emotions from whispered speech based on acoustic feature transfer learning,” _IEEE Access_ , vol. 5, pp. 5235–5246, 2017.
  * [33] H. Kaya, F. Gürpınar, and A. A. Salah, “Video-based emotion recognition in the wild using deep transfer learning and score fusion,” _Image and Vision Computing_ , vol. 65, pp. 66–75, 2017.
  * [34] J. Gideon, S. Khorram, Z. Aldeneh, D. Dimitriadis, and E. M. Provost, “Progressive neural networks for transfer learning in emotion recognition,” in _Proc. INTERSPEECH_ , 2017, pp. 1098–1102.
  * [35] M. S. Akhtar, D. S. Chauhan, D. Ghosal, S. Poria, A. Ekbal, and P. Bhattacharyya, “Multi-task learning for multi-modal emotion recognition and sentiment analysis,” in _Proc. of the 2019 Conf. of the Association for Computational Linguistics (Vol. 1)_ , 2019, p. 370–379.
  * [36] M. S. Akhtar, T. Garg, and A. Ekbal, “Multi-task learning for aspect term extraction and aspect sentiment classification,” _Neurocomputing_ , 2020\.
  * [37] P. Verduyn and K. Brans, “The relationship between extraversion, neuroticism and aspects of trait affect,” _Personality and Individual Differences_ , vol. 52, no. 6, pp. 664–669, 2012.
  * [38] G. Mohammadi and A. Vinciarelli, “Automatic personality perception: Prediction of trait attribution based on prosodic features,” _IEEE Trans. on Affective Computing_ , vol. 3, no. 3, pp. 273–284, 2012.
  * [39] N. Majumder, S. Poria, A. Gelbukh, and E. Cambria, “Deep learning-based document modeling for personality detection from text,” _IEEE Intelligent Systems_ , vol. 32, no. 2, pp. 74–79, 2017.
  * [40] L. Teijeiro-Mosquera, J.-I. Biel, J. L. Alba-Castro, and D. Gatica-Perez, “What your face vlogs about: Expressions of emotion and big-five traits impressions in youtube,” _IEEE Trans. on Affective Computing_ , vol. 6, no. 2, pp. 193–205, 2014.
  * [41] Y. Güçlütürk, U. Güçlü, X. Baro, H. J. Escalante, I. Guyon, S. Escalera, M. A. J. Van Gerven, and R. Van Lier, “Multimodal first impression analysis with deep residual networks,” _IEEE Trans. on Affective Computing_ , vol. 9, no. 3, pp. 316–329, 2017.
  * [42] L. Batrinca, N. Mana, B. Lepri, N. Sebe, and F. Pianesi, “Multimodal personality recognition in collaborative goal-oriented tasks,” _IEEE Trans. on Multimedia_ , vol. 18, no. 4, pp. 659–673, 2016.
  * [43] S. Aslan, U. Güdükbay, and H. Dibeklioğlu, “Multimodal assessment of apparent personality using feature attention and error consistency constraint,” _Image and Vision Computing_ , vol. 110, p. 104163, 2021.
  * [44] A. E. Başak, U. Güdükbay, and F. Durupınar, “Using real life incidents for creating realistic virtual crowds with data-driven emotion contagion,” _Computers & Graphics_, vol. 72, pp. 70–81, 2018.
  * [45] F. Durupinar, N. Pelechano, J. Allbeck, U. Güdükbay, and N. I. Badler, “How the ocean personality model affects the perception of crowds,” _IEEE Computer Graphics and Applications_ , vol. 31, no. 3, pp. 22–31, 2009.
  * [46] H.-C. Yang and Z.-R. Huang, “Mining personality traits from social messages for game recommender systems,” _Knowledge-Based Systems_ , vol. 165, pp. 157–168, 2019.
  * [47] N. Capuano, G. D’Aniello, A. Gaeta, and S. Miranda, “A personality based adaptive approach for information systems,” _Computers in Human Behavior_ , vol. 44, pp. 156–165, 2015.
  * [48] J. Lin, W. Mao, and D. D. Zeng, “Personality-based refinement for sentiment classification in microblog,” _Knowledge-Based Systems_ , vol. 132, pp. 204–214, 2017.
  * [49] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, “Covarep—a collaborative voice analysis repository for speech technologies,” in _2014 ieee inter. conf. on acoustics, speech and signal processing (icassp)_. IEEE, 2014, pp. 960–964.
  * [50] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proc. of the 18th ACM inter. conf. on Multimedia_ , 2010, pp. 1459–1462.
  * [51] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in _2018 13th IEEE Inter. Conf. on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 59–66.
  * [52] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
  * [53] S. Hosseini, H. Yin, M. Zhang, Y. Elovici, and X. Zhou, “Mining subgraphs from propagation networks through temporal dynamic analysis,” in _2018 19th IEEE Inter. Conf. on Mobile Data Management (MDM)_. IEEE, 2018, pp. 66–75.
  * [54] S. Hosseini, S. Unankard, X. Zhou, and S. Sadiq, “Location oriented phrase detection in microblogs,” in _Inter. Conf. on Database Systems for Advanced Applications_. Springer, 2014, pp. 495–509.
  * [55] J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in _Proc. of the 2014 Conf. on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1532–1543.
  * [56] A. V. Arkhangel’Skii and V. V. Fedorchuk, _General topology I: basic concepts and constructions dimension theory_. Springer Science & Business Media, 2012, vol. 17.
  * [57] T. Ge, K. He, Q. Ke, and J. Sun, “Optimized product quantization,” _IEEE Trans. on pattern analysis and machine intelligence_ , vol. 36, no. 4, pp. 744–755, 2013.
  * [58] E. W. T. Ngai, S. S. C. Tao, and K. K. L. Moon, “Social media research: Theories, constructs, and conceptual frameworks,” _Inter. journal of information management_ , vol. 35, no. 1, pp. 33–44, 2015.
  * [59] Y. A. Malkov and D. A. Yashunin, “Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs,” _IEEE Trans. on pattern analysis and machine intelligence_ , vol. 42, no. 4, pp. 824–836, 2018.
  * [60] A. A. Taha and A. Hanbury, “An efficient algorithm for calculating the exact hausdorff distance,” _IEEE Trans. on pattern analysis and machine intelligence_ , vol. 37, no. 11, pp. 2153–2163, 2015.
  * [61] M. Li, W. Zuo, S. Gu, J. You, and D. Zhang, “Learning content-weighted deep image compression,” _IEEE Trans. on pattern analysis and machine intelligence_ , 2020.
  * [62] J. Basak and R. Krishnapuram, “Interpretable hierarchical clustering by constructing an unsupervised decision tree,” _IEEE Trans. on knowledge and data engineering_ , vol. 17, no. 1, pp. 121–132, 2005.
  * [63] F. Iglesias, T. Zseby, and A. Zimek, “Absolute cluster validity,” _IEEE Trans. on pattern analysis and machine intelligence_ , vol. 42, no. 9, pp. 2096–2112, 2019.
  * [64] K. D. Feuz and D. J. Cook, “Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (fsr),” _ACM Trans. on Intelligent Systems and Technology (TIST)_ , vol. 6, no. 1, pp. 1–27, 2015.
  * [65] W. Yuan, C. Liu, F. Liu, S. Li, and D. W. K. Ng, “Learning-based predictive beamforming for uav communications with jittering,” _IEEE Wireless Communications Letters_ , vol. 9, no. 11, pp. 1970–1974, 2020.

|  Sana Rahmani is a researcher in the Computational Cognitive laboratory of
Iran University of Science and Technology (IUST). She received M.Sc. from IUST
and B.S. from the University of Kurdistan, Iran, in software engineering. Her
research interests include multimodal affective analysis, Human-Computer-
Interaction, machine learning, and data analysis.
---|---
|  Saeid Hosseini currently works as an assistant professor at Sohar
University. He won the Australian Postgraduate Award and received Ph.D. degree
in Computer Science from the University of Queensland, Australia, in 2017. He
has also completed two post docs in Singapore and Iran. His research interests
include spatiotemporal database, dynamical processes, data and graph mining,
big data analytics, recommendation systems, and machine learning.
---|---
|  Raziyeh Zall received the B.S degree from University of Shahid Beheshti of
Iran, and M.Sc degree from the Alzahra university of Iran. She is currently
working toward the PHD degree in computational cognitive models laboratory at
Iran University of Science and Technology. Her research interests include
affective and cognitive computing, NLP, and Multi view learning.
---|---
|  Mohammad Reza Kangavari received B.Sc. in computer science from the Sharif
University of Technology, M.Sc. from Salford, and Ph.D. from the University of
Manchester. He is an associate professor at the Iran University of Science and
Technology. His research interests include Intelligent Systems, Human-
Computer-Interaction, Cognitive Computing, Machine Learning, and Sensor
Networks.
---|---
|  Sara Kamran is a researcher in the Computational Cognitive laboratory of
Iran University of Science and Technology (IUST). She received M.Sc. from IUST
and B.S. from the Urmia University of Technology, Iran, in software
engineering. Her research interests include affective and cognitive computing,
Human-Computer-Interaction, NLP, machine learning, and data analysis.
---|---
|  Wen Hua currently works as a Lecturer at the University of Queensland. She
received her doctoral and bachelor degrees in Computer Science from Renmin
University. Her current research interests include natural language
processing, information extraction and retrieval, text mining, social media
analysis, and spatiotemporal data analytics. She has published articles in
reputed venues including SIGMOD, TKDE, VLDBJ.
---|---