# Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment
Analysis

Wei Han  ISTD, SUTDSingaporeSingapore<EMAIL_ADDRESS>,  Hui Chen
ISTD, SUTDSingaporeSingapore<EMAIL_ADDRESS>,  Alexander Gelbukh
CIC, IPNMexico CityCDMXMexico<EMAIL_ADDRESS>,  Amir Zadeh  LTI,
CMUPittsburghPAUSA<EMAIL_ADDRESS>,  Louis-philippe Morency  LTI,
CMUPittsburghPAUSA<EMAIL_ADDRESS>and  Soujanya Poria  ISTD,
SUTDSingaporeSingapore<EMAIL_ADDRESS>
(2021)

###### Abstract.

Multimodal sentiment analysis aims to extract and integrate semantic
information collected from multiple modalities to recognize the expressed
emotions and sentiment in multimodal data. This research area’s major concern
lies in developing an extraordinary fusion scheme that can extract and
integrate key information from various modalities. However, previous work is
restricted by the lack of leveraging dynamics of independence and correlation
between modalities to reach top performance. To mitigate this, we propose the
Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs
fusion (relevance increment) and separation (difference increment) on pairwise
modality representations. The two parts are trained simultaneously such that
the combat between them is simulated. The model takes two bimodal pairs as
input due to the known information imbalance among modalities. In addition, we
leverage a gated control mechanism in the Transformer architecture to further
improve the final output. Experimental results on three datasets (CMU-MOSI,
CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the
SOTA. The implementation of this work is available at
https://github.com/declare-lab/multimodal-deep-learning.

cross-modal processing; multimodal fusion; multimodal representations

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: Montreal ’21: Proceedings of the 23rd ACM
International Conference on Multimodal Interaction; ††booktitle: ;††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer methodologies Neural
Networks††ccs: Information System Multimedia information systems††ccs:
Sentiment analysis

##  1\. Introduction

Figure 1. Task formulation of the multimodal sentiment analysis and two types
of fusion schemes. The displayed data are sampled from CMU-MOSI dataset.

With the unprecedented prevalence of social media in recent years and the
availability of phones with high-quality cameras, we witness an explosive
boost of multimodal data, such as video clips posted on different social media
platforms. Such multimodal data consist of three channels: visual (image),
acoustic (voice), and transcribed linguistic (text) data. Different modalities
in the same data segment are often complementary to each other, providing
extra cues for semantic and emotional disambiguation (Ngiam et al., 2011). For
example, the phrase “apple tree” can indicate what the blurred red fruit on
the tree is in an image, and a smiling face can clarify that some seemingly
impolite words are a friendly joke. On the other hand, the three modalities
usually possess unique statistical properties that make them to some extent
mutually independent—say, one modality can stay practically constant while the
other one exhibits large changes (Srivastava and Salakhutdinov, 2014).
Accordingly, a crucial issue in multimodal language processing is how to
integrate heterogeneous data efficiently. A good fusion scheme should extract
and integrate meaningful information from multiple modalities while preserving
their mutual independence.

In this paper, we focus on the problem of multimodal sentiment analysis (MSA).
As Fig. 1 suggests, given data from multiple modality sources, the goal of MSA
is to exploit fusion techniques to combine these modalities to make
predictions for the labels. In the context of emotion recognition and
sentiment analysis, multimodal fusion is essential since emotional cues are
often spread across different modalities (Atrey et al., 2010). Previous
research in this field (Zadeh et al., 2017; Tsai et al., 2019a; Hazarika et
al., 2020) mostly adopted ternary-symmetric architectures, where bidirectional
relationships in every modality pair are modeled in some way. However, as it
has been pointed out by several past research (Chen et al., 2017; Tsai et al.,
2019a; Sun et al., 2020), the task-related information is not evenly
distributed between the modalities. The architectures that do not account for
this difference in the relative importance of the three modalities fail to
fuse them correctly, which degrades the model’s performance.

To address this issue, we introduce a fusion scheme that we call Bi-Bimodal
Fusion Network (BBFN) to balance the contribution of different modality pairs
properly. This fusion scheme, consisting of two bi-modal fusion modules, is
quite different from traditional ternary symmetric one; see Fig. 1. Since it
has been empirically shown that the text modality is most significant (Tsai et
al., 2019a; Pham et al., 2019b), our model takes two text-related modality
pairs, TV (text-visual) and TA (text-acoustic), as respective inputs for its
two bimodal learning modules. Then it iteratively impels modalities to
complement their information through interactive learning with their
corresponding counterparts. To ensure fairness in the bidirectional learning
process for both modalities, the two learning networks in each model should be
identical. The basic framework of our model is layers of stacked Transformers,
which have been proven efficient in multimodal learning (Yu et al., 2020).

However, a new problem arises in our implementation. As fusion proceeds, the
representation vectors of the fusion results involved with a modality pair
tend to become closer in the hidden space; we call it feature space collapse.
Moreover, the repeated structures of transformers in the stacked architecture
exacerbate this trend, impairing the mutual independence between different
modalities present in the multimodal data–a crucial property for the
feasibility of multimodal fusion. To tackle this problem, we introduce in our
BBFN the layer-wise feature space separator, as a local regularizer that
divides the feature space of different modalities in order to assure mutual
independence between modalities.

We evaluated our model on two subtasks of MSA—sentiment intensity prediction
and humor recognition—using three datasets: CMU-MOSI, CMU-MOSEI, and UR-FUNNY.
Our experimental results show that our model outperforms state-of-the-art
models on almost all metrics. Moreover, ablation study and further analysis
show the effectiveness of our architecture.

Our contributions can be summarized as follows:

  * •

Bi-bimodal fusion: We introduce a novel fusion scheme for MSA, which consists
of two Transformer-based bimodal learning modules, each one taking a modality
sequence pair as input and performing a progressive fusion locally in its two
modality complementation modules.

  * •

Regularization: To enforce modality representations to be unique and different
from each other, we use a modality-specific feature separator, which
implicitly clusters homogeneous representations and splits heterogeneous ones
apart in order to maintain mutual independence between modalities.

  * •

Control: We introduce a gated control mechanism to enhance the Transformer-
based fusion process.

##  2\. Related Work

In this section, we briefly overview related work in MSA and multimodal
fusion.

###  2.1. Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) mainly focuses on integrating multiple
resources, such as acoustic, visual, and textual information, to comprehend
varied human emotions (Morency et al., 2011). In the past few years, deep
neural networks have been employed in learning multimodal representation in
sentiment analysis, such as Long Short-Term Memory (LSTM), which is used to
model long-range dependencies from low-level multimodal features (Wöllmer et
al., 2013; Chen et al., 2017) and SAL-CNN (Wang et al., 2017), which utilizes
a select-additive learning procedure to improve the generalizability of
trained neural networks.

Most of the previous work in this area focuses on early or late fusion. For
example, Zadeh et al. (2017) proposed a Tensor Fusion Network, which blends
different modality representations at a deeper layer. As attention mechanism
becomes more and more popular, Zadeh et al. (2018b) modified LSTM with a novel
Multi-attention Block to capture interactions between modalities through time.
Also, Gu et al. (2018) introduced a hierarchical attention strategy with word-
level fusion to classify utterance-level sentiment. Moreover, Akhtar et al.
(2019) presented a deep multi-task learning framework to jointly learn
sentiment polarities and emotional intensity in a multimodal background.
Rahman et al. (2020) directly worked on BERT and designed functional gates to
control the dataflow of one modality from other two modalities.

Pham et al. (2019a) proposed a method that cyclically translates between
modalities to learn robust joint representations for sentiment analysis. More
recently, Hazarika et al. (2020) attempted to factorize modality features in
joint spaces to effectively capture commonalities across different modalities
and reduce their gaps. Tsai et al. (2020) proposed a Capsule Network-based
method to dynamically adjust weights between modalities. Most of these works
incorporate interactions in every modality pairs into their design. In
contrast, our model only includes two pairs involving a common central
modality.

###  2.2. Multimodal Language Learning

#### Correlation-based Approach

Correlation has been learned as an important metric for objects showing
concurrently. There are many previous works that include this item for various
purposes. Sun et al. (2019, 2020) directly optimized over a correlation-
related DCCA loss to learn multimodal representations useful for downstream
tasks. Mittal et al. (2020) instead used correlation as a selection criteria
to guide multimodal data to orderly form a union representation. Although all
these works took correlation into account, they ignored the importance of
modality’s independence and the competition between the two opponents.

#### Alignment-based Approach

Alignment is the process to map signals of different sampling rates to the
same frequency. Early multimodal alignment approaches (Zadeh et al., 2016;
Blikstein and Worsley, 2016) usually firstly chose a target frequency and then
calculated the frames in each modality that need mapping to that position.
Some classical loss functions like CTC (Graves et al., 2006) and their
variants are widely used to facilitate alignment improvement. Thanks to the
advent attention mechanism, the Transformer architecture shows state-of-the-
art performance in multiple disciplines in both text and visual fields
(Vaswani et al., 2017; Dosovitskiy et al., 2020). Unlike traditional alignment
routines, attention naturally formulates a point-to-point mapping between two
modalities, which is called “soft alignment” and has been proven effective in
more general cases of multimodal representation learning and feature fusion.
For example, Yu et al. (2020) designed a Unified Multimodal Transformer to
jointly model text and visual representations in the NER problem. Moreover,
Tsai et al. (2019a) employed the Transformer to model as well as align
sequences from visual, textual, and acoustic sources. Our fusion architecture
is built on Transformer, but performs fusion in a progressive manner with
feature space regularization and fine-grained gate control.

#### Application in Other Tasks

Besides multimodal sentiment analysis, multimodal learning has been applied in
many other language tasks, such as Machine Translation (MT) (Specia et al.,
2016; Elliott, 2018; Yao and Wan, 2020; Huang et al., 2020), Named Entity
Recognition (NER) (Moon et al., 2018; Zhang et al., 2018; Lu et al., 2018; Yu
et al., 2020), and parsing (Shi et al., 2018, 2019; Zhao and Titov, 2020).

##  3\. Methodology

In this section, we first briefly define the problem and then describe our
BBFN model.

#### Task Definition

The task of MSA aims to predict sentiment intensity, polarity, or emotion
label of given multimodal input (video clip). The video consists of three
modalities: $t$ (text), $v$ (visual) and $a$ (acoustic), which are 2D tensors
denoted by $M_{t}\in\mathbb{R}^{T_{t}\times d_{t}}$,
$M_{v}\in\mathbb{R}^{T_{v}\times d_{v}}$, and $M_{a}\in\mathbb{R}^{M_{a}\times
d_{a}}$, where $T_{m}$ and $d_{m}$ represent sequence length (e.g., number of
frames) and feature vector size of modality $m$.

###  3.1. Overall Description

Figure 2. Overview of our Bi-Bimodal Fusion Network (BBFN). It learns two
text-related pairs of representations, TA and TV, by causing each pair of
modalities to complement mutually. Finally, the four (two pairs) head
representations are concatenated to generate the final prediction.

The overview of our model is shown in Fig. 2. It consists of two modality
complementation modules, each accomplishing a bimodal fusion process in its
two fusion pipelines. After receiving context-aware representations from the
underlying modality sequence encoders, bimodal fusion proceeds iteratively
through stacked complementation layers.

The feature space separator is another key idea of our model. Each modality
has its own feature representations. However, in a deep neural network, when
these unique unimodal representations propagate through multiple layers, their
mutual independence can be compromised, i.e., they may not be as separable as
they were initially; we call this feature space collapse. The separability of
the unimodal representations and their mutual independence is necessary for
multimodal fusion; otherwise, one modality can hardly learn something new from
its counterparts through heterogeneous attention on respective hidden
representations. Accordingly, we enforce these representations to preserve
more modality-specific features to prevent them from collapsing into a pair of
vectors with similar distributions.

Finally, the conventional heterogeneous Transformer purely uses a residual
connection to combine attention results and input representations without any
controlled decision made for the acceptance and rejection along the hidden
dimension of these vectors. Instead, we incorporate a gated control mechanism
in the multi-head attention of the Transformer network, which also couples the
feature separator and transformer fusion pipelines.

###  3.2. Modality Sequence Encoder

We encode all modality sequences to guarantee a better fusion outcome in the
subsequent modality complementation modules.

#### Word Embedding

We use the Transformer-based pre-trained model, BERT (Devlin et al., 2019) as
the text encoder. The raw sentence $S=(w_{1},\cdots,w_{n})$ composed of word
indices is firstly concatenated with two special tokens—[CLS] at the head and
[SEP] at the tail and then fed into the encoder to generate contextualized
word embeddings, as the input of text modality
$M_{t}=(m_{0},m_{1},...,m_{n+1})$.

#### Sequence Encoder

The input modality sequences $M_{m}$, $m\in\\{t,v,a\\}$, are essentially time
series and exhibit temporal dependency. We use a single-layer bidirectional
gated recurrent unit (BiGRU) (Chung et al., 2014) followed by a linear
projection layer to capture their internal dependency and cast all the hidden
vectors to the same length for the convenience of further processing. The
resulting sequences are

(1) $\displaystyle X^{0}_{m}=(x_{m,0}^{0},x_{m,1}^{0},...,x_{m,n+1}^{0}),$

where $m\in\\{t,v,a\\}$ denotes a modality. These outputs serve as the initial
inputs to the modality complementation module.

###  3.3. Modality Complementation Module

In the modality complementation module, the modality representation pairs
exchange information with their counterparts to “complement” the missing cues
when passing through the multimodal complementation layers that interconnect
two fusion pipelines with layer-wise modality-specific feature separators. We
further improve the attention-based fusion procedure by adding a gated control
mechanism to enhance its performance and robustness. The module is built in a
stacked manner to realize an iterative fusion routine.

#### Modality-Specific Feature Separator

To maintain mutual independence among these modalities, we leverage the
regularization effect exerted by a discriminator loss, which tells how well
the hidden representations can be distinguished from their counterparts in the
same complementation module. A straightforward solution for a separator
according to prior work (Hazarika et al., 2020; Sun et al., 2020) is to add
some geometric measures to the total loss as regularization term, such as (1)
euclidean distances or cosine correlation or (2) distribution similarity
measures such as KL-Divergence or Wasserstein distance along the hidden
dimension. However, we chose the discriminator loss because—unlike geometric
measures, which directly use hidden vectors—it is a parametric method, so it
is more suitable to be coalesced into the entire model.

Namely, after collecting the outputs from the previous complementation layer
$X_{m}^{i-1}=(x_{m,0}^{i-1},x_{m,1}^{i-1},...,x_{m,n+1}^{i-1})$, we encode the
sequence with a bidirectonal GRU and then apply an average pooling to acquire
sequence-level hidden representations:

$\displaystyle\bar{\mathbf{h}}_{m}^{i}$
$\displaystyle=\operatorname{avgpool}(\mathbf{h}_{m}^{i})=\operatorname{avgpool}(\operatorname{BiGRU}(X_{m}^{i-1};\theta_{m}^{i})).$

where $\theta_{m}^{i}$ are the parameters of the $\operatorname{BiGRU}$ in the
$i$th layer. Here we choose BiGRU as the intermediate sequence encoder because
with fewer parameters, in our experiments it provided results comparable with
those of BiLSTM. Note that until now we just described the data flow of a
single modality. In a complementation module, at each layer $i$ there are
always two pipelines that generate the sequences of hidden representations
concurrently for two different modalities, $m_{1}$ and $m_{2}$.

Next we want to separate the possibly entangled intermediate modality
representations. Different from previous works that rely on explicit distance
maximization, we train a classifier to discern which modality these
representations come from. A straightforward approach is directly fed all of
them into the classifier, but it may occur serious issue: random noises in
sequence representations cause the classifier to pay worthless effort on
trivial features. We introduce a simple group strategy to mitigate this issue,
which applies average operation on representations in the same group to
generate a smoother representation. Specifically, after setting group size as
$K$, the representation for the $r^{th}$ ($r=1,2,...,N/K$) group is:

(2) $\tilde{\textbf{h}}_{m}^{i,r}=\frac{1}{K}\sum_{j=1+(r-1)\times
K}^{K}\bar{\textbf{h}}_{m}^{i,j}$ (3)
$\hat{c}^{i}_{r},\hat{c}^{i}_{r+N_{b}/K}=D_{i}(\tilde{\mathbf{h}}_{m_{1}}^{i,r},\tilde{\mathbf{h}}_{m_{2}}^{i,r}).$

We leave a short explanation about how this reduces noise here. Suppose
vectors possessing similar property (i.e. from the same modality in context)
can be fitted with a set of gaussian distribution
$\\{\mathcal{N}(\mu_{1},\sigma_{1}),\mathcal{N}(\mu_{2},\sigma_{2}),\mathcal{N}(\mu_{3},\sigma_{3}),...,\mathcal{N}(\mu_{n},\sigma_{n})\\}$
and corresponding weights $\\{w_{1},w_{2},w_{3},...,w_{n}\\}$. According to
the rule for the summation of gaussian distributions, we have

$e\sim~{}\mathcal{N}\left(\sum_{n}w_{n}\mu_{n},\sqrt{\sum_{n}w_{n}^{2}\sigma_{n}^{2}}\right)=\mathcal{N}(\mu,\sigma)$

By introducing the grouping trick, for each group representation the new
expectation term holds constant while the variance term turns to

$\sigma_{g}=\frac{\sqrt{\sum_{n}w_{n}^{2}\sigma_{n}^{2}}}{K}=\sigma/K$

which decreases as group size increases.

The discriminators are distinct in every layer because of the diverse
manifestations of the same modality in the semantic space as fusion progresses
which thus requires different sets of parameters to discern. We calculate the
Binary Cross Entropy between predictions and their corresponding pseudo labels
that are automatically generated during training time as the discriminator
loss:

$\mathcal{L}^{i}_{sep}=-\frac{K}{2N_{b}}\sum_{r=1}^{2N_{b}/K}\left(c^{i}_{r}\log{\hat{c}}^{i}_{r}+(1-c^{i}_{r})\log(1-\hat{c}^{i}_{r})\right),$

where $N_{b}$ is the batch size and $j$ represents the $j$-th sample.

#### Gated Complementation Transformer (GCT)

The main body of the modality complementation module is the Gated
Complementation Transformer, which are stacked into two pipelines to form the
symmetric structure. For convenience of explanation, we will call the modality
that keeps forwarding in the same fusion pipeline inside a complementation
module the main modality, denoted by $main$, and the modality that joins
bimodal fusion in one pipeline but comes as an external source from the other
pipeline, the complementary modality, denoted by $comp$. Noted that
distinguishing main and complementary modalities makes sense only in the
context of a specified pipeline.

The cross-modal fusion process occurs mainly at the multi-head attention
operation, which we found to show suboptimal performance due to the lack of
information flow control. To improve it in a fine-grained and controllable
way, we introduce two gates: the retain gate $\mathbf{g}_{r}$, which decides
how much proportion of the target modality’s components to be kept forwarding,
and the compound gate $\mathbf{g}_{c}$, which decides how much proportion of
compounded components to be injected to the target modality.

We generate these two gate signals from the sequential representation of the
two modalities in the same layer:

$\displaystyle\mathbf{g}_{r}^{i}=\sigma(\mathbf{W}^{i,main}_{r}[\bar{\mathbf{h}}_{main}^{i}\|\bar{\mathbf{h}}_{comp}^{i}]),$
$\displaystyle\mathbf{g}_{c}^{i}=\sigma(\mathbf{W}^{i,main}_{c}[\bar{\mathbf{h}}_{main}^{i}\|\bar{\mathbf{h}}_{comp}^{i}]),$

where $\mathbf{W}_{*}\in\mathbb{R}^{2d\times d}$ is the projection matrix and
$\|$ represents concatenation. After receiving the query
$Q^{i}=\mathbf{W}_{Q}^{i}X^{i}_{main}$, key
$K^{i}=\mathbf{W}_{K}^{i}X^{i}_{comp}$ and value
$V^{i}=\mathbf{W}_{V}^{i}X^{i}_{comp}$, these gates are then employed on the
multi-head attention to limit the information flow of the residual block as a
part of bimodal combination:

$\displaystyle\textbf{m}^{i}$ $\displaystyle=\text{MH-
ATT}(Q^{i},K^{i},V^{i}),$ $\displaystyle\tilde{X}^{i}_{main}$
$\displaystyle=\text{LN}(\mathbf{g}_{c}^{i}\odot\mathbf{m}^{i}+\mathbf{g}_{r}^{i}\odot
X^{i}_{main}),$

where MH-ATT represents multi-head attention, $\odot$ means component-wise
multiplication and LN is layer normalization. Next, the attention results pass
through the feed-forward network (similar to a conventional Transformer
network) to produce the final output of the current complementation layer:

(4)
$X^{i}_{main}=\text{LN}(\tilde{X}^{i}_{main}+\text{FFN}(\tilde{X}^{i}_{main})).$

Figure 3. A single complementation layer: two identical pipelines (left and
right) propagate the main modality and fuse that with complementary modality
with regularization and gated control.

###  3.4. Output Layer and Training

According to (4), a given complementation layer produces the output
$X^{i}_{m}$, where, similarly to (1),
$X^{i}_{m}=(x^{i}_{m,0},\dots,x^{i}_{m,n})$. When speaking of a layer of a
specific complementation module $M$, where $M\in\\{TA,TV,VA\\}$, we will add
$M$ as index: $X^{i}_{M,m}=(x^{i}_{M,m,0},\dots,x^{i}_{M,m,n+1})$. The final
output of the module $M$ for the modality $m$ is $X^{L}_{M,m}$, where $L$ is
the number of layers; see Fig. 2.

We compute the final representation by first extracting the heads
$\mathbf{h}^{L}_{M,m}=x_{M,m,0}^{L}$ from the outputs of the last layer in
each module and then concatenating them. We have also tried other methods such
as average pooling and LSTM or GRU and found them producing similar results,
so we chose the most computationally efficient one. As there are two heads in
each of the two multimodal complementation layers, concatenating all the
outputs of these four heads in total gives the final representation
$\mathbf{h}_{\text{final}}\in\mathbb{R}^{4d}$, where the dimension of each
head output is $d$. Finally, the representation vector is fed to a feed-
forward network to produce the final prediction $\hat{y}$.

The loss function comprises two parts: the task loss and the sum of all
separators’ classification loss. In the case of sentiment intensity
prediction, the task loss is the mean squared error (MSE), since it is a
regression problem. In the case of humor detection, we used binary cross-
entropy (BCE) loss to facilitate the training for this binary classification
task. Separator loss is a layer-wise loss so we sum up the results that are
computed in each layer and add them to the total loss. The total loss is
calculated as

$\begin{split}\mathcal{L}&=\frac{1}{N_{b}}\sum_{j=1}^{N_{b}}\left(\tau(y_{j},\hat{y}_{j})+\frac{\lambda
K}{2L}\sum_{i=1}^{L}\sum_{M}\mathcal{L}_{sep}^{M,i}\right),\end{split}$

where $\tau$ denotes the task loss and $\lambda$ is a tunable parameter to
control the power of regularization.

##  4\. Experiments

###  4.1. Datasets

We evaluated our BBFN model on two tasks: sentiment intensity prediction and
humor detection, with three datasets involved.

  * •

CMU-MOSI. The CMU-MOSI dataset (Zadeh et al., 2016) is a prevalent benchmark
for evaluating fusion networks’ performance on the sentiment intensity
prediction task. The dataset is composed of many YouTube video blogs or vlogs,
in which a speaker narrates their opinions on some topic. It contains 2,199
utterance-video segments sliced from 93 videos played by 89 distinct
narrators. Each segment is manually annotated with a real number score ranged
from $-3$ to $+3$, indicating the relative strength of negative (score below
zero) or positive (score above zero) emotion.

  * •

CMU-MOSEI. The CMU-MOSEI dataset (Zadeh et al., 2018a) is an upgraded version
of CMU-MOSI concerning the number of samples. It is also enriched in terms of
the versatility of speakers and covers a broader scope of topics. The dataset
contains 23,453 video segments, which are annotated in the same way as CMU-
MOSI. These segments are extracted from 5,000 videos involving 1,000 distinct
speakers and 250 different topics.

  * •

UR-FUNNY. UR-FUNNY (Hasan et al., 2019) is a popular humor detection dataset,
as our test benchmark. This dataset contains 16,514 multimodal punchlines
sampled from the TED talks. Each sample is annotated with an equal number of
binary labels indicating if the protagonist in a video expresses a sort of
humor.

###  4.2. Preprocessing

To produce machine-understandable inputs for our model and ensure fair
competition with other baselines, following many previous works we process
data from the three modalities into typical tensors as introduced below.

#### Text Modality

Many previous works adopted (Pennington et al., 2014) as word embedding
sources. But recent works including current SOTA preferred advanced pretrained
language models. Therefore as we stated before in Section 3.2 to encode input
raw text in all experiments.

#### Visual Modality

Specifically, for experiments on CMU-MOSI and CMU-MOSEI, we use Facet, an
analytical tool built on the Facial Action Coding Systems (FACS) (Ekman, 1997)
to extracted facial features. For experiments on UR-FUNNY we use another
facial behavioral analysis tool, Openface (Baltrušaitis et al., 2016) to
capture facial gesture variance of every speaker. The resulting vector lengths
for the three datasets (MOSI, MOSEI and UR-FUNNY) are 47, 35 and 75
respectively.

#### Acoustic Modality

Acoustic features were extracted with COVAREP (Degottex et al., 2014), a
professional acoustic analysis framework.

#### Modality Alignment

The input signals in our experiments were word-aligned. Following many
previous works (Tsai et al., 2019a; Pham et al., 2019a; Zadeh et al., 2018b),
we used P2FA (Yuan and Liberman, 2008) to align visual and acoustic signals to
the same resolution of text. The tool automatically separates numerous frames
into several groups and match each group with a token by averaging their
representation vectors to a new one. We used BERT-base-uncased as the text
embedding source for all models in all experiments.

###  4.3. Baselines and Metrics

We compared our results with several advanced multimodal fusion frameworks:

  * •

DFF-ATMF (Chen et al., 2019): It is the first bimodal model which first learns
individual modality features then executes attention-based modality fusion.

  * •

Low-rank Matrix Fusion (LMF) (Liu et al., 2018): It decomposes high-order
tensors into many low-rank factors then performs efficient fusion based on
these factors.

  * •

Tensor Fusion Network (TFN) (Zadeh et al., 2017): This approach models intra-
modality and inter-modality dynamics concurrently with local feature
extraction network and 3-fold Cartesian product.

  * •

Multimodal Factorization Model (MFM) (Tsai et al., 2019b): To enhance the
robustness of the model of capturing intra- and inter-modality dynamics, MFM
is a cycle style generative-discriminative model.

  * •

Interaction Canonical Correlation Network (ICCN) (Sun et al., 2020):
Correlation between modalities is a latent trend to be excavated under the
fusion process. ICCN purely relies on mathematical measure to accomplish the
fusion process.

  * •

MulT (Tsai et al., 2019a): To alleviate the drawback of hard temporal
alignment for multimodal signals, MulT utilizes stacked transformer networks
to perform soft alignment to extend the number of positions on the time axis
that each frame of signal can interact with.

  * •

MISA (Hazarika et al., 2020): Motivated by previous work in domain separation
task, this work regards signals from different modalities as data in different
domains and similarly constructs two kinds of feature spaces to finish the
fusion process.

We used five different metrics to evaluate the performance on CMU-MOSI and
CMU-MOSEI: mean absolute error (MAE), which directly calculates the error
between predictions and real-number labels; seven-class accuracy (Acc-7),
positive/negative (excluding zero labels) accuracy (Acc-2) and F1 score, which
coarsely estimate the model’s performance by comparing with quantified values;
and Pearson correlation (Corr) with human-annotated truth, which measures
standard deviation. As for humor recognition on UR-FUNNY, it is a binary
classification problem and we only report the binary classification accuracy
(Acc-2).

##  5\. Results and Analysis

###  5.1. Summary of Results

We list the results with baselines on all the three datasets in Table 1. From
these results, it can be found that our BBFN outperforms other models on
almost all metrics (except for the correlation coefficient on CMU-MOSI). We
attained an improvement of around 1% over the state-of-the-art approaches in
terms of binary classification accuracy and more than 2.5% in terms of 7-class
accuracy. We attribute the difference partly to the insensitivity of coarse
metrics to the variation in the model’s predictions. The best performance
boost, more than 4%, was on the MAE of CMU-MOSEI.

To better illustrate how BBFN beats the SOTA (MISA), we compute the absolute
errors of all the predictions on the test set of CMU-MOSEI and paint their
distributions in Fig. 4. It can be observed that in the low error part
(error$<0.25$) the curve of BBFN has more peaks than MISA, which demonstrates
the higher precision that our model can reach.

Figure 4. Distribution of absolute error when testing BBFN and MISA on CMU-
MOSEI dataset.

Models | CMU-MOSI | CMU-MOSEI | UR-FUNNY
---|---|---|---
MAE | Corr | Acc-7 | Acc-2 | F1 | MAE | Corr | Acc-7 | Acc-2 | F1 | Acc-2
$\text{DFF-ATMF}^{\triangle}$ | - | - | - | 80.9 | 81.2 | - | - | - | 77.1 | 78.3 | -
$\text{LMF}^{\triangle}$ | 0.917 | 0.695 | 33.2 | 82.5 | 82.4 | 0.623 | 0.677 | 48.0 | 82.0 | 82.1 | 67.53
$\text{TFN}^{\triangle}$ | 0.901 | 0.698 | 34.9 | 80.8 | 80.7 | 0.593 | 0.700 | 50.2 | 82.5 | 82.1 | 68.57
$\text{MFM}^{\triangle}$ | 0.877 | 0.706 | 35.4 | 81.7 | 81.6 | 0.568 | 0.717 | 51.3 | 84.4 | 84.3 | -
$\text{ICCN}^{\triangle}$ | 0.862 | 0.714 | 39.0 | 83.0 | 83.0 | 0.565 | 0.713 | 51.6 | 84.2 | 84.2 | -
$\text{MulT}^{{\dagger}}$ | 0.832 | 0.745 | 40.1 | 83.3 | 82.9 | 0.570 | 0.758 | 51.1 | 84.5 | 84.5 | 70.55
$\text{MISA}^{{\dagger}}$ | 0.817 | 0.748 | 41.4 | 82.1 | 82 | 0.557 | 0.748 | 51.7 | 84.9 | 84.8 | 70.61
$\text{BBFN}^{{\ddagger}}$ (Ours) | 0.776 | 0.755 | 45.0 | 84.3 | 84.3 | 0.529 | 0.767 | 54.8 | 86.2 | 86.1 | 71.68

Table 1. Results on the test set of CMU-MOSI and CMU-MOSEI dataset. Notation:
$\triangle$ indicates results in the corresponding line are excerpted from
previous papers; ${\dagger}$ means the results are reproduced with publicly
visible source code and applicable hyperparameter setting; ${\ddagger}$ shows
the results have experienced paired t-test with $p<0.05$ and demonstrate
significant improvement over MISA, the state-of-the-art model.

###  5.2. Ablation Study

To examine the functionality of the overall architecture and the components
introduced in this work, we conducted an ablation study on CMU-MOSEI dataset;
see 3.

Case | Input by Modality |  Prediction & Truth
---|---|---
Text | Visual | Acoustic | Prediction | Truth | ABS Error
A |  But, I mean, if you’re going to watch a movie like that, go see Saw again or something, because this movie is really not good at all. |  Widely opened eyes |  Pause and Stress | $-1.973$ | $-2.000$ | $0.027$
B |  Plot to it than that the action scenes were my favorite parts though it’s. |  Smiling face Relaxed wink |  Stress Pitch variation | $+1.638$ | $+1.666$ | $0.028$
C |  (umm) So if you’re looking for something it’s sort of lighthearted. |  No expression |  Normal Voice Peaceful tone | $-0.016$ | $0.000$ | $0.016$
Table 2. Input and predictions of two samples in our case study.

In five experiments we verified the effect of the bimodal fusion architecture.
Specifically, we (1) only used one pair as input; (2) replaced input text-
related modality pairs (TV, TA) with visual/acoustic-related ones; and (3)
added a visual-acoustic complementation module to make a ternary symmetric
model. In all cases, separators and gates were used.

The models of type (2) outperformed those of type (1) on MAE and Acc-7 (the
most accurate measures), which indicates that all three modalities are
important. Moreover, the performance of visual-focused input (TV+VA) is close
to that of text-focused input (our TV+TA), i.e., our architecture can operate
on these modality pairs, too.

On the other hand, the performance degrades on type (3), when visual-acoustic
input pairs are added. That is, even after including all modalities in the
input, redundant network architecture can cause harmful effects bringing in
malicious noise, which damages collected useful information and confuses the
model.

We also explored the benefits of the feature-space separator and gated control
by removing the separator, the two gates, or both from our BBFN. The outcome
shows some degradation in all metrics except the correlation. This proves that
including gated control and modality separator improves the model’s
performance, though the greatest improvement over the baselines shown in Table
3 comes from our overall bi-bimodal architecture.

Figure 5. Visualization of eight gated control signals in the second layer of
our BBFN for two case study samples. “XY-X/Y-c/r” denotes the compound/retain
gate in the transformer pipeline for X/Y modality in XY complementation
module.

###  5.3. Further Analysis

To study how the gates affect the information flow, we visualized the weights
in all the gates per dimension; see Fig. 5. We hypothesize that the
discrepancy in weight distributions reflects the relative importance of
modalities. Specifically, for the two gates associated with one modality in
the same module, if the weights in the compound gate are greater than those in
the retain gate, it implies that the model enforces the corresponding modality
to learn much from its counterpart in the module and the modality thus is less
important. Conversely, if the weights of the retain gate are greater, then the
modality is more important than its counterpart. Fig. 2 shows three typical
samples, including raw data input (for visual and acoustic we only give short
descriptions), predictions and truths from the test set. In case A, most cues
are attained from the text modality to express sort of disappointment, while
data from the visual and acoustic modalities are not so informative. Hence for
the acoustic and visual, the model makes the corresponding modalities A and V
pay more attention to the heterogeneous attention results instead of
themselves, which is indicated by the larger weights in the compound gates of
the two modalities (TA-A-c & TV-V-c).

In case B, the V and A modalities are seen to be providing key information
along with T. Therefore, the weight difference is indiscernible, and two paths
of information flow achieve a balance. Surprisingly, Fig. 5 shows that for the
text modality in the TV complementation module, the weights in the compound
gate are slightly higher than those in the retain gate. This implies that the
textual modality can be complemented by the information attained from the
visual modality.

In case C, no single modality can provide clear evidence for the neutral
sentiment, but each of them serves as a favorable supplementary to others.
Therefore, in Fig. 5 we find the weights in the gates of each modality are
comparable, indicating the similar dependency of bimodal fusion results on
both modalities.

We also compared our BBFN with MISA for the final prediction in the two cases.
As shown in Table 2, BBFN’s outputs are closer to the ground truth, owing to
the fine-grained control offered by these gates, whereas MISA makes opposite
(case A) or conservative (case B) predictions because, as a result of ternary-
symmetric architectures, it is distracted by insignificant modalities, which
add disturbing factors and dilute the pertinent features.

Description | MAE | Corr | Acc-7 | Acc-2 | F1
---|---|---|---|---|---
TV only | 0.546 | 0.761 | 51.8 | 85.6 | 85.6
TA only | 0.548 | 0.759 | 51.7 | 85.5 | 85.5
VA only | 0.816 | 0.261 | 41.1 | 71.1 | 64.5
VA+TA | 0.533 | 0.773 | 54.1 | 84.8 | 84.9
TV+VA | 0.531 | 0.775 | 54.5 | 85.7 | 85.7
TV+TA (BBFN) | 0.529 | 0.767 | 54.8 | 86.2 | 86.1
w/o separator | 0.533 | 0.766 | 54.1 | 85.7 | 85.4
w/o gates | 0.531 | 0.768 | 53.9 | 85.8 | 85.7
w/o both | 0.540 | 0.763 | 53.0 | 85.1 | 85.0
TV+TA+VA | 0.547 | 0.768 | 52.8 | 84.3 | 84.4

Table 3. An ablation study of BBFN’s architecture and functional components on
the test set of CMU-MOSEI.

##  6\. Conclusion

We have presented Bi-Bimodal Fusion Network (BBFN), a fusion architecture for
multimodal sentiment analysis that focuses on bimodal fusion process. Pairwise
fusion process proceeds progressively through stacked complementation layers
in each learning module. To alleviate the issue of feature space collapse and
lack of control at fusion time, we introduced into our model the structure of
modality-specific feature space separator and gated control mechanism,
respectively. Comprehensive experiments and analysis show that our model
outperforms the current state-of-the-art approaches. Despite good performance
of our model, we plan to explore more advanced fusion methods and
architectures. Also besides sentiment analysis, in multimodal research there
are many other important tasks, for which we can combine task-specific
techniques with appropriate fusion schemes. Accordingly, we plan to improve
the fusion quality of multimodal data as well as the coordination of fusion
and task solving modules.

## Acknowledgements

This material is based upon work partially supported by the SUTD SRG grant
#T1SRIS19149, National Science Foundation (Awards #1722822 and #1750439), and
National Institutes of Health (Awards #R01MH125740, #R01MH096951,
#U01MH116925, #U01MH116923). Any opinions, findings, conclusions, or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the National Science Foundation or
National Institutes of Health, and no official endorsement should be inferred.

## References

  * (1)
  * Akhtar et al. (2019) Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019.  Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 370–379.  https://www.aclweb.org/anthology/N19-1034.pdf
  * Atrey et al. (2010) Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. 2010.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ 16, 6 (2010), 345–379.
  * Baltrušaitis et al. (2016) Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016.  Openface: an open source facial behavior analysis toolkit. In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1–10.
  * Blikstein and Worsley (2016) Paulo Blikstein and Marcelo Worsley. 2016.  Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks.  _Journal of Learning Analytics_ 3, 2 (2016), 220–238.
  * Chen et al. (2019) Feiyang Chen, Ziqian Luo, Yanyan Xu, and Dengfeng Ke. 2019\.  Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis.  _arXiv preprint arXiv:1904.08138_ (2019).  http://ceur-ws.org/Vol-2614/AffCon20_session1_complementary.pdf
  * Chen et al. (2017) Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017.  Multimodal sentiment analysis with word-level fusion and reinforcement learning. In _Proceedings of the 19th ACM International Conference on Multimodal Interaction_. 163–171.  https://dl.acm.org/doi/pdf/10.1145/3136755.3136801
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling. In _NIPS 2014 Workshop on Deep Learning, December 2014_.  https://arxiv.org/pdf/1412.3555.pdf
  * Degottex et al. (2014) Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014.  COVAREP—A collaborative voice analysis repository for speech technologies. In _2014 ieee international conference on acoustics, speech and signal processing (icassp)_. IEEE, 960–964.  https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6853739
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.  https://www.aclweb.org/anthology/N19-1423.pdf
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).  https://arxiv.org/pdf/2010.11929.pdf
  * Ekman (1997) Rosenberg Ekman. 1997\.  _What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS)_.  Oxford University Press, USA.
  * Elliott (2018) Desmond Elliott. 2018\.  Adversarial evaluation of multimodal machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 2974–2978.  https://www.aclweb.org/anthology/D18-1329.pdf
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In _Proceedings of the 23rd international conference on Machine learning_. 369–376.
  * Gu et al. (2018) Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018\.  Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2225–2235.  https://www.aclweb.org/anthology/P18-1207.pdf
  * Hasan et al. (2019) Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019.  UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 2046–2056.  https://www.aclweb.org/anthology/D19-1211.pdf
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020\.  MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis. In _Proceedings of the 28th ACM International Conference on Multimedia_. 1122–1131.  https://dl.acm.org/doi/pdf/10.1145/3394171.3413678
  * Huang et al. (2020) Po-Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann. 2020.  Unsupervised multimodal neural machine translation with pseudo visual pivoting.  _arXiv preprint arXiv:2005.03119_ (2020).
  * Liu et al. (2018) Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018\.  Efficient Low-rank Multimodal Fusion With Modality-Specific Factors. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2247–2256.  https://www.aclweb.org/anthology/P18-1209.pdf
  * Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018.  Visual attention model for name tagging in multimodal social media. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1990–1999.  https://www.aclweb.org/anthology/P18-1185.pdf
  * Mittal et al. (2020) Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020\.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 1359–1367.
  * Moon et al. (2018) Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018\.  Multimodal Named Entity Recognition for Short Social Media Posts. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. 852–860.  https://www.aclweb.org/anthology/N18-1078.pdf
  * Morency et al. (2011) Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011\.  Towards multimodal sentiment analysis: Harvesting opinions from the web. In _Proceedings of the 13th international conference on multimodal interfaces_. 169–176.  https://dl.acm.org/doi/pdf/10.1145/2070481.2070509
  * Ngiam et al. (2011) Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011\.  Multimodal deep learning. In _International Conference on Machine Learning (ICML)_. 689–696.  https://icml.cc/2011/papers/399_icmlpaper.pdf
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Pham et al. (2019a) Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019a.  Found in translation: Learning robust joint representations by cyclic translations between modalities. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 6892–6899.  https://arxiv.org/pdf/1812.07809.pdf
  * Pham et al. (2019b) Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alex Waibel. 2019b.  Improving Zero-shot Translation with Language-Independent Constraints.  _arXiv preprint arXiv:1906.08584_ (2019).
  * Rahman et al. (2020) Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020.  Integrating multimodal information in large pretrained transformers. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , Vol. 2020. NIH Public Access, 2359.
  * Shi et al. (2019) Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019\.  Visually Grounded Neural Syntax Acquisition. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 1842–1861.  https://www.aclweb.org/anthology/P19-1180.pdf
  * Shi et al. (2018) Haoyue Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang, and Jian Sun. 2018.  Learning Visually-Grounded Semantics from Contrastive Adversarial Samples. In _Proceedings of the 27th International Conference on Computational Linguistics_. 3715–3727.  https://www.aclweb.org/anthology/C18-1315.pdf
  * Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016.  A shared task on multimodal machine translation and crosslingual image description. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_. 543–553.  https://www.aclweb.org/anthology/W16-2346.pdf
  * Srivastava and Salakhutdinov (2014) Nitish Srivastava and Ruslan Salakhutdinov. 2014.  Multimodal learning with deep Boltzmann machines.  _The Journal of Machine Learning Research_ 15, 1 (2014), 2949–2980.  https://www.jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf
  * Sun et al. (2020) Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. 2020.  Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 8992–8999.  Issue 5. https://ojs.aaai.org//index.php/AAAI/article/view/6431
  * Sun et al. (2019) Zhongkai Sun, Prathusha K Sarma, William Sethares, and Erik P Bucy. 2019.  Multi-modal sentiment analysis using deep canonical correlation analysis.  _arXiv preprint arXiv:1907.08696_ (2019).
  * Tsai et al. (2019a) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019a.  Multimodal Transformer for Unaligned Multimodal Language Sequences. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 6558–6569.  https://www.aclweb.org/anthology/P19-1656.pdf
  * Tsai et al. (2019b) Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b.  Learning Factorized Multimodal Representations. In _International Conference on Representation Learning_.  https://openreview.net/pdf?id=rygqqsA9KX
  * Tsai et al. (2020) Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1823–1833.  https://www.aclweb.org/anthology/2020.emnlp-main.143.pdf
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.  https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
  * Wang et al. (2017) Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2017.  Select-additive learning: Improving generalization in multimodal sentiment analysis. In _2017 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 949–954.  https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8019301
  * Wöllmer et al. (2013) Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013\.  YouTube movie reviews: Sentiment analysis in an audio-visual context.  _IEEE Intelligent Systems_ 28, 3 (2013), 46–53.  https://ieeexplore.ieee.org/iel7/9670/5196652/06487473.pdf
  * Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020.  Multimodal transformer for multimodal machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 4346–4350.
  * Yu et al. (2020) Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020\.  Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. Association for Computational Linguistics.
  * Yuan and Liberman (2008) Jiahong Yuan and Mark Liberman. 2008.  Speaker identification on the SCOTUS corpus.  _The Journal of the Acoustical Society of America_ 123, 5 (2008), 3878–3878.  https://www.ling.upenn.edu/~jiahong/publications/c09.pdf
  * Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017.  Tensor Fusion Network for Multimodal Sentiment Analysis. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. 1103–1114.  https://www.aclweb.org/anthology/D17-1115.pdf
  * Zadeh et al. (2018b) Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018b.  Multi-attention recurrent network for human communication comprehension. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.  Issue 1. https://arxiv.org/pdf/1802.00923.pdf
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ 31, 6 (2016), 82–88.  https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7742221
  * Zadeh et al. (2018a) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2236–2246.  https://www.aclweb.org/anthology/P18-1208.pdf
  * Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018\.  Adaptive co-attention network for named entity recognition in tweets. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 32.  Issue 1. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16432/16127
  * Zhao and Titov (2020) Yanpeng Zhao and Ivan Titov. 2020.  Visually Grounded Compound PCFGs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 4369–4379.  https://www.aclweb.org/anthology/2020.emnlp-main.354.pdf