# PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with
Application in Restaurant Search and Booking

Matthew Henderson Ivan Vulić  Iñigo Casanueva  Paweł Budzianowski  Daniela
Gerz
Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su

PolyAI Limited  London  UK
poly-ai.com

###### Abstract

We present PolyResponse, a conversational search engine that supports task-
oriented dialogue. It is a retrieval-based approach that bypasses the complex
multi-component design of traditional task-oriented dialogue systems and the
use of explicit semantics in the form of task-specific ontologies. The
PolyResponse engine is trained on hundreds of millions of examples extracted
from real conversations: it learns what responses are appropriate in different
conversational contexts. It then ranks a large index of text and visual
responses according to their similarity to the given context, and narrows down
the list of relevant entities during the multi-turn conversation. We introduce
a restaurant search and booking system powered by the PolyResponse engine,
currently available in 8 different languages.

##  1 Introduction and Background

Task-oriented dialogue systems are primarily designed to search and interact
with large databases which contain information pertaining to a certain
dialogue domain: the main purpose of such systems is to assist the users in
accomplishing a well-defined task such as flight booking El Asri et al.
(2017), tourist information Henderson et al. (2014), restaurant search
Williams (2012), or booking a taxi Budzianowski et al. (2018). These systems
are typically constructed around rigid task-specific ontologies Henderson et
al. (2014); Mrkšić et al. (2015) which enumerate the constraints the users can
express using a collection of slots (e.g., price range for restaurant search)
and their slot values (e.g., cheap, expensive for the aforementioned slots).
Conversations are then modelled as a sequence of actions that constrain slots
to particular values. This explicit semantic space is manually engineered by
the system designer. It serves as the output of the natural language
understanding component as well as the input to the language generation
component both in traditional modular systems Young (2010); Eric et al. (2017)
and in more recent end-to-end task-oriented dialogue systems (Wen et al.,
2017; Li et al., 2017; Bordes et al., 2017; Budzianowski et al., 2018, inter
alia).

Working with such explicit semantics for task-oriented dialogue systems poses
several critical challenges on top of the manual time-consuming domain
ontology design. First, it is difficult to collect domain-specific data
labelled with explicit semantic representations. As a consequence, despite
recent data collection efforts to enable training of task-oriented systems
across multiple domains El Asri et al. (2017); Budzianowski et al. (2018),
annotated datasets are still few and far between, as well as limited in size
and the number of domains covered.111For instance, the recently published
MultiWOZ dataset Budzianowski et al. (2018) contains a total of 115,424
dialogue turns scattered over 7 target domains. Other standard task-based
datasets are typically single-domain and by several orders of magnitude
smaller: DSTC2 Henderson et al. (2014) contains 23,354 turns, Frames El Asri
et al. (2017) 19,986 turns, and M2M Shah et al. (2018) spans 14,796 turns. On
the other hand, the Reddit corpus which supports our system comprises 3.7B
comments spanning a multitude of topics, divided into 256M (Reddit)
conversational threads and generating 727M context-reply pairs. Second, the
current approach constrains the types of dialogue the system can support,
resulting in artificial conversations, and breakdowns when the user does not
understand what the system can and cannot support. In other words, training a
task-based dialogue system for voice-controlled search in a new domain always
implies the complex, expensive, and time-consuming process of collecting and
annotating sufficient amounts of in-domain dialogue data.

In this paper, we present a demo system based on an alternative approach to
task-oriented dialogue. Relying on non-generative response retrieval we
describe the PolyResponse conversational search engine and its application in
the task of restaurant search and booking. The engine is trained on hundreds
of millions of real conversations from a general domain (i.e., Reddit), using
an implicit representation of semantics that directly optimizes the task at
hand. It learns what responses are appropriate in different conversational
contexts, and consequently ranks a large pool of responses according to their
relevance to the current user utterance and previous dialogue history (i.e.,
dialogue context).

The technical aspects of the underlying conversational search engine are
explained in detail in our recent work Henderson et al. (2019b), while the
details concerning the Reddit training data are also available in another
recent publication Henderson et al. (2019a). In this demo, we put focus on the
actual practical usefulness of the search engine by demonstrating its
potential in the task of restaurant search, and extending it to deal with
multi-modal data. We describe a PolyReponse system that assists the users in
finding a relevant restaurant according to their preference, and then
additionally helps them to make a booking in the selected restaurant. Due to
its retrieval-based design, with the PolyResponse engine there is no need to
engineer a structured ontology, or to solve the difficult task of general
language generation. This design also bypasses the construction of dedicated
decision-making policy modules. The large ranking model already encapsulates a
lot of knowledge about natural language and conversational flow.

Since retrieved system responses are presented visually to the user, the
PolyResponse restaurant search engine is able to combine text responses with
relevant visual information (e.g., photos from social media associated with
the current restaurant and related to the user utterance), effectively
yielding a multi-modal response. This setup of using voice as input, and
responding visually is becoming more and more prevalent with the rise of smart
screens like Echo Show and even mixed reality. Finally, the PolyResponse
restaurant search engine is multilingual: it is currently deployed in 8
languages enabling search over restaurants in 8 cities around the world.
System snapshots in four different languages are presented in Figure 2, while
screencast videos that illustrate the dialogue flow with the PolyResponse
engine are available at: https://tinyurl.com/y3evkcfz.

##  2 PolyResponse: Conversational Search

Figure 1: The PolyResponse ranking model: it encodes conversational contexts,
replies, and photos to respective vectors $h_{c}$, $h_{r}$, and $h_{p}$.

The PolyResponse system is powered by a single large conversational search
engine, trained on a large amount of conversational and image data, as shown
in Figure 1. In simple words, it is a ranking model that learns to score
conversational replies and images in a given conversational context. The
highest-scoring responses are then retrieved as system outputs. The system
computes two sets of similarity scores: 1) $S(r,c)$ is the score of a
candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the
score of a candidate photo $p$ given a conversational context $c$. These
scores are computed as a scaled cosine similarity of a vector that represents
the context ($h_{c}$), and a vector that represents the candidate response: a
text reply ($h_{r}$) or a photo ($h_{p}$). For instance, $S(r,c)$ is computed
as $S(r,c)=Ccos(h_{r},h_{c})$, where $C$ is a learned constant. The part of
the model dealing with text input (i.e., obtaining the encodings $h_{c}$ and
$h_{r}$) follows the architecture introduced recently by Henderson et al.
(2019b). We provide only a brief recap here; see the original paper for
further details.

### Text Representation.

The model, implemented as a deep neural network, learns to respond by training
on hundreds of millions context-reply $(c,r)$ pairs. First, similar to
Henderson et al. (2017), raw text from both $c$ and $r$ is converted to
unigrams and bigrams. All input text is first lower-cased and tokenised,
numbers with 5 or more digits get their digits replaced by a wildcard symbol
#, while words longer than 16 characters are replaced by a wildcard token
LONGWORD. Sentence boundary tokens are added to each sentence. The vocabulary
consists of the unigrams that occur at least 10 times in a random 10M subset
of the Reddit training set (see Figure 1) plus the 200K most frequent bigrams
in the same random subset.

During training, we obtain $d$-dimensional feature representations ($d=320$)
shared between contexts and replies for each unigram and bigram jointly with
other neural net parameters.222The model deals with out-of-vocabulary unigrams
and bigrams by assigning a random id from 0 to 50,000 to each; this is then
used to look up their embedding. A state-of-the-art architecture based on
transformers Vaswani et al. (2017) is then applied to unigram and bigram
vectors separately, which are then averaged to form the final
$320$-dimensional encoding. That encoding is then passed through three fully-
connected non-linear hidden layers of dimensionality $1,024$. The final layer
is linear and maps the text into the final $l$-dimensional ($l=512$)
representation: $h_{c}$ and $h_{r}$. Other standard and more sophisticated
encoder models can also be used to provide final encodings $h_{c}$ and
$h_{r}$, but the current architecture shows a good trade-off between speed and
efficacy with strong and robust performance in our empirical evaluations on
the response retrieval task using Reddit Al-Rfou et al. (2016), OpenSubtitles
Lison and Tiedemann (2016), and AmazonQA Wan and McAuley (2016) conversational
test data, see Henderson et al. (2019a) for further details.333The comparisons
of performance in the response retrieval task are also available online at:
https://github.com/PolyAI-LDN/conversational-datasets/.

In training the constant $C$ is constrained to lie between 0 and
$\sqrt{l}$.444It is initialised to a random value between 0.5 and 1, and
invariably converges to $\sqrt{l}$ by the end of training. Empirically, this
helps with learning. Following Henderson et al. (2017), the scoring function
in the training objective aims to maximise the similarity score of context-
reply pairs that go together, while minimising the score of random pairings:
negative examples. Training proceeds via SGD with batches comprising 500 pairs
(1 positive and 499 negatives).

### Photo Representation.

Photos are represented using convolutional neural net (CNN) models pretrained
on ImageNet Deng et al. (2009). We use a MobileNet model with a depth
multiplier of 1.4, and an input dimension of $224\times 224$ pixels as in
Howard et al. (2017).555The pretrained model downloaded from TensorFlow Slim.
This provides a $1,280\times 1.4=1,792$-dimensional representation of a photo,
which is then passed through a single hidden layer of dimensionality $1,024$
with ReLU activation, before being passed to a hidden layer of dimensionality
512 with no activation to provide the final representation $h_{p}$.

### Data Source 1: Reddit.

For training text representations we use a Reddit dataset similar to Al-Rfou
et al. (2016). Our dataset is large and provides natural conversational
structure: all Reddit data from January 2015 to December 2018, available as a
public BigQuery dataset, span almost 3.7B comments Henderson et al. (2019a).
We preprocess the dataset to remove uninformative and long comments by
retaining only sentences containing more than 8 and less than 128 word tokens.
After pairing all comments/contexts $c$ with their replies $r$, we obtain more
than 727M context-reply $(c,r)$ pairs for training, see Figure 1.

### Data Source 2: Yelp.

Once the text encoding sub-networks are trained, a photo encoder is learned on
top of a pretrained MobileNet CNN, using data taken from the Yelp Open
dataset:666https://www.yelp.com/dataset it contains around 200K photos and
their captions. Training of the multi-modal sub-network then maximises the
similarity of captions encoded with the response encoder $h_{r}$ to the photo
representation $h_{p}$. As a result, we can compute the score of a photo given
a context using the cosine similarity of the respective vectors. A photo will
be scored highly if it looks like its caption would be a good response to the
current context.777Note that not all of the Yelp dataset has captions, which
is why we need to learn the photo representation. If a photo caption is
available, then the response vector representation of the caption is averaged
with the photo vector representation to compute the score. If a caption is not
available at inference time, we use only the photo vector representation.

### Index of Responses.

The Yelp dataset is used at inference time to provide text and photo
candidates to display to the user at each step in the conversation. Our
restaurant search is currently deployed separately for each city, and we limit
the responses to a given city. For instance, for our English system for
Edinburgh we work with 396 restaurants, 4,225 photos (these include additional
photos obtained using the Google Places API without captions), 6,725 responses
created from the structured information about restaurants that Yelp provides,
converted using simple templates to sentences of the form such as “Restaurant
X accepts credit cards.”, 125,830 sentences extracted from online reviews.

### PolyResponse in a Nutshell.

The system jointly trains two encoding functions (with shared word embeddings)
$f(context)$ and $g(reply)$ which produce encodings $h_{c}$ and $h_{r}$, so
that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit
training data and low for random pairs. The encoding function $g()$ is then
frozen, and an encoding function $t(photo)$ is learnt which makes the
similarity between a photo and its associated caption high for all (photo,
caption) pairs from the Yelp dataset, and low for random pairs. $t$ is a CNN
pretrained on ImageNet, with a shallow one-layer DNN on top. Given a new
context/query, we then provide its encoding $h_{c}$ by applying $f()$, and
find plausible text replies and photo responses according to functions $g()$
and $t()$, respectively. These should be responses that look like answers to
the query, and photos that look like they would have captions that would be
answers to the provided query.

At inference, finding relevant candidates given a context reduces to computing
$h_{c}$ for the context $c$ , and finding nearby $h_{r}$ and $h_{p}$ vectors.
The response vectors can all be pre-computed, and the nearest neighbour search
can be further optimised using standard libraries such as Faiss Johnson et al.
(2017) or approximate nearest neighbour retrieval Malkov and Yashunin (2016),
giving an efficient search that scales to billions of candidate responses.

The system offers both voice and text input and output. Speech-to-text and
text-to-speech conversion in the PolyResponse system is currently supported by
the off-the-shelf Google Cloud tools.888https://cloud.google.com/speech-to-
text/; https://cloud.google.com/text-to-speech/

##  3 Dialogue Flow

The ranking model lends itself to the one-shot task of finding the most
relevant responses in a given context. However, a restaurant-browsing system
needs to support a dialogue flow where the user finds a restaurant, and then
asks questions about it. The dialogue state for each search scenario is
represented as the set of restaurants that are considered relevant. This
starts off as all the restaurants in the given city, and is assumed to
monotonically decrease in size as the conversation progresses until the user
converges to a single restaurant. A restaurant is only considered valid in the
context of a new user input if it has relevant responses corresponding to it.
This flow is summarised here:

S1. Initialise $R$ as the set of all restaurants in the city. Given the user’s
input, rank all the responses in the response pool pertaining to restaurants
in $R$.

S2. Retrieve the top $N$ responses $r_{1},r_{2},\ldots,r_{N}$ with
corresponding (sorted) cosine similarity scores: $s_{1}\geq
s_{2}\geq\ldots\geq s_{N}$.

S3. Compute probability scores $p_{i}\propto\exp(a\cdot s_{i})$ with
$\sum_{i=1}^{N}p_{i}$, where $a>0$ is a tunable constant.

S4. Compute a score $q_{e}$ for each restaurant/entity $e\in R$,
$q_{e}=\sum_{i:r_{i}\in e}p_{i}$.

S5. Update $R$ to the smallest set of restaurants with highest $q$ whose
$q$-values sum up to more than a predefined threshold $t$.

S6. Display the most relevant responses associated with the updated $R$, and
return to S2.

If there are multiple relevant restaurants, one response is shown from each.
When only one restaurant is relevant, the top $N$ responses are all shown, and
relevant photos are also displayed. The system does not require dedicated
understanding, decision-making, and generation modules, and this dialogue flow
does not rely on explicit task-tailored semantics. The set of relevant
restaurants is kept internally while the system narrows it down across
multiple dialogue turns. A simple set of predefined rules is used to provide a
templatic spoken system response: e.g., an example rule is “One review of $e$
said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response
associated with $e$. Note that while the demo is currently focused on the
restaurant search task, the described “narrowing down” dialogue flow is
generic and applicable to a variety of applications dealing with similar
entity search.

The system can use a set of intent classifiers to allow resetting the dialogue
state, or to activate the separate restaurant booking dialogue flow. These
classifiers are briefly discussed in §4.

(a) English system. City: Edinburgh.

(b) French system. City: Paris.

(c) German system. City: Berlin.

(d) Mandarin system. City: Taipei.

Figure 2: Snapshots of the PolyResponse demo system for restaurant search in
four different languages. Restaurant names are anonymised. Translations of
non-English sentences are provided in parentheses; they are not part of the
system output. The output also comprises relevant photos associated with the
current restaurant.

##  4 Other Functionality

Figure 3: An example showing how the system can retrieve parts of the menu as
responses to the current user utterance (if they are relevant to the
utterance).

### Multilinguality.

The PolyResponse restaurant search is currently available in 8 languages and
for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish
(Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean
(Seoul), and Serbian (Belgrade). Selected snapshots are shown in Figure 2,
while we also provide videos demonstrating the use and behaviour of the
systems at: https://tinyurl.com/y3evkcfz. A simple MT-based translate-to-
source approach at inference time is currently used to enable the deployment
of the system in other languages: 1) the pool of responses in each language is
translated to English by Google Translate beforehand, and pre-computed
encodings of their English translations are used as representations of each
foreign language response; 2) a provided user utterance (i.e., context) is
translated to English on-the-fly and its encoding $h_{c}$ is then learned. We
plan to experiment with more sophisticated multilingual models in future work.

### Voice-Controlled Menu Search.

An additional functionality enables the user to get parts of the restaurant
menu relevant to the current user utterance as responses. This is achieved by
performing an additional ranking step of available menu items and retrieving
the ones that are semantically relevant to the user utterance using exactly
the same methodology as with ranking other responses. An example of this
functionality is shown in Figure 3.

### Resetting and Switching to Booking.

The restaurant search system needs to support the discrete actions of
restarting the conversation (i.e., resetting the set $R$), and should enable
transferring to the slot-based table booking flow. This is achieved using two
binary intent classifiers, that are run at each step in the dialogue. These
classifiers make use of the already-computed $h_{c}$ vector that represents
the user’s latest text. A single-layer neural net is learned on top of the
$512$-dimensional encoding, with a ReLU activation and 100 hidden
nodes.999Using the Reddit encoding has shown better generalisation when
compared to models learned from scratch. This follows a recent trend where
small robust classifiers are learned on pretrained large models Devlin et al.
(2018). To train the classifiers, sets of 20 relevant paraphrases (e.g.,
“Start again”) are provided as positive examples. Finally, when the system
successfully switches to the booking scenario, it proceeds to the slot filling
task: it aims to extract all the relevant booking information from the user
(e.g., date, time, number of people to dine). The entire flow of the system
illustrating both the search phase and the booking phase is provided as the
supplemental video material.

##  5 Conclusion and Future Work

This paper has presented a general approach to search-based dialogue that does
not rely on explicit semantic representations such as dialogue acts or slot-
value ontologies, and allows for multi-modal responses. In future work, we
will extend the current demo system to more tasks and languages, and work with
more sophisticated encoders and ranking functions. Besides the initial
dialogue flow from this work (§3), we will also work with more complex flows
dealing, e.g., with user intent shifts.

## References

  * Al-Rfou et al. (2016) Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016.  Conversational contextual cues: The case of personalization and history for response ranking.  _CoRR_ , abs/1606.00372.
  * Bordes et al. (2017) Antoine Bordes, Y.-Lan Boureau, and Jason Weston. 2017.  Learning end-to-end goal-oriented dialog.  In _ICLR_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In _EMNLP_ , pages 5016–5026.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009\.  ImageNet: A large-scale hierarchical image database.  In _CVPR_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * El Asri et al. (2017) Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017.  Frames: A corpus for adding memory to goal-oriented dialogue systems.  In _SIGDIAL_ , pages 207–219.
  * Eric et al. (2017) Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017\.  Key-value retrieval networks for task-oriented dialogue.  In _SIGDIAL_ , pages 37–49.
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient natural language response suggestion for smart reply.  _CoRR_ , abs/1705.00652.
  * Henderson et al. (2019a) Matthew Henderson, Pawel Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen. 2019a.  A repository of conversational datasets.  In _NLP4ConvAI Workshop_ , pages 1–10.
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Steve Young. 2014.  Word-based dialog state tracking with recurrent neural networks.  In _SIGDIAL_.
  * Henderson et al. (2019b) Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b.  Training neural response selection for task-oriented dialogue systems.  In _ACL_ , pages 5392–5404.
  * Howard et al. (2017) Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017.  MobileNets: Efficient convolutional neural networks for mobile vision applications.  _CoRR_ , abs/1704.04861.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with GPUs.  _arXiv preprint arXiv:1702.08734_.
  * Li et al. (2017) Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017.  End-to-end task-completion neural dialogue systems.  In _IJCNLP_ , pages 733–743.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _LREC_.
  * Malkov and Yashunin (2016) Yury A. Malkov and D. A. Yashunin. 2016.  Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.  _CoRR_ , abs/1603.09320.
  * Mrkšić et al. (2015) Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015\.  Multi-domain dialog state tracking using recurrent neural networks.  In _ACL_ , pages 794–799.
  * Shah et al. (2018) Pararth Shah, Dilek Hakkani-Tür, Bing Liu, and Gokhan Tür. 2018.  Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning.  In _NAACL-HLT_ , pages 41–51.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_ , pages 6000–6010.
  * Wan and McAuley (2016) Mengting Wan and Julian McAuley. 2016.  Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems.  In _ICDM_ , pages 489–498.
  * Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017.  A network-based end-to-end trainable task-oriented dialogue system.  In _EACL_ , pages 438–449.
  * Williams (2012) Jason D. Williams. 2012.  A belief tracking challenge task for spoken dialog systems.
  * Young (2010) Steve Young. 2010.  Still talking to machines (cognitively speaking).  In _INTERSPEECH_ , pages 1–10.


# InterMulti:Multi-view Multimodal Interactions with Text-dominated
Hierarchical High-order Fusion for Emotion Analysis

Feng Qiu 1, Wanzeng Kong 2, Yu Ding 1 * Corresponding author.

###### Abstract

Humans are sophisticated at reading interlocutors’ emotions from multimodal
signals, such as speech contents, voice tones and facial expressions. However,
machines might struggle to understand various emotions due to the difficulty
of effectively decoding emotions from the complex interactions between
multimodal signals. In this paper, we propose a multimodal emotion analysis
framework, InterMulti, to capture complex multimodal interactions from
different views and identify emotions from multimodal signals. Our proposed
framework decomposes signals of different modalities into three kinds of
multimodal interaction representations, including a modality-full interaction
representation, a modality-shared interaction representation, and three
modality-specific interaction representations. Additionally, to balance the
contribution of different modalities and learn a more informative latent
interaction representation, we developed a novel Text-dominated Hierarchical
High-order Fusion(THHF) module. THHF module reasonably integrates the above
three kinds of representations into a comprehensive multimodal interaction
representation. Extensive experimental results on widely used datasets, i.e.,
MOSEI, MOSI and IEMOCAP, demonstrate that our method outperforms the state-of-
the-art.

##  1 Introduction

Endowing machines with emotional intelligence has been a long-standing goal
for engineers and researchers working on artificial intelligence (Hu and
Flaxman 2018; Hazarika et al. 2018). In daily conversations, humans are
skilled in conveying and understanding emotions through multimodal signals,
including spoken words, voice tones and facial expressions. These multimodal
signals are supplementary and complementary to each other in expressing
emotional information (Baltrušaitis, Ahuja, and Morency 2018) by skillfully
interacting with each other. Their interaction is a sophisticated and delicate
process (Zadeh et al. 2017). Therefore, it is vital to understand the
relationship and interaction among the multimodal signals for emotion
analysis.

It has still been challenging to explore complex multimodal interactions for
emotion analysis effectively. The exploration aims at learning a fusion
representation to offer effective information for emotion analysis. Previous
works usually integrate the multimodal information for emotion analysis by
direct concatenation (Zadeh et al. 2017), attention-assisted fusion (Zadeh et
al. 2018c, a) or correlation-based fusion (Zadeh et al. 2017; Liu et al. 2018;
Hou et al. 2019). Although these methods have attempted to take advantage of
complementary information among different modalities, they are often
challenged by the gaps among heterogeneous modalities (Hazarika, Zimmermann,
and Poria 2020). So these methods may be incapable of fully modeling the
complex interaction of multiple modalities. The refinement of modeling their
complex interactions may contribute to improving emotion analysis.

Later, the recent works capture more complex interactions between different
modalities by refining the learning of multimodal fusion representation
(Hazarika, Zimmermann, and Poria 2020; Han et al. 2021; Yu et al. 2021; Han,
Chen, and Poria 2021; Tsai et al. 2019a). These works decouple unimodal
features into multiple representations via regularizers and then fuse them.
They aim to capture independence and correlation between modalities. These
works approve the benefit of the decoupling operation in minimizing redundancy
between multimodal information. However, the decoupling operation with a
regularizer requires several auxiliary orthogonal constraint losses and hyper-
parameters balancing these losses, as well as a large quantity of additional
trainable parameters. These requirements make the training complex and may
limit further improvement.

To investigate complex multimodal interactions, our framework InterMulti
decomposes multiple unimodal features into three kinds of multimodal
interaction representations via a nonparametric decoupling method, including:

  1. 1.

The modality-full representations reserve intact information of the single
modality;

  2. 2.

The modality-shared interaction representation focuses on the contribution of
coordination among multiple modalities;

  3. 3.

The modality-specific interaction representations highlight the contribution
of specific information from individual modality, excluding common
information.

Importantly, different from previous works, our decoupling method is
nonparametric and instance-based. It makes a lower dependency between
modality-shared and modality-specific representations at the instance-level
than the distribution-based method like MISA (Hazarika, Zimmermann, and Poria
2020) (Please refer to Table 3). Our instance-based decoupling method requires
neither regular loss (e.g. similarity or difference loss) nor a discriminator
with additional learnable parameters and hyper-parameters. Therefore,
InterMulti is an effective and easy-training framework for modeling complex
multimodal interactions.

Then, these three kinds of interaction representations are merged into a
comprehensive fusion representation. It has been demonstrated that the text
modality plays the most significant role in multimodal emotion analysis (Pham
et al. 2019b; Tsai et al. 2019b). Therefore, we develop a novel module, named
Text-dominated Hierarchical High-order Fusion (THHF), to merge the different
kinds of interaction representations. THHF guides the model to balance the
contribution of different modalities and further unleash the representation
capacity of the multimodal interaction for emotion analysis.

To demonstrate the effectiveness of our framework, extensive experiments are
conducted on three datasets: MOSI (Fukui et al. 2016), MOSEI (Zadeh et al.
2018b) and IEMOCAP (Busso et al. 2008). The results show that our model
achieves state-of-the-art performance on nearly all benchmarks. Our
contributions can be summarised as follows:

  * •

We propose a novel framework InterMulti that emphasizes and validates the
importance of modeling multi-view multimodal interactions for emotion
analysis.

  * •

We develop a nonparametric instance-based decoupling operation and a novel
module called Text-dominated Hierarchical High-order Fusion (THHF) to
decompose and fuse multimodal features more reasonably and guide the model to
explore more informative and effective multimodal interaction representations.

  * •

Our method achieves state-of-the-art performance on three prevailing
benchmarks for emotion analysis.

##  2 Related Work

The goal of learning multimodal fusion representations from different unimodal
features is to model the interaction of multimodal features to offer effective
information for emotion analysis. As mentioned, most previous works carry out
the fusion by directly integrating the three-modal features, highlighting the
co-working of the intact unimodal features. This method is named integrating
learning in our work. Another work (Hazarika, Zimmermann, and Poria 2020)
first decouples the unimodal features into several intermediate
representations and then fuses them. Its underlying idea is that emotion is
encoded by not only the coordination of multimodalities but also the
separately working. This method is named decoupling learning in our work. The
reported integrating learning and decoupling learning methods will be briefly
reviewed.

#### Integrating learning.

A straightforward method is the concatenation of unimodal features
(Rajagopalan et al. 2016; Poria et al. 2017). Recently, a large number of
works have introduced the attention-based mechanism to integrate three-modal
features (Zadeh et al. 2018c, a; Han et al. 2021; Zadeh et al. 2018c, a; Han
et al. 2021; Wang et al. 2019). However, Zadeh et al. (2017) indicates that
both concatenation and attention-based methods over-fit to a specific modality
while ignoring the others and introducing modality bias. Some works attempt to
capture more relationships between modalities with various regularizations on
the learning of joint representation, such as CCA loss (Sun et al. 2020),
adversarial mechanism (Mai, Hu, and Xing 2020; Yuan et al. 2020), mutual
information (Han, Chen, and Poria 2021) and self-supervised manner (Pham et
al. 2019a; Sahu and Vechtomova 2021; Yu et al. 2021). These methods focus on
extracting the common information but ignoring the individual role of each
modality in emotion analysis. Other works use tensor fusion to highlight both
individual modality representations and their high-order relationships (Zadeh
et al. 2017; Hou et al. 2019; Liu et al. 2018).

#### Decoupling learning.

In multimodal emotion analysis, MISA (Hazarika, Zimmermann, and Poria 2020) is
the first work to utilize several orthogonal constraint losses to align
features from different modalities before fusion which bridges modality gaps.
MISA decouples three unimodal features into a modality-invariant
representation and a modality-specific representation, respectively. Their
decoupling operation relies on auxiliary orthogonality constraints, as well as
additional hyper-parameters to balance these losses. In training, the
supervision under several losses makes the training difficult, as the training
has to satisfy all the loss functions. In fact, the complex training cannot
guarantee the intention of a low similarity between the decoupling
representations, which probably restrains the performance of emotion analysis.
It is critical to make ease of training but effective performance when
decoupling representations.

##  3 Method

Figure 1: Illustration of InterMulti framework. $M^{enc}$ takes the utterance-
level unimodal features as input and produces unimodal representations
(${\textbf{{h}}_{t}}$, ${\textbf{{h}}_{v}}$, and ${\textbf{{h}}_{a}}$). Then
$M^{dec}$ decomposes ${\textbf{{h}}_{t}}$, ${\textbf{{h}}_{v}}$ and
${\textbf{{h}}_{a}}$ and obtain three kinds of interaction representations.
Later, the $M^{THHF}$ performs the semantic-based hierarchical fusion from
these interaction representations.

This section describes our InterMulti for multimodal emotion analysis from
three kinds of temporal input signals of text, audio and vision in a talking-
face clip. As illustrated in Figure 1, our framework InterMulti consists of
three modules including Unimodal Encoding Module ($M^{enc}$), Nonparametric
Decoupling Module ($M^{dec}$) and Text-dominated Hierarchical High-order
Fusion (THHF) Module ($M^{THHF}$). Given a talking-face clip, we use $M^{enc}$
to get three unimodal utterances-based features according to their temporal
input signals. Then $M^{dec}$ decomposes these three unimodal features into
one modality-shared representation and three modality-specific
representations. Additionally, we also get three modality-full representations
from utterances-based features. After that, $M^{THHF}$ is dedicated to
modeling the complex interaction across multiple modalities from different
views by fusing diverse interaction representations. Lastly, two linear
projection layers are introduced to enhance the representation capacity of
multimodal interaction representation further. More details will be described
as follows.

###  3.1 Unimodal Encoding Module

Given a talking-face clip consisting of three modalities (e.g. text, visual,
acoustic), $M^{enc}$ aims to extract unimodal features for each modality. We
first obtain the utterance features $\textbf{{u}}_{m}^{l}$
(${{m}\in\\{{t,v,a}\\}}$, where $t$ refers to text, $a$ to acoustic and $v$ to
vision; ${l\in\\{1,2,...,L\\}}$ refers to the index of temporal sequence of
modality $m$ where L refers to the total length of the sequence.) by pre-
trained unimodal feature extractors (details in Section 4.1). Then unimodal
encoders $\mathbb{E}_{m}$ compute the utterance-based 64-dimensional unimodal
representations $\textbf{{h}}_{m}$ (i.e. $\textbf{{h}}_{t}$,
$\textbf{{h}}_{a}$ and $\textbf{{h}}_{v}$). For each unimodal encoder, we
apply two single-layer bidirectional gated recurrent units (BiGRU) (Chung et
al. 2014) followed by a fully connected layer to capture the temporal
dependence and cast all the latent vectors into the same length for the
convenience of further fusion. The end-to-end learned $\textbf{{h}}_{m}$ is
capable of reducing the modality gaps between heterogeneous signals and
aligning them to the same representation space, facilitating their
interactions modeling later.

###  3.2 Nonparametric Self-decoupling Module

Taking unimodal representations $\textbf{{h}}_{m}$ as inputs, $M^{dec}$ aims
to decompose them into different kinds of intermediate representations. The
goal of decoupling is to facilitate complex interactions modeling. To this
end, we develop an nonparametric instance-based self-decoupling method to
decompose $\textbf{{h}}_{m}$ into two kinds of representations, including a
modality-shared representation $\mathcal{S}$ and three modality-specific
representations $\textbf{{i}}_{t}$, $\textbf{{i}}_{a}$ and $\textbf{{i}}_{v}$.
$\mathcal{S}$ represents common information across three modalities and
$\textbf{{i}}_{m}$ refers to the specific information for each modality. To
facilitate training, our instance-based decoupling requires neither additional
trainable parameters nor auxiliary loss functions.

To be specific, $\mathcal{S}$ is viewed as the average of $\textbf{{h}}_{t}$,
$\textbf{{h}}_{a}$ and $\textbf{{h}}_{v}$. Then $\textbf{{i}}_{t}$,
$\textbf{{i}}_{a}$ and $\textbf{{i}}_{v}$ are computed by subtracting
$\mathcal{S}$ from $\textbf{{h}}_{t}$, $\textbf{{h}}_{a}$ and
$\textbf{{h}}_{v}$, respectively. The subtraction moves out the shared
information from $\textbf{{h}}_{m}$ and reserves the specific information for
each modality. This nonparametric method obtains three modality-specific
representations with low dependence between different modalities. Otherwise,
we would obtain distribution-based low dependence between different
modalities, which cannot guarantee low dependence for samples far from the
center of the data distribution (Please refer to Table 3). Our nonparametric
self-decoupling operation is formulated as follows:

$\begin{array}[]{cc}\mathcal{S}&=\frac{1}{3}\sum\limits_{m}{\textbf{{h}}_{m}};\quad{m\in\\{{t,v,a}\\}}\\\
{\textbf{{i}}_{m}}&=\textbf{{h}}_{m}-\mathcal{S};\quad{m\in\\{{t,v,a}\\}}\\\
\end{array}$ (1)

To make modality-specific representations more compact, a fully connected
layer is introduced to transform each modality-specific representation
$\textbf{{i}}_{m}$ into the 16-dimensional representation
$\tilde{\textbf{{i}}}_{m}$, respectively. The end-to-end training allows
$\tilde{\textbf{{i}}}_{m}$ to learn appropriate modality-specific
representation spaces with the guide of the nonparametric decoupling operation
and the trainable fully connected layer. It prompts instance-based cross-
modality independence and modality-specific representation capacity.

Additionally, besides the modality-shared ($\mathcal{S}$) and modality-
specific ($\tilde{\textbf{{i}}}_{m}$) representations, we still retain the
modality-full features $\textbf{{h}}_{m}$ with all the information of unimodal
signals as information supplement. In order to reduce the number of parameters
and prepare for further fusion, we first reduce the dimension of
$\textbf{{h}}_{m}$ from 64 to 16 through a fully-connected layer and get the
representations $\tilde{\textbf{{h}}}_{m}$.

###  3.3 Text-dominated Hierarchical High-order Fusion Module

According to the semantic distinction, the above representations are
classified into three groups: the full unimodal representations
(${\tilde{\textbf{{h}}}}_{m}$), the modality-specific representations
($\tilde{\textbf{{i}}}_{m}$), and the modality-shared representation
($\mathcal{S}$). $M^{THHF}$ aims at reasonably merging diverse types of
representations from different views for emotion analysis by a hierarchical
structure. Considering the modality imbalance during multimodal fusion, we
further introduce the Text-dominated mechanism to guide the network to explore
more effective and informative interaction representations and capture their
modality-semantic relationships.

As illustrated in Figure 1, $M^{THHF}$ consists of two branches with similar
structure, modality-specific branch and modality-full branch. The former aims
to fuse $\tilde{\textbf{{i}}}_{m}$ into modality-specific interaction
representation $\mathcal{I}$ and the latter aims to fuse
$\tilde{\textbf{{h}}}_{m}$ into the modality-full one $\mathcal{M}$. For each
branch, to extract the text-dominated fused features, the textual features are
first combined with visual and acoustic features through Text-dominated High-
order Fusion blocks, respectively. Then, through the Flat High-order Fusion
block, we merge these two text-dominated fused features into a three-modal
interaction representation.

After that, we obtain three types of interaction representations
$\mathcal{M}$, $\mathcal{S}$, and $\mathcal{I}$. The underlying idea is that
$\mathcal{M}$, $\mathcal{S}$, and $\mathcal{I}$ all have three-modal
information but encode multimodal interactions in different ways and
complement each other. Such an multi-view encoding is able to model
sophisticated and informative interactions. Therefore, $\mathcal{S}$,
$\mathcal{I}$ and $\mathcal{M}$ are concatenated together, denoted as
comprehensive interaction representation $\mathcal{F}_{0}$. More details can
be seen below. For simplicity, we take the modality-specific branch as the
example.

#### Text-dominated High-order Fusion.

To fully model complex interaction among three-modal modality-full and
modality-specific representations ($\tilde{\textbf{{h}}}_{m}$ and
$\tilde{\textbf{{i}}}_{m}$), the outer product is utilized to integrate them.
The outer product is able to learn their high-order correlations (He et al.
2018) and tightly entangles the learning process of multimodal representation
elements with a multiplicative interaction between all elements. Therefore, it
avoids modality-bias overfitting and has been proved beneficial to the fusion
of multimodal features (Fukui et al. 2016; Lin, RoyChowdhury, and Maji 2015).

Figure 2: Illustration of the modality-specific branch of $M^{THHF}$.
Modality-full branch is similar to this.

Specifically, $\tilde{\textbf{{i}}}_{t}$ and $\tilde{\textbf{{i}}}_{v}$ are
first merged into a high-order tensor (16$\times$16) through the outer
product. Although outer product tightly entangles the learning processing of
multimodal representations, it also brings redundant information. To alleviate
this problem, a Max Pooling Layer is introduced to reduce the dimension of
high-order tensor into 8$\times$8\. Later, the tensor is flattened and reduced
into a 16-dimensional attention vector $\tilde{\textbf{{a}}}_{tv}$ which
contains the information about the interaction between text and visual
modalities. Since empirical research shows that text modality plays the most
significant role in mutlimodal emotion analysis, we take text as the main
modality and integrate the attention vector $\tilde{\textbf{{a}}}_{tv}$
through the dot product to get the fused representation
$\tilde{\textbf{{i}}}_{tv}$, which is formulated as follows:

$\mathcal{\tilde{\textbf{{i}}}}_{tv}=\tilde{\textbf{{i}}}_{t}\odot\textit{FC}(\textit{Flatten}(\textit{Pool}(\tilde{\textbf{{i}}}_{t}\otimes\tilde{\textbf{{i}}}_{v})))\\\
$ (2)

where $\otimes$ refers to outer product and $\odot$ refers to dot product.
Similarly, $\tilde{\textbf{{i}}}_{t}$ and $\tilde{\textbf{{i}}}_{a}$ are
merged into $\tilde{\textbf{{i}}}_{ta}$. The fusion process is formulated as
follows:

$\mathcal{\tilde{\textbf{{i}}}}_{ta}=\tilde{\textbf{{i}}}_{t}\odot\textit{FC}(\textit{Flatten}(\textit{Pool}(\tilde{\textbf{{i}}}_{t}\otimes\tilde{\textbf{{i}}}_{a})))\\\
$ (3)

Table 1: Performances of emotion analysis on MOSEI and MOSI with InterMulti (Ours) and other baselines. Best results are highlighted in bold. $\bigtriangleup$ means results are excerpted from previous papers; $\dagger$ indicates results are reproduced from open-source code with hyper-parameters provided in original papers. In the column of Acc-2 and F1, The */* refers to two results on the metric of _negative/non-negative_ or _negative/positive_ , respectively. And ${}^{\prime}-^{\prime}$ denotes that the corresponding terms are not reported in the original works. $C$ refers to the classical features provided by datasets; $B^{T}$, $B^{S}$ and $B^{TS}$ refer to using Text-BERT features, Speech-BERT and both Text-BERT and Speech-BERT features, respectively, with classical features for other modalities. $w/o$ $V$ refers to moving out the visual modality.  Method | MOSEI | MOSI
---|---|---
| MAE $\downarrow$ | Corr $\uparrow$ | Acc-2 $\uparrow$ | F1 $\uparrow$ | Acc-7 $\uparrow$ | MAE $\downarrow$ | Corr $\uparrow$ | Acc-2 $\uparrow$ | F1 $\uparrow$ | Acc-7 $\uparrow$
| MFN(C)△ | - | - | 76.0/- | 76.0/- | - | 0.965 | 0.632 | 77.4/- | 77.3/- | 34.1
| MV-LSTM(C)△ | - | - | 76.4/- | 76.4/- | - | 1.019 | 0.601 | 73.9/- | 74.0/- | 33.2
| SPT(C)△ | - | - | -/82.6 | -/82.8 | - | - | - | -/82.8 | -/82.9 | -
| MulT(C)△ | 0.580 | 0.703 | -/82.5 | -/82.3 | 51.8 | 0.871 | 0.698 | -/83.0 | -/82.8 | 40.0
| RAVEN(C)△ | 0.614 | 0.662 | 79.1/- | 79.5/- | 50.0 | 0.915 | 0.691 | 78.0/- | 76.6/- | 33.2
| MTAG(C)† | 0.572 | 0.714 | 79.6/82.1 | 79.6/82.4 | 50.9 | 0.852 | 0.715 | 80.6/82.4 | 80.5/82.4 | 38.7
|  InterMulti(C) | 0.543 | 0.764 | 83.4/85.7 | 83.6/85.8 | 51.9 | 0.793 | 0.756 | 81.1/83.0 |  80.8/82.0 | 42.1
| LMF($B^{T}$)△ | 0.623 | 0.677 | -/82.0 | -/82.1 | 48.0 | 0.917 | 0.695 | -/82.5 | -/82.4 | 33.2
| TFN($B^{T}$)△ | 0.593 | 0.700 | -/82.5 | -/82.1 | 50.2 | 0.901 | 0.698 | -/80.8 | -/80.7 | 34.9
| MFM($B^{T}$)△ | 0.568 | 0.717 | -/84.4 | -/84.3 | 51.3 | 0.877 | 0.706 | -/81.7 | -/81.6 | 35.4
| ICCN($B^{T}$)△ | 0.565 | 0.713 | -/84.2 | -/84.2 | 51.6 | 0.860 | 0.710 | -/83.0 | -/83.0 | 39.0
| MISA($B^{T}$)△ | 0.555 | 0.756 | 83.6/85.5 | 83.8/85.3 | 52.2 | 0.783 | 0.761 | 81.8/83.4 | 81.7/83.6 | 42.3
| BBFN($B^{T}$)△ | 0.557 | 0.767 | -/86.2 | -/86.1 | 54.8 | 0.776 | 0.755 | -/84.3 | -/84.3 | 45.0
| MMIM($B^{T}$)† | 0.530 | 0.766 | 82.4/85.8 | 82.5/85.9 | 54.0 | 0.720 | 0.787 |  82.8/84.6 | 82.7/84.6 | 44.8
| MAG-BERT($B^{T}$)† | 0.543 | 0.755 | 82.5/84.8 | 83.0/84.7 | 52.7 | 0.776 | 0.781 | 82.4/84.4 | 82.5/84.6 | 43.6
| Self-MM($B^{T}$)† | 0.532 | 0.764 | 82.5/84.9 | 82.5/85.1 | 53.1 | 0.725 | 0.789 |  82.8/84.2 |  82.8/84.2 | 44.8
|  InterMulti($B^{T}$) | 0.528 | 0.776 |  85.1/86.3 |  85.1/86.1 | 53.9 | 0.714 | 0.795 | 82.1/84.8 | 81.9/84.8 | 45.3
| SSL($B^{TS}$, $w/o$ $V$)△ | 0.491 | - | -/88.0 | -/88.1 | 56.0 | 0.577 | - | -/88.3 | -/88.6 | 47.2
|  InterMulti ($B^{TS}$, $w/o$ $V$) | 0.483 | 0.773 | 87.0/88.3 | 86.9/88.1 | 56.5 | 0.586 | 0.871 | 87.8/89.2 | 87.8/89.1 | 51.5
|  |  |  |  |  |  |  |  |  |  |

#### Flat High-order Fusion.

Next, to further capture the interactions across multiple modalities, we merge
two Text-dominated fusion representations $\tilde{\textbf{{i}}}_{tv}$ and
$\tilde{\textbf{{i}}}_{ta}$ into a modality-specific interaction
representation $\mathcal{I}$ through Flat High-order Fusion. To be specific,
we use an outer product following a pooling layer and a fully connected layer
to perform fusion, which is formulated as follows:

$\mathcal{I}=\textit{FC}(\textit{Flatten}(\textit{Pool}(\tilde{\textbf{{i}}}_{ta}\otimes\tilde{\textbf{{i}}}_{tv})))\\\
$ (4)

Similarly, in the modality-full branch, we merged $\tilde{\textbf{{h}}}_{m}$
into a modality-full interaction representation $\mathcal{M}$.

#### Comprehensive Fusion.

Lastly, we take three intermediate multimodal interaction representations,
$\mathcal{S}$, $\mathcal{I}$ and $\mathcal{M}$, as inputs to characterize
multi-view multimodal interactions from different views:

  1. 1.

A modality-shared interaction representation $\mathcal{S}$ modeling three-
modal common information;

  2. 2.

A three-modal modality-specific interaction representation $\mathcal{I}$
modeling the interaction among three modality-specific representations;

  3. 3.

Another three-modal modality-full interaction representation $\mathcal{M}$
integrates three full unimodal representations $\textbf{{h}}_{m}$ and models
three-modal intact information interaction, while $\mathcal{S}$ and
$\mathcal{I}$ model interactions relying on partial information from
$\textbf{{h}}_{m}$.

The well-designed end-to-end training allows $\mathcal{S,M,I}$ to model
distinguishable multimodal interactions according to their specific essences
mentioned above. To further unleash the representation capacity of the
multimodal interaction for emotion analysis, three intermediate interaction
representations $\mathcal{S}$, $\mathcal{I}$ and $\mathcal{M}$ are then
concatenated into a comprehensive interaction representation
$\mathcal{F}_{0}$.

###  3.4 Training Loss

Our experiments perform emotion analysis, including emotion classification or
sentiment intensity prediction, depending on the labels provided by the
datasets (details in Section 4.1). Specifically, when performing emotion
classification, the standard cross-entropy loss is employed as follows:

$\mathcal{L}_{\text{CE}}=-\frac{1}{N}\sum_{i=0}^{N}\mathrm{y}_{i}\cdot\log\hat{\mathrm{y}}_{i},$
(5)

When performing intensity prediction, we take the mean squared error (MSE)
loss as follows:

$\mathcal{L}_{\text{MSE
}}=-\frac{1}{N}\sum_{i=0}^{N}\left\|\mathrm{y}_{i}-\hat{\mathrm{y}}_{i}\right\|^{2},$
(6)

where $N$ is the mini-batch size and ${\mathrm{y}}_{i}$ and
$\hat{\mathrm{y}}_{i}$ are the ground-truth and predicted intensity for
$i$-$th$ utterance, respectively.

##  4 Experiment

Table 2: Comparative results of emotion analysis on IEMOCAP. Best results are highlighted in bold. $C$ refers to the classical features provided by datasets; $B^{TS}$ refers to using both Text-BERT and Speech-BERT features. $w/o$ $V$ refers to moving out the visual modality.  Models | IEMOCAP
---|---
Emotions | Happy | Angry | Sad | Neutral
Acc-2$\uparrow$ | F1$\uparrow$ | Acc-2$\uparrow$ | F1$\uparrow$ | Acc-2$\uparrow$ | F1$\uparrow$ | Acc-2$\uparrow$ | F1$\uparrow$
| MV-LSTM(C) | 85.9 | 81.3 | 85.1 | 84.3 | 80.4 | 74.0 | 67.0 | 66.7
MARN(C) | 86.7 | 83.6 | 84.6 | 84.2 | 82.0 | 81.2 | 66.8 | 65.9
MFN(C) | 86.5 | 84.0 | 85.0 | 83.7 | 83.5 | 82.1 | 69.6 | 69.2
RMFN(C) | 87.5 | 85.8 | 84.6 | 84.2 | 82.9 | 85.1 | 69.5 | 69.1
RAVEN(C) | 87.3 | 85.8 | 87.3 | 86.7 | 83.4 | 83.1 | 69.7 | 69.3
TFN(C) | 86.7 | 84.0 | 87.1 | 87.0 | 85.6 | 85.8 | 68.9 | 68.0
LMF(C) | 86.1 | 83.9 | 86.2 | 86.4 | 84.3 | 84.4 | 69.6 | 68.8
MulT(C) | 90.7 | 88.6 | 87.4 | 87.0 | 86.7 | 86.0 | 72.4 | 70.7
| MFM(C) | 86.7 | 84.7 | 87.0 | 86.7 | 85.7 | 85.7 | 70.3 | 69.9
| ICCN(C) | 87.4 | 84.7 | 88.6 | 88.0 | 86.3 | 85.9 | 69.7 | 68.5
| MTAG(C) | - | 86.0 | - | 76.7 | - | 79.9 | - | 64.1
|  InterMulti(C) | 87.4 | 86.3 | 90.1 | 89.8 | 86.8 | 86.5 | 74.1 | 73.9
| SSL($B^{TS}$, $w/o$ $V$) | 84.1 | 83.8 | 93.6 | 93.6 | 90.9 | 90.7 | 81.6 | 81.1
|  InterMulti ($B^{TS}$, $w/o$ $V$) | 85.1 | 84.3 | 94.0 | 93.9 | 91.5 | 91.3 | 82.6 | 81.2
|  |  |  |  |  |  |  |  |

Extensive experiments are conducted to evaluate the proposed InterMulti.
First, Our InterMulti is validated by comparing to baselines and analyzing on
its performance. Secondly, ablation studies are conducted to look into the
performance of the InterMulti.

Implementation Details. The optimization uses the Adam optimizer (Kingma and
Ba 2017) with learning rate 0.0001. The training duration of each model is
governed by an early-stopping strategy with the patience of 10 epochs. The
size of the mini-batch is 64. The training relies on one GTX3090 GPU. The
partition of the data for training, validation, and testing follows the
official setting adopted by most of the previous works.

Datasets. Our experiments perform on the widely-used MOSI (Zadeh et al. 2016),
MOSEI (Zadeh et al. 2018b) and IEMOCAP (Busso et al. 2008). MOSI contains
2,199 utterance samples from YouTube, spanning 89 different speakers. Each
sample is manually annotated with seven emotional intensities ranging from
$-3$ to $+3$, where -3/+3 indicates a strong negative/positive emotion. MOSEI
is an extended version of MOSI with unified annotation labels. It includes
23,453 annotated utterance samples, 1,000 distinct speakers, and 250 different
topics. The IEMOCAP contains 9,958 utterance samples from professional actors.
In IEMOCAP, each utterance sample is labelled with one emotion label,
including seven kinds of emotion. Due to the imbalance of some emotion labels,
the four categories of Happy, Sad, Anger, and Neutral are most commonly used
in the previous works, which is followed by our work.

Due to the difference in the set of labels in the above datasets, our
experiments predict emotion intensity in MOSI/MOSEI and classify emotion
labels in IEMOCAP.

Preprocessing. Traditionally, text modality features are GloVe (Pennington,
Socher, and Manning 2014) embeddings; for visual modality, previous works use
Facet to extract facial expression features (Ekman, Freisen, and Ancoli 1980);
for acoustic modality, COVAREP (Degottex et al. 2014) is used to extract the
features. We call these three features as the classical features denoted as
$C$. Recent works (Rahman et al. 2020a; Hazarika, Zimmermann, and Poria 2020)
have demonstrated that BERT (Devlin et al. 2019) can provide better features
than GloVe for emotion analysis and boost the performance. Later on,
Siriwardhana et al. (2020) further introduces Speech-BERT (Baevski and Mohamed
2020) to extract audio representations. For a fair comparison, we also conduct
experiments on BERT features and Speech-BERT features denoted as $B^{T}$ and
$B^{S}$, respectively.

Additionally, we perform word-level alignment to obtain aligned data by the
prevalent tool kit P2FA (Yuan, Liberman et al. 2008), which is regularly
employed in previous works.

Table 3: Dependency between different decoupled representations based on MOSEI. Methods |  ${\textbf{{i}}}_{a}$ \- ${\textbf{{i}}}_{t}$ |  ${\textbf{{i}}}_{a}$ \- ${\textbf{{i}}}_{v}$ |  ${\textbf{{i}}}_{t}$ \- ${\textbf{{i}}}_{v}$ |  ${\textbf{{i}}}_{a}$ \- S |  ${\textbf{{i}}}_{t}$ \- S |  ${\textbf{{i}}}_{v}$ \- S
---|---|---|---|---|---|---
InterMulti | 4.3 | 4.2 | 4.4 | 4.3 | 4.4 | 2.8
MISA | 28.9 | 28.8 | 30.3 | 28.6 | 30.58 | 30.2
|  |  |  |  |  |

###  4.1 Comparisons with the State-of-the-arts

The baseline methods include MFN (Zadeh et al. 2018a), MV-LSTM (Rajagopalan et
al. 2016), RAVEN (Wang et al. 2019), MARN (Zadeh et al. 2018c), RMFN (Liang et
al. 2018), SPT (Cheng et al. 2021), MCTN (Pham et al. 2019a), MulT (Tsai et
al. 2019c), TFN (Zadeh et al. 2017), LMF (Liu et al. 2018), MFM (Tsai et al.
2019d), ICCN (Sun et al. 2020), MISA (Hazarika, Zimmermann, and Poria 2020),
BBFN (Han et al. 2021), MMIM (Han, Chen, and Poria 2021), Self-MM (Yu et al.
2021), MAG-BERT (Rahman et al. 2020b), MTAG (Tsai et al. 2019a) and
SSL(Siriwardhana et al. 2020). As done in their works, MAE (mean absolute
error), Corr (Pearson correlation coefficient), Acc-7 (7-class accuracy),
Acc-2 (binary accuracy) and F1 score are taken as the metrics on MOSI/MOSEI,
while Acc-2 and F1 score on IEMOCAP. Specially, Acc-2 may refer to
_negative/non-negative_ or _negative/positive_ on MOSI/MOSEI while refers to
the latter on IEMOCAP.

Table 4: Results of the ablation study. All the experiments employ the classical features (C) provided by datasets. Dataset | MOSEI | MOSI | IEMOCAP
---|---|---|---
Ablation Study | MAE $\downarrow$ | Corr $\uparrow$ | F1 $\uparrow$ | MAE $\downarrow$ | Corr $\uparrow$ | F1 $\uparrow$ | F1 $\uparrow$
Happy | Angry | Sad | Neutral
A0 | InterMulti | 0.543 | 0.764 | 85.8 | 0.793 | 0.756 | 82.0 | 86.3 | 89.8 | 86.5 | 73.9
A1 |  $\mathcal{S}$ \+ $\mathcal{M}$ | 0.549 | 0.759 | 83.2 | 0.807 | 0.740 | 81.2 | 85.6 | 88.7 | 82.6 | 71.7
A2 |  $\mathcal{S}$ \+ $\mathcal{I}$ | 0.546 | 0.760 | 85.4 | 0.819 | 0.735 | 79.3 | 81.3 | 88.4 | 83.8 | 69.7
A3 |  $\mathcal{M}$ \+ $\mathcal{I}$ | 0.547 | 0.760 | 85.4 | 0.830 | 0.726 | 81.3 | 85.6 | 89.0 | 80.3 | 72.3
A4 |  $\mathcal{S}$ only | 0.546 | 0.759 | 84.8 | 0.841 | 0.717 | 79.6 | 85.1 | 88.8 | 80.1 | 70.0
A5 |  $\mathcal{M}$ only | 0.572 | 0.740 | 85.0 | 0.831 | 0.739 | 81.3 | 85.7 | 88.5 | 83.5 | 70.5
A6 |  $\mathcal{I}$ only | 0.545 | 0.758 | 85.0 | 0.811 | 0.741 | 80.9 | 85.2 | 88.0 | 82.8 | 72.4
A7 | w/o text | 0.823 | 0.273 | 67.7 | 1.455 | 0.015 | 59.4 | 83.0 | 86.8 | 78.7 | 65.1
A8 | w/o visual | 0.551 | 0.755 | 85.5 | 0.880 | 0.708 | 80.6 | 81.7 | 88.2 | 81.4 | 69.7
A9 | w/o acoustic | 0.553 | 0.752 | 85.0 | 0.873 | 0.719 | 81.7 | 84.4 | 85.2 | 81.3 | 69.5
A10 | ortho. constraint | 0.557 | 0.757 | 85.5 | 0.813 | 0.744 | 80.7 | 85.0 | 87.0 | 82.1 | 71.0
A11 | w/o $M^{THHF}$ | 0.556 | 0.743 | 84.9 | 0.814 | 0.732 | 80.4 | 85.2 | 88.8 | 84.9 | 72.1
A12 | w/o outer product | 0.541 | 0.755 | 85.4 | 0.809 | 0.736 | 81.5 | 85.9 | 88.6 | 85.8 | 73.6
A13 | w/o text-dominated fusion | 0.551 | 0.759 | 85.1 | 0.806 | 0.731 | 80.1 | 85.0 | 88.2 | 86.1 | 71.9
A14 | co-attention | 0.562 | 0.751 | 84.8 | 0.826 | 0.742 | 80.2 | 84.9 | 88.5 | 84.2 | 71.5
A15 | visual-dominated fusion | 0.549 | 0.741 | 85.4 | 0.804 | 0.743 | 81.4 | 85.5 | 88.7 | 85.9 | 73.6
A16 | acoustic-dominated fusion | 0.569 | 0.752 | 84.6 | 0.812 | 0.731 | 81.3 | 85.8 | 89.5 | 86.1 | 73.4
|  |  |  |  |  |  |  |  |  |  |  |

#### Results.

To make fair comparisons, our results rely on the identical features with the
baselines. As shown in Table 1, compared with the methods followed ($C$) using
the classical features, our results outperform all the baselines on MOSI and
MOSEI and almost baselines on IEMOCAP except MulT. However, our InterMulti
outperforms MulT on all the metrics on MOSI and MOSEI and three out of four
emotions on IEMOCAP. Looking into those methods using the classical audio and
vision features but text-BERT features($B^{T}$), our InterMulti outperforms
the baselines on almost all the metrics except Acc-2(negative/non-negative) on
MOSI and ACC-7 on MOSEI. Particularly, compared to SSL using two-modal
features of text-BERT and audio-Bert, our InterMulti ($B^{TS}$) performs
better on nine metrics, except for MAE on the smaller dataset of MOSI. The
above observations validate our proposed framework InterMulti.

Figure 3: The visualization of the representations ${\textbf{{i}}}_{t}$,
${\textbf{{i}}}_{v}$, ${\textbf{{i}}}_{a}$, ${\textbf{{h}}}_{t}$,
${\textbf{{h}}}_{v}$, ${\textbf{{h}}}_{a}$ and $\mathcal{S}$ , based on the
test set of MOSEI. The dimension reduction is conducted with tSNE.

To further obtain insight into our method, we compare the dependency between
our decoupling representations with MISA (Hazarika, Zimmermann, and Poria
2020). MISA decouples multimodal features by the distribution-based training
that requires additional trainable parameters. Since the dot product
represents the length of projection from one vector to the other, we employ it
to measure the dependency between two vectors. The results in Table 3 indicate
that our nonparametric decoupling operation leads to distinguishable
representations with lower dependency, facilitating encoding different
information and improving representation capacity. Additionally, Figure 3
visualizes the learned representations ${\textbf{{i}}}_{t}$,
${\textbf{{i}}}_{v}$, ${\textbf{{i}}}_{a}$, ${\textbf{{h}}}_{t}$,
${\textbf{{h}}}_{v}$, ${\textbf{{h}}}_{a}$ and $\mathcal{S}$, which are fed
into the Hierarchical Fusion Module, by tSNE-based (Maaten and Hinton 2008)
dimension reduction. Obvious gaps between their distributions can be seen,
which indicating that these representations learn to provide different
information and contribute to comprehensive fusion.

###  4.2 Ablation Study

To further validate the effectiveness of our framework and the proposed
components, we conduct ablation studies relying on MOSEI, MOSI and IEMOCAP.
For simplicity, we only report on the metric of _negative/positive_ on F1. The
results are reported in Table 4.

To confirm the necessity of three kinds of interaction representations in our
proposed framework, we conduct ablation studies A1-A6. A1-A6 are conducted by
employing one or two representations of $\mathcal{S}$, $\mathcal{I}$ and
$\mathcal{M}$ while keeping other parts the same as A0. The results validate
the effectiveness of our proposed framework. The proposed method (A0) with
$\mathcal{S,M,I}$ is better than using one or two of them. It proves that
these three interaction representations carry on different interaction
information and are capable of modeling the complex multimodal interactions
together. All $\mathcal{S,M,I}$ cannot be neglected and cannot replace each
other.

To confirm the capability of fitting to each modality, we conduct A7-A9 in
Table 4 by using only two modal information as inputs and ignoring one
modality of text, audio or vision, respectively. It is found that A7-A9 are
worse than InterMulti (A0) using all three modalities. It means that
InterMulti is capable of extracting meaningful but different information from
each modality.

To furthermore confirm our decoupling operation, we conduct A10 which replaces
self-decoupling operation in InterMulti (A0) by the orthogonal constraints
(including the similarity, difference and reconstruction losses) inspired by
MISA (Hazarika, Zimmermann, and Poria 2020). The result shows that the
performance of A10 goes worse in comparison with A0. This observation
furthermore validates our decoupling operation in an end-to-end training.

To validate the effectiveness of $M^{THHF}$, in A11, we replaces $M^{THHF}$ by
just concatenating three types of representations, $\tilde{\textbf{{i}}}_{m}$,
$\tilde{\textbf{{h}}}_{m}$, and $\mathcal{S}$. The result shows that the
performance of A11 goes worse without hierarchical fusion, which means that
our hierarchical fusion can effectively model the complex interaction between
different modalities. We further replace the outer product with concatenation
in A12 and the Text-dominated High-order Fusion with Flat High-order Fusion in
A13. Besides, we bring in co-attentional transformer layers (Lu et al. 2019)
to replace the outer product in A14. The performance of A12, A13 and A14 also
goes worse compared to A0, which means that the outer product is necessary for
tightly entangling the elements in three-modal representations. In A15 and
A16, we also try visual/acoustic-dominated fusion, which is worse than text-
domonated fusion. The observation in A11 to A16 validates the well-designed
THHF Module.

All the above experimental results confirm the effectiveness of our InterMulti
from three-modal signals, the use of three diverse interaction representations
$\mathcal{S,M,I}$, the nonparametric decoupling operation and the text-
dominated hierarchical high-order fusion.

##  5 Conclusion

In this paper, we propose a novel framework named InterMulti for multimodal
emotion analysis. InterMulti is dedicated to modeling the complex multimodal
interactions across multiple modalities in diverse views. Based on that, a
novel nonparametric decoupling operation is developed to learn modality-
shared/specific interaction representations. It benefits from ease of
implementation and no additional parameters or constraints. Additionally, a
powerful Text-dominated Hierarchical High-order Fusion (THHF) is proposed to
reasonably integrates the diverse representations for emotion analysis. The
experiments validate the effectiveness of our InterMulti.

## References

  * Baevski and Mohamed (2020) Baevski, A.; and Mohamed, A. 2020.  Effectiveness of Self-Supervised Pre-Training for ASR.  _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 7694–7698.
  * Baltrušaitis, Ahuja, and Morency (2018) Baltrušaitis, T.; Ahuja, C.; and Morency, L.-P. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2): 423–443.
  * Busso et al. (2008) Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42(4): 335.
  * Cheng et al. (2021) Cheng, J.; Fostiropoulos, I.; Boehm, B.; and Soleymani, M. 2021.  Multimodal Phased Transformer for Sentiment Analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2447–2458.
  * Chung et al. (2014) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In _NIPS 2014 Workshop on Deep Learning, December 2014_.
  * Degottex et al. (2014) Degottex, G.; Kane, J.; Drugman, T.; Raitio, T.; and Scherer, S. 2014.  COVAREP—A collaborative voice analysis repository for speech technologies.  In _International conference on acoustics, speech and signal processing (icassp)_ , 960–964. IEEE, Citeseer.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Association for Computational Linguistics.
  * Ekman, Freisen, and Ancoli (1980) Ekman, P.; Freisen, W. V.; and Ancoli, S. 1980.  Facial signs of emotional experience.  _Journal of personality and social psychology_ , 39(6): 1125.
  * Fukui et al. (2016) Fukui, A.; Park, D. H.; Yang, D.; Rohrbach, A.; Darrell, T.; and Rohrbach, M. 2016\.  Multimodal compact bilinear pooling for visual question answering and visual grounding.  _arXiv preprint arXiv:1606.01847_.
  * Han et al. (2021) Han, W.; Chen, H.; Gelbukh, A.; Zadeh, A.; Morency, L.-p.; and Poria, S. 2021.  Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , 6–15.
  * Han, Chen, and Poria (2021) Han, W.; Chen, H.; and Poria, S. 2021.  Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 9180–9192.
  * Hazarika et al. (2018) Hazarika, D.; Poria, S.; Zadeh, A.; Cambria, E.; Morency, L.-P.; and Zimmermann, R. 2018.  Conversational memory network for emotion recognition in dyadic dialogue videos.  In _Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting_ , volume 2018, 2122. NIH Public Access.
  * Hazarika, Zimmermann, and Poria (2020) Hazarika, D.; Zimmermann, R.; and Poria, S. 2020.  _MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis_ , 1122–1131.  Association for Computing Machinery.
  * He et al. (2018) He, X.; Du, X.; Wang, X.; Tian, F.; Tang, J.; and Chua, T.-S. 2018.  Outer product-based neural collaborative filtering.  _arXiv preprint arXiv:1808.03912_.
  * Hou et al. (2019) Hou, M.; Tang, J.; Zhang, J.; Kong, W.; and Zhao, Q. 2019.  Deep multimodal multilinear fusion with high-order polynomial pooling.  _Advances in Neural Information Processing Systems_ , 32: 12136–12145.
  * Hu and Flaxman (2018) Hu, A.; and Flaxman, S. 2018.  _Multimodal Sentiment Analysis To Explore the Structure of Emotions_ , 350–358.  New York, NY, USA: Association for Computing Machinery.
  * Kingma and Ba (2017) Kingma, D. P.; and Ba, J. 2017.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980.
  * Liang et al. (2018) Liang, P. P.; Liu, Z.; Bagher Zadeh, A.; and Morency, L.-P. 2018.  Multimodal Language Analysis with Recurrent Multistage Fusion.  In _Conference on Empirical Methods in Natural Language Processing_ , 150–161. Brussels, Belgium: Association for Computational Linguistics.
  * Lin, RoyChowdhury, and Maji (2015) Lin, T.-Y.; RoyChowdhury, A.; and Maji, S. 2015.  Bilinear cnn models for fine-grained visual recognition.  In _Proceedings of the IEEE international conference on computer vision_ , 1449–1457.
  * Liu et al. (2018) Liu, Z.; Shen, Y.; Lakshminarasimhan, V. B.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2018.  Efficient low-rank multimodal fusion with modality-specific factors.  _arXiv preprint arXiv:1806.00064_.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_.
  * Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(Nov): 2579–2605.
  * Mai, Hu, and Xing (2020) Mai, S.; Hu, H.; and Xing, S. 2020.  Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(01): 164–172.
  * Pennington, Socher, and Manning (2014) Pennington, J.; Socher, R.; and Manning, C. 2014.  GloVe: Global Vectors for Word Representation.  In _Conference on Empirical Methods in Natural Language Processing)_ , 1532–1543.
  * Pham et al. (2019a) Pham, H.; Liang, P. P.; Manzini, T.; Morency, L.-P.; and Póczos, B. 2019a.  Found in translation: Learning robust joint representations by cyclic translations between modalities.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 6892–6899.
  * Pham et al. (2019b) Pham, N.-Q.; Niehues, J.; Ha, T.-L.; and Waibel, A. 2019b.  Improving Zero-shot Translation with Language-Independent Constraints.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , 13–23.
  * Poria et al. (2017) Poria, S.; Cambria, E.; Hazarika, D.; Majumder, N.; Zadeh, A.; and Morency, L.-P. 2017.  Context-Dependent Sentiment Analysis in User-Generated Videos.  In _Annual Meeting of the Association for Computational Linguistics_ , 873–883. Vancouver, Canada.
  * Rahman et al. (2020a) Rahman, W.; Hasan, M. K.; Lee, S.; Zadeh, A.; Mao, C.; Morency, L.-P.; and Hoque, E. 2020a.  Integrating multimodal information in large pretrained transformers.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2020, 2359. NIH Public Access.
  * Rahman et al. (2020b) Rahman, W.; Hasan, M. K.; Lee, S.; Zadeh, A.; Mao, C.; Morency, L.-P.; and Hoque, E. 2020b.  Integrating multimodal information in large pretrained transformers.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2020, 2359. NIH Public Access.
  * Rajagopalan et al. (2016) Rajagopalan, S. S.; Morency, L.-P.; Baltrusaitis, T.; and Goecke, R. 2016.  Extending long short-term memory for multi-view structured learning.  In _European Conference on Computer Vision_ , 338–353. Springer.
  * Sahu and Vechtomova (2021) Sahu, G.; and Vechtomova, O. 2021.  Adaptive Fusion Techniques for Multimodal Data.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 3156–3166.
  * Siriwardhana et al. (2020) Siriwardhana, S.; Reis, A.; Weerasekera, R.; and Nanayakkara, S. 2020.  Jointly Fine-Tuning “BERT-Like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition.  _Proc. Interspeech 2020_ , 3755–3759.
  * Sun et al. (2020) Sun, Z.; Sarma, P.; Sethares, W.; and Liang, Y. 2020.  Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis.  _arXiv e-prints_ , 34: 8992–8999.
  * Tsai et al. (2019a) Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019a.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, 6558. NIH Public Access.
  * Tsai et al. (2019b) Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019b.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, 6558. NIH Public Access.
  * Tsai et al. (2019c) Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019c.  Multimodal Transformer for Unaligned Multimodal Language Sequences.  In _Annual Meeting of the Association for Computational Linguistics_ , 6558–6569. Florence, Italy: Association for Computational Linguistics.
  * Tsai et al. (2019d) Tsai, Y.-H. H.; Liang, P. P.; Zadeh, A.; Morency, L.-P.; and Salakhutdinov, R. 2019d.  Learning Factorized Multimodal Representations.  arXiv:1806.06176.
  * Wang et al. (2019) Wang, Y.; Shen, Y.; Liu, Z.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2019.  Words can shift: Dynamically adjusting word representations using nonverbal behaviors.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 7216–7223.
  * Yu et al. (2021) Yu, W.; Xu, H.; Yuan, Z.; and Wu, J. 2021.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 10790–10797.
  * Yuan, Liberman et al. (2008) Yuan, J.; Liberman, M.; et al. 2008.  Speaker identification on the SCOTUS corpus.  _Journal of the Acoustical Society of America_ , 123(5): 3878.
  * Yuan et al. (2020) Yuan, Z.; Sun, S.; Duan, L.; Li, C.; and Xu, C. 2020.  Adversarial Multimodal Network for Movie Story Question Answering.  _IEEE Transactions on Multimedia_ , PP(99): 1–1.
  * Zadeh et al. (2017) Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017.  Tensor fusion network for multimodal sentiment analysis.  _arXiv preprint arXiv:1707.07250_.
  * Zadeh et al. (2018a) Zadeh, A.; Liang, P. P.; Mazumder, N.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018a.  Memory fusion network for multi-view sequential learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Zadeh et al. (2018b) Zadeh, A.; Liang, P. P.; Mazumder, N.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018b.  Memory fusion network for multi-view sequential learning.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , volume 32.
  * Zadeh et al. (2018c) Zadeh, A.; Liang, P. P.; Poria, S.; Vij, P.; Cambria, E.; and Morency, L.-P. 2018c.  Multi-attention recurrent network for human communication comprehension.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Zadeh et al. (2016) Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6): 82–88.