# Interpretable Tensor Fusion

Saurabh Varshneya1 Antoine Ledent2 Philipp Liznerski1 Andriy Balinskyy1
Purvanshi Mehta3 Waleed Mustafa1&Marius Kloft1
1RPTU Kaiserslautern-Landau
2Singapore Management University
3Lica World
{varshneya, liznerski, balinskyy, mustafa<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Conventional machine learning methods are predominantly designed to predict
outcomes based on a single data type. However, practical applications may
encompass data of diverse types, such as text, images, and audio. We introduce
interpretable tensor fusion (InTense), a multimodal learning method for
training neural networks to simultaneously learn multimodal data
representations and their interpretable fusion. InTense can separately capture
both linear combinations and multiplicative interactions of diverse data
types, thereby disentangling higher-order interactions from the individual
effects of each modality. InTense provides interpretability out of the box by
assigning relevance scores to modalities and their associations. The approach
is theoretically grounded and yields meaningful relevance scores on multiple
synthetic and real-world datasets. Experiments on six real-world datasets show
that InTense outperforms existing state-of-the-art multimodal interpretable
approaches in terms of accuracy and interpretability. ††Code will be available
at: https://github.com/srb-cv/InTense

##  1 Introduction

The vast majority of machine learning systems are designed to predict outcomes
based on a single datatype or “modality”. However, in various
applications—spanning fields from biology and medicine to engineering and
multimedia—multiple modalities are frequently in play He et al. (2020); Lunghi
et al. (2019). The main challenge in multimodal learning is how to effectively
fuse these diverse modalities. The most common approach is to combine the
modalities in an additive way Poria et al. (2015); Chagas et al. (2020). Such
linear combinations suffice in some cases. However, numerous applications
necessitate capturing non-linear interactions between modalities. One such
instance is sarcasm detection, described in Figure 6 Hessel and Lee (2020).
The arguably most popular approach to capture non-linear interactions of
modalities is “Tensor fusion” Zadeh et al. (2017); Tsai et al. (2019); Liang
et al. (2021). The main idea in tensor fusion is to concatenate modalities via
tensor products in a neural network.

Figure 1:  Left is an excerpt of the MUStARD dataset on sarcasm detection,
where the proposed InTense method sets a new state-of-the-art. (See Section 4
for details.) A linear combination of modalities fails here because the
expressions of happiness and anxiety combine to something neutral rather than
sarcasm. To detect sarcasm, the interactions among modalities are crucial.
InTense captures these interactions and assigns them with interpretable
relevance scores, shown in the pie chart. Scores for individual modalities and
their interactions are colored green and blue, respectively. InTense reveals
that interactions are crucial for successful sarcasm detection.

A substantial drawback of tensor fusion is its inherent lack of
interpretability, which can significantly hinder its application in real-world
scenarios. Interpretable multimodal models may reveal the relative importance
of modalities Büchel et al. (1998); Hessel and Lee (2020), unveiling spurious
modalities and social biases in the data. Identifying interactions among
modalities is the main goal in several application domains. For instance, in
statistical genetics, it is crucial to identify the interactions among Single
Nucleotide Polymorphisms (SNPs) that contribute to the inheritance of a
disease Behravan et al. (2018); Elgart et al. (2022). Although some
interpretable multimodal methods exist, they are limited to linear
combinations or require resource-intensive post hoc algorithms for
interpretation.

In this paper, we introduce _interpretable tensor fusion_ (InTense), which
jointly learns multimodal neural representations and their interpretable
fusion. InTense provides out-of-the-box interpretability by assigning
relevance scores to all modalities and their interactions. Our approach is
inspired by multiple kernel learning Kloft et al. (2011), a classic kernel-
based approach to interpretable multimodal learning, which we generalize to
deep neural networks and term _Multiple Neural Learning_ (MNL). While both MNL
and InTense provide relevance scores for the modalities, InTense additionally
produces scores for the interactions of modalities. These interaction scores
are made possible through a novel interpretation of neural network weight
matrices: We show that neural networks tend to favor higher-order tensor
products, leading to spurious interpretations (i.e., overstating high-order
interactions between modalities). We resolve this issue by deriving a
theoretically well-founded normalization approach. In the theoretical
analysis, we prove that this produces genuine relevance scores, avoiding
spurious interpretations. In extensive experiments, we empirically validate
the relevance scores on data and show that InTense outperforms existing state-
of-the-art multimodal interpretable approaches in terms of accuracy and
interpretability.

In summary, our contributions are:

  * •

We introduce _Multiple Neural Learning_ (MNL), a theoretically guided
adaptation of the established Multiple Kernel Learning algorithm to deep
learning.

  * •

We introduce InTense, an extension of MNL and tensor fusion designed to
capture non-linear interactions among modalities in an interpretable manner.

  * •

We provide a rigorous theoretical analysis that provides evidence of the
correct disentanglement within our fusion framework.

  * •

We validate our approach through extensive experiments, where we meet the
state-of-the-art classification accuracy while providing robust
interpretability.

##  2 Related Work

We now review existing multimodal learning methods that produce
interpretability scores for the modalities.

##### Interpretable Methods for Learning _Linear_ Combinations of Modalities.

The vast majority of interpretable multimodal learning methods consider linear
combinations of modalities. The arguably most popular instance is Multiple
Kernel Learning (MKL), where kernels from different modalities are combined
linearly. Here, a weight is learned for each kernel determining its importance
in the resulting linear combination of kernels Kloft et al. (2011);
Rakotomamonjy et al. (2008). However, the performance of MKL is limited by the
quality of the kernels. Finding adequate kernels can be especially problematic
for structured high-dimensional data, such as text or images. Addressing this,
several authors have studied combining multiple modalities using neural
networks in a linear manner Poria et al. (2015); Chen et al. (2014); Arabacı
et al. (2021). However, these representations are independently learned to
form basis kernels and later combined in a second step through an SVM or
another shallow learning method. Such independently learned representations
cannot properly capture modality interactions.

##### Methods for Learning _Non-linear_ Combinations of Modalities.

Hessel and Lee Hessel and Lee (2020) map neural representations to a space
defined by a linear combination of the modalities. While they quantify the
overall importance of non-linear interactions, they do not provide scores for
individual modality interactions. Tsai et al. Tsai et al. (2020) introduce
multimodal routing, which is based on dynamic routing Sabour et al. (2017), to
calculate scores for the modality interactions. These scores depend on the
similarity of a modality’s representation to so-called concept vectors, where
one such vector is defined for each label. However, routing does not
distinguish between linear and non-linear combinations and is thus misled by
partially redundant information in the combinations. Indeed, we show through
experiments (see Section 4) that the non-linear combinations learned by
routing are incorrectly overestimated. Gat et al. Gat et al. (2021) propose a
method to obtain modality relevances by computing differences of accuracies on
a test set and a permuted test set. However, this method has limited
interpretability and requires multiple forward passes through the trained
network to obtain relevance scores. Wörtwein et al. Wörtwein et al. (2022)
learn an aggregated representation for unimodal, bimodal, and trimodal
interactions, respectively. However, their method does not learn fine-grained
relevance scores for the various combinations of modalities. Alongside methods
offering limited interpretability, there exist methods that non-linearly
combine modalities without adding any interpretability Zhang et al. (2023);
Liang et al. (2021); Tan and Bansal (2019).

In summary, none of these methods learns proper relevance scores of
interactions between modalities.

##### Post-hoc Explanation Methods.

There exist several methods for post-hoc explanation of multimodal learning
methods Gat et al. (2021); Chandrasekaran et al. (2018); Park et al. (2018);
Kanehira et al. (2019); Cao et al. (2020); Frank et al. (2021). These methods
consist of two steps: first, training a multimodal model that is not
inherently interpretable, followed by the calculation of relevance scores in
hindsight. However, their two-step nature makes these methods challenging to
analyze theoretically. Moreover, since the initial model disregards
interpretability, it may lead to inherent limitations in the explanatory
process. Additionally, these methods come with the added computational burden
of producing relevance scores. Another limitation is their applicability,
which is confined to specific types of modalities.

##  3 Methodology

In the following sections, we introduce several components comprising our
approach. First, we review the classical $L_{p}$-norm Multiple Kernel Learning
(MKL)framework Kloft et al. (2011), which we extend to Multiple Neural
Learning. Subsequently, we propose Interpretable Tensor Fusion (InTense),
which captures non-linear modality interactions. Furthermore, we show how
InTense learns disentangled neural representations, thereby computing correct
relevance scores.

###  3.1 Preliminaries

We consider a dataset $\\{(x_{i},y_{i})\\}_{i=1}^{n}$ with labels
$y_{i}\in\\{-1,1\\}$. The inputs have $M$ modalities, where
$x_{i}^{m}\in\mathcal{X}^{m}$ for $m\in\\{1,\ldots,M\\}$ denotes the $m^{th}$
modality of the datapoint $x_{i}$, and $\mathcal{X}^{m}$ is the input space
associated with the modality $m$. In MKL, one considers kernel mixtures of the
form $k(u,v)=\sum_{m=1}^{M}\beta_{m}k_{m}(u,v),$ where $k_{m}(u,v)$ is a base
kernel and $\beta_{m}\geq 0$ for all $m$. Imposing an $L_{p}$-norm constraint
on the vector $\beta\in\mathbb{R}^{M}$ gives rise to the following classic
optimization problem:

$\operatorname*{minimize}_{\begin{subarray}{c}w_{1},w_{2},\cdots,w_{L},\beta\\\
\beta\in{\mathbb{R}}^{M},\beta\geq 0,\|\beta\|_{p}\leq
1\end{subarray}}\Bigg{(}\operatorname{\sum_{i=1}^{n}}\ell\Big{(}\sum_{m=1}^{M}\sqrt{\beta_{m}}\langle
w_{m},\Psi_{m}(x_{i}^{m})\rangle_{\mathcal{H}^{m}}\\\
+b,y_{i}\Big{)}+\frac{\lambda}{2}\sum_{m=1}^{M}\|w_{m}\|_{L^{2}(\mathcal{H}^{m})}^{2}\Bigg{)},\;$
(1)

where $\ell$ is a loss function, and
$\Psi_{m}:\mathcal{X}^{m}\rightarrow\mathcal{H}^{m}$ are feature maps from the
input space $\mathcal{X}^{m}$ to the Hilbert space $\mathcal{H}^{m}$
associated with kernel $k_{m}$ such that for each $m\in\\{1,2,\ldots,M\\}$ and
$u,v\in\mathcal{X}^{m}$,
$k_{m}(u,v)=\langle\Psi_{m}(u),\Psi_{m}(v)\rangle_{\mathcal{H}^{m}}$, where
$\langle\cdot,\cdot\rangle_{\mathcal{H}}^{m}$ denotes the inner-product
associated with the Hilbert space $\mathcal{H}^{m}$. The base kernels $k_{m}$
are assumed to be fixed functions. This is in sharp contrast to Multiple
Neural Learning (MNL)introduced in the next section, where each feature map
$\Psi_{m}(x)$ is learned from the data.

###  3.2 Multiple Neural Learning

In this section, we propose Multiple Neural Learning (MNL), an interpretable
method for linear combination of modalities. In MNL, we train a neural network
composed of two components: 1) modality subnetworks that output a neural
representation for each modality and 2) a linear fusion layer that combines
the representations in an interpretable manner. We define the optimization
problem as:

$\operatorname*{minimize}_{\begin{subarray}{c}w_{L}^{1},\cdots,w_{L}^{M},\beta,\\\
W^{1},W^{2},\cdots,W^{M},\\\ \beta\in{\mathbb{R}}^{M},\beta\geq
0,\|\beta\|_{p}\leq
1\end{subarray}}\Bigg{(}\sum_{i=1}^{n}\ell\Big{(}\sum_{m=1}^{M}\sqrt{\beta_{m}}\langle
w_{L}^{m},f^{m}(x_{i}^{m})\rangle\\\\[-20.0pt]
+b,y_{i}\Big{)}+\Lambda\sum_{l=1}^{L}\sum_{m=1}^{M}\|w^{m}_{l}\|_{2}^{2}\Bigg{)},$
(2)

where $f^{m}~{}$is the $m^{th}$ modality’s subnetwork composed of $L-1$ layers
with weights $W^{m}=\\{w^{m}_{1},\ldots,w^{m}_{L-1}\\}$. A representation for
the $i^{th}$ data point’s $m^{th}$ modality is obtained by $f^{m}(x_{i}^{m})$.
The fusion layer $L$ with weights $w_{L}^{1},\dots,w_{L}^{M}$ learns a linear
combination of the modality representations. $\Lambda$ and $p\;(1\leq
p<\infty)$ are hyperparameters and
$\ell(t,y)=-\log\left(\frac{\exp(ty)}{1+\exp(ty)}\right)$ is the cross-entropy
loss function. This setup can also be seen as additive fusion because it
represents a linear combination of the modalities with weights
$\sqrt{\beta_{m}}$.

Notably, $\sqrt{\beta_{m}}$ is a positive weight for the $m$’th modality,
indicating its relevance score. The vector $\beta$ is simultaneously optimized
with the network weights. However, the constraints on $\beta$ introduce an
increased difficulty in optimizing equation 2. The following theorem presents
a simplified optimization problem by eliminating $\beta$ from equation 2 along
with a method to retrieve $\beta$ from the learned weights.

######  Theorem 1.

The optimization problem in equation 2 is equivalent to the following problem,
where the parameters $\beta$ are no longer present:

$\displaystyle\operatorname*{minimize}_{\begin{subarray}{c}w_{L}^{1},w_{L}^{2},\ldots,w_{L}^{M},\\\
W^{1},W^{2},\cdots,W^{M}\end{subarray}}\sum_{i=1}^{n}\ell\left(\sum_{m=1}^{M}\langle
w_{L}^{m},f^{m}(x^{m}_{i})\rangle+b,y_{i}\right)$
$\displaystyle+\Lambda\sum_{l=1}^{L-1}\sum_{m=1}^{M}\|w^{m}_{l}\|_{2}^{2}+\Lambda\left(\sum_{m=1}^{M}\|w^{m}_{L}\|_{2}^{q}\right)^{\frac{2}{q}},$
(3)

where $q=\frac{2p}{p+1}$ (and therefore $1\leq q\leq 2$). The corresponding
values of relevance score $\beta$ can be recovered after the optimization as:

$\displaystyle\beta_{m}=\frac{\|w^{m}_{L}\|_{2}^{\frac{2}{p+1}}}{\left(\sum_{\tilde{m}=1}^{M}\|w^{\tilde{m}}_{L}\|_{2}^{\frac{2p}{p+1}}\right)^{\frac{1}{p}}}.$
(4)

The theorem states that the relevance of a modality in our jointly trained
network can be obtained by applying a suitable $p-$norm over the weights of
the fusion layer $L$. A detailed proof of the theorem can be found in Appendix
A. The central idea is observing that the parameters $\beta$ can be absorbed
into the weights $w_{L}^{m}$, pushing $\beta$ into the regularization term.
Subsequently, by showing that $\beta$ can be minimized independently from the
weights, the optimal value is attained through equation 4. Absorbing $\beta$
in fusion weights in turn introduces the additional block $L^{q}$ norm
regularization term
$\Lambda(\sum_{m=1}^{M}\|w^{m}_{L}\|_{2}^{q})^{\frac{2}{q}}$.

##### Correct Relevance Scores Through Normalization.

In pre-experiments (see Appendix F.1) we found that the relevance scores can
be misleading, especially when the network outputs high activation values for
some modalities. We address this issue with proper normalization techniques.
We propose an adaptation of the standard Batch Normalization (batch norm)
Ioffe and Szegedy (2015), which we call Vector-wise Batch Normalization (VBN).
VBN ensures that the $L_{2}$-norm of the activation values, averaged over a
mini-batch, is constant. Let $B$ be a mini-batch of datapoints’ indices. We
define VBN as:

$\displaystyle\tilde{f}^{m}(x_{i}^{m})=\frac{f^{m}(x^{m}_{i})-\mu_{B,m}}{\sigma_{B,m}},\text{where}$
(5) $\displaystyle\mu_{B,m}=\frac{\sum_{i\in
B}f^{m}(x^{m}_{i})}{|B|},\text{and}$ (6)
$\displaystyle\sigma_{B,m}^{2}=\frac{\sum_{i\in
B}\|f^{m}(x^{m}_{i})-\mu_{B,m}\|_{2}^{2}}{|B|}.$ (7)

The mean $\mu_{B,m}$ is computed element-wise as in the standard batch norm.
However, in equation 7, instead of computing the variance element-wise, we
calculate the average of the squared $L_{2}-$norm of a modality representation
across the mini-batch. Unlike batch norm, we do not shift and scale the
representations element-wise after the normalization step. Using VBN, the loss
in equation A changes to:

$\sum_{i=1}^{n}\ell\left(\sum_{m=1}^{M}\left\langle
w_{L}^{m},\tilde{f}^{m}(x^{m}_{i})\right\rangle+b,y_{i}\right).$

Note that VBN is applied after the activation function to obtain
$\tilde{f}^{m}(x^{m})$. We found empirically that proper normalization is
crucial for MNL to achieve competitive performance.

###  3.3 Interpretable Tensor Fusion

In this section, we propose Interpretable Tensor Fusion (InTense), an
extension of MNL that additionally produces scores for interactions (non-
linear combinations) of modalities. InTense is based on tensor fusion, which
captures multiplicative interactions among modalities by computing a tensor
product over the individual modality representations Zadeh et al. (2017).
InTense operates as follows. For a dataset with $M$ modalities, we consider
interactions up to a given order of $D$, where $D\leq M$. An order of $D$
implies interaction among $D$ modalities. A multiplicative interaction of
modalities is defined by a subset $I\in\mathcal{I}$, where
$\mathcal{I}=\\{J\subset\\{1,\dots,M\\}:|J|\leq D\\}$, and a tensor product
$f^{I}(x):=f^{I_{1}}\otimes f^{I_{2}}\otimes\ldots f^{I_{|I|}}$, where
$f^{I_{m}}$ is the representation of modality $I_{m}$, and $\otimes$ denotes
the tensor product operator. Analogously to equation 2, we obtain a new
objective:

$\operatorname*{minimize}_{\begin{subarray}{c}w_{L}^{I},\beta_{I}:I\in\mathcal{I},\\\
\|\beta\|_{p}\leq 1,\beta_{I}\geq 0\\\
W^{1},W^{2},\cdots,W^{M}\end{subarray}}\Bigg{(}\sum_{i=1}^{n}\ell\Big{(}\sum_{I\subset\mathcal{I}}\sqrt{\beta_{I}}\langle
w_{L}^{I},f^{I}(x)\rangle+b,y_{i}\Big{)}\\\\[-20.0pt]
+\Lambda\sum_{l=1}^{L}\sum_{m=1}^{M}\|w^{m}_{l}\|_{2}^{2}\Bigg{)}$ (8)

This optimization problem can be seen as a special case of MNL, where the
multiplicative interactions are treated as separate modalities. Therefore, in
combination with equation 8, Theorem 1 computes the relevance scores for all
modalities and their interactions.

####  3.3.1 What Can Go Wrong?

In our experiments with synthetic multimodal datasets (Section 4.1), we found
that relevance scores of higher-order interactions are greatly overestimated.
Scores can be high even when no true interactions exist in the data. We call
this phenomenon higher-order interaction bias. The bias is caused by higher-
order tensor products corresponding to very large function classes, which
approximately include the function classes corresponding to lower-order
tensors as subsets.

Indeed, it is possible that a linear combination of the components of a tensor
product learns the same functions as a linear combination of the individual-
modality representations. For instance, consider two modalities
$(m_{u},m_{v})$ and their representation vectors as
$\mathbf{u},\mathbf{v}\in\mathbb{R}^{3}$. Assume the first component of the
learned representations is constant (e.g., 1), i.e.,
$\mathbf{u}=[1,u_{2},u_{3}]^{\intercal}$ and
$\mathbf{v}=[1,v_{2},v_{3}]^{\intercal}$. In such a scenario, the linear
combination $\alpha_{1}u_{2}+\alpha_{2}v_{2}$ (for
$\alpha_{1},\alpha_{2}\in\mathbb{R}$) can also be represented as
$\alpha_{1}(\mathbf{u}\otimes\mathbf{v})_{2,1}+\alpha_{2}(\mathbf{u}\otimes\mathbf{v})_{1,2}.$
Using the MNL algorithm, the relevance scores for modalities $m_{u}$ and
$m_{v}$ are $\alpha_{1}$ and $\alpha_{2}$ respectively. However, the relevance
score for the modality with tensor product ($m_{u\otimes v}$) is
$\sqrt{(\alpha_{1})^{2}+(\alpha_{2})^{2}}$. Here, the $L^{p}$-norm
regularization with any $p<2$ will favor the representation $m_{u\otimes v}$.
Therefore, if the dimensions of the modality representations $\mathbf{u}$ and
$\mathbf{v}$ are strictly greater than required to represent the ground truth
(which is usually the case in modern networks), lower-order functions will be
preferably represented inside the higher-order products by learning a constant
in the representations of each modality. Our experiments show that a trained
network typically exhibits such behavior. We propose a solution to this
problem of higher-order interaction bias in the rest of this section.

##### Correct Bi-modal Interactions.

We now address the problem of higher-order interaction bias. The key idea is
to introduce a normalization scheme that downweights higher-order
interactions. Our normalization scheme is a sophisticated generalization of
the Vector-wise Batch Normalization (VBN)scheme described in equation 5. Let
$m_{1}$ and $m_{2}$ be two modalities and define their representations as
$f^{m_{1}}$ and $f^{m_{2}}$. The representation of the bi-modal interaction is
defined as $f^{\\{m_{1},m_{2}\\}}=f^{m_{1}}\otimes f^{m_{2}}$. In this simple
bi-modal case, our solution can be summarized as follows: we apply VBN to
$f^{m_{1}}$ and $f^{m_{2}}$ before taking the product, and finally apply VBN
again to the result. Formally, normalize each modality representation
according to equation 5 as:

$\tilde{f}^{m}(x_{i})=\frac{f^{m}(x_{i})-\mathbb{E}(f^{m}(x_{i}))}{\sqrt{\mathbb{E}\left(\|f^{m}(x_{i})-\mathbb{E}(f^{m}(x_{i}))\|^{2}_{2}\right)}},$

then compute the tensor-product as:

$\hat{f}^{\\{m_{1},m_{2}\\}}(x_{i})=\tilde{f}^{m_{1}}\otimes\tilde{f}^{m_{2}},$

and similarly apply VBN to the tensor-product to obtain
$\tilde{f}^{\\{m_{1},m_{2}\\}}$.

The centering step of normalization is applied element-wise over a mini-batch.
Thus, if a few components were to be non-zero constants in a mini-batch, they
would become zero after the normalization. This ensures that
$\tilde{f}^{m_{1},m_{2}}$ cannot easily access lower degree information
contained in $\tilde{f}^{m_{1}}$, because elements in $\tilde{f}^{m_{2}}$
cannot be a non-zero constant, and vice versa. The normalization could
seemingly be trivially extended to more than two modalities by applying the
normalization iteratively up to an n-order tensor product. However, such an
extension may still lead to a high-interaction bias. We illustrate why such a
trivial extension may not work for more than two modalities and later
generalize the normalization scheme for any number of modalities.

##### Generalization Over M-modal Interactions.

Extending the aforementioned normalization to cases where $D>2$ is not
straightforward. This complexity arises because, when fusing more than two
modalities, potentially, the representations of a subset of $M$ modalities
conspire to produce a constant even though each individual modality
representation is non-constant. For instance, consider three modalities with
one-dimensional representations and apply VBN to the representations $f^{1}$,
$f^{2}$, $f^{3}$, then to $f^{1,2}$, and finally to
$f^{1,2,3}=\tilde{f}^{1,2}\otimes\tilde{f}^{3}$, it is still possible that
representation $f^{1}$ is learned in the higher-order tensor product. For
instance, assume the components of $f^{2}$ and $f^{3}$ are learned to satisfy
the following for each datapoint: (1) $f^{2}=f^{3}$; (2) we have that $f^{2}$
is a Rademacher variable ($P[f^{2}=1]=P[f^{2}=-1]=0.5$); and (3) $f^{2}$ is
independent of $f^{1}$. Since $f^{1,2,3}=f^{1}f^{2}f^{3}$ and $f^{2}f^{3}=1$
for all datapoints, we actually have $f^{1,2,3}=f^{1}$, and this seemingly
higher order combination can still recover the first modality.

We address this issue by carefully normalizing the modalities features. The
key is to prevent the combination of features of one or more modalities from
resulting in a constant value. Similar to the bi-modal scenario, we need to
ensure that the contribution of a subset of modalities to the larger fusion
set is, on average, zero. This guarantees that no constant value, other than
zero, is multiplied by the product of the complement of that subset within the
original fusion set. Formally, for each $I\subset\\{1,2,\ldots,M\\}$, the
centering step of our batch norm procedure is defined as follows, where we
first assume each modality is one-dimensional to simplify the exposition:

$\displaystyle\hat{f}^{I}$
$\displaystyle=\sum_{\ell=0}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},\ldots,S^{\ell}\subset I\atop
S^{1},\ldots,S^{\ell}\text{disjoint}}\prod_{m\in I\setminus(\cup S^{k})}f^{m}$
$\displaystyle\;\prod_{k\in\\{1,2,\ldots,\ell\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right).$ (9)

When the modalities are multi-dimensional, the above operation is applied
independently to each multi-index component 111In particular, the multivariate
case could be expressed with a similar formula as equation 3.3.1 with the
products replaced by outer tensor products, but this would require a different
reordering of the components for each term of the sum.. After preforming the
centering step above for each multi-index component, we perform the
generalized normalization step as follows

$\displaystyle\tilde{f}^{I}=\frac{\hat{f}^{I}}{\sqrt{\mathbb{E}\|\hat{f}^{I}\|_{\operatorname{Fr}}^{2}}}.$
(10)

While the solution can no longer be easily interpreted as a composition of
standard batch norm operations, it is, in fact, possible to show that lower-
order fusion can not be represented by a linear combination of their higher-
order counterparts. Theorem 2 formalizes this result.

######  Theorem 2.

The centering step described in equation 3.3.1 can be represented as the multi
variate polynomial:

$\displaystyle\sum_{J\subset I}\mathcal{G}_{J}\prod_{m\in J}f^{m},$

for some real coefficients $\mathcal{G}_{J}$. Furthermore, the expected
contribution of a subset of modalities $J$ in the fusion of the set of
modalities $I$, where $J\subsetneq I$ is zero. That is, we have for any
$J\subsetneq I$ (including the empty set),

$\displaystyle\mathbb{E}\left(\sum_{K:J\subset K\subset
I}\mathcal{G}_{K}\prod_{m\in K\setminus J}f^{m}\right)=0.$ (11)

The theorem states that the expected value of the contribution of any subset
$J\subsetneq I$ of modalities is zero in the fusion of $I$. Thus,
$\tilde{f}^{I}$ (higher-order) can not learn a linear combination of the
$\tilde{f}^{J}$ (lower-order). Appendix B contains a comprehensive proof of
the theorem.

To make the exposition clearer, we provide as an example the case
$I=\\{1,2,3\\}$ and the individual representations $f^{m}$ are standardized
using VBN. In this case, we have, using the notation
$\overline{f^{1}f^{2}}=\mathbb{E}(f^{1}f^{2})$:

$\displaystyle\hat{f}^{1,2,3}$ $\displaystyle=f^{1}\times f^{2}\times
f^{3}-\overline{f^{1}\times f^{2}}\times f^{3}-\overline{f^{1}\times
f^{3}}\times f^{2}$ $\displaystyle-\overline{f^{2}\times f^{3}}\times
f^{1}-\overline{f^{1}\times f^{2}\times f^{3}}.$

An elaborated centering step, without normalizing the individual modality
representations, and strictly following equation 3.3.1 is described in
Appendix B.1.

In this section, we introduced _Iterative Batch Normalization (IterBN)_, a
normalization scheme addressing higher-order interaction bias in multimodal
learning. In the next section, we show the effectiveness of our method on
synthetic and real-world datasets.

##  4 Experiments

First, we experiment on synthetic data, where we control the amount of
relevant information in the modalities, and compare InTense’s relevance scores
to the established ground truth. Second, we compare the predictive performance
of InTense with popular multimodal fusion methods on six real-world multimodal
datasets.

Figure 2: An excerpt of three modalities of SynthGene, our self-curated binary
classification dataset, where each sequence is made from a set of letters
{A,C,G,T}. A positive class-sequence “TCG” and a negative class-sequence “AGC”
is added according to the probability $p_{m}$.

###  4.1 Evaluating the Relevance Scores

We created a multimodal dataset where each modality of a datapoint is a
sequence of letters chosen randomly from a predefined set. For each datapoint
$x$ and modality $m$, an informative subsequence is inserted at a random
position with a probability of $p_{m}$. We call our dataset SynthGene. More
details about it can be found in Appendix C.

We perform two experiments to determine the correctness of the relevance
scores obtained from InTense. First, we construct a binary classification
dataset with labels that ensure the modalities are independent and do not
interact. Second, we generate another set of labels that can only be predicted
using non-linear interactions among the modalities.

####  4.1.1 InTense Assigns Correct Relevance Scores to _Independent_
Modalities

In this set of experiments, we create a synthetic dataset with independent
modalities (i.e., without interactions). As a baseline, we train one model for
each modality and then compare the accuracies of those models with the
relevance scores of InTense trained on all modalities together.

##### Dataset.

We create the SynthGene dataset, where for each modality $m$, each datapoint
with a positive/negative class label contains a class-specific sequence with a
probability $p_{m}$ independent of other modalities. A high value of $p_{m}$
indicates that most sequences in the modality $m$ contain a class-specific
sequence. Thus, the higher the value of $p_{m}$, the more relevant the
modality $m$ becomes. The labels for all datapoints are uniformly distributed
between the two classes. Figure 2 shows how probability $p_{m}$ affects
modality relevance. We use $10$ modalities where the informative subsequence
is inserted into modalities $M_{2}$, $M_{4}$, and $M_{7}$. There is no
discriminative information present in other modalities.

##### Results.

Figure 3 shows the relevance scores calculated by InTense on SynthGene.
InTense assigns the correct relevance scores as they align with human
intuition. The higher the probability $p_{m}$, the more informative signal is
contained in modality $m$, and the higher the predicted relevance score. We
further validate the correctness of InTense’s interpretability by comparing it
with the accuracies obtained from unimodal models trained on each modality
separately. Again, InTense’s relevance scores correlate with the unimodal
accuracies.

Figure 3: The figure shows a high correlation of InTense’ relevance scores and
accuracies of unimodal models on SynthGene. The modalities $M_{2},M_{4},M_{7}$
achieve high relevance scores and high accuracy as they contain class-specific
information. Other modalities contain no class-specific information, which
leads to a very low relevance score and an accuracy of around $50\%$
(equivalent to random guessing).

####  4.1.2 InTense Assigns Correct Relevance Scores to _Interacting_
Modalities

We now turn to a situation where the label depends on a non-linear interaction
among the modalities by design.

##### Dataset.

We also create the SynthGene-tri dataset, a trimodal version of SynthGene.
However, this time, the informative subsequence is not class-specific.
Instead, the label is defined by an exclusive-or (XOR) relationship between
the first two modalities ($M_{1}$ and $M_{2}$). The label is $0$ if both
modalities contain the subsequence or none of them does, and the label is $1$
otherwise (i.e., when one of the modalities contains the subsequence). Note
that modality $M_{3}$ does not contain any informative subsequence; thus, it
is irrelevant. As before, we generate a balanced dataset with 50% of the
samples being positive and 50% negative.

##### Results.

Figure 4 shows the results. We observe that the global relevance scores
calculated by the MultiRoute baseline Tsai et al. (2020) are biased toward
high-order interactions. This occurs even when the interactions do not add any
useful information and thus should have been discarded by MultiRoute. We
further see that the proposed InTense method does avoid this bias and
correctly assigns a high relevance score solely to the interaction of $M_{1}$
and $M_{2}$. This shows that InTense can ensure the correctness of relevance
scores even when higher-order interactions are involved.

Figure 4: Illustration of the relevance scores calculated by the proposed
InTense and the MultiRoute baseline when higher-order modality interactions
are involved in the ground truth. MultiRoute leads to biased results (blue
bars), where the relevance scores are concentrated toward higher-order
interactions $M_{1\otimes 2\otimes 3}$. In contrast, InTense (orange bars)
correctly assigns a high relevance score only to the interaction $M_{1\otimes
2}$, which contains all class-specific signals.

###  4.2 InTense Performs SOTA in Real-World Applications

We demonstrate the effectiveness of InTense in providing interpretability
without compromising predictive performance across a range of real-world
applications. In order to compare performance and ensure reproducibility, we
followed the experimental setup (e.g., data preprocessing, encodings of
different modalities) of the MultiBench Liang et al. (2021) benchmark for all
the experiments.

##### Sentiment analysis.

In sentiment analysis, also known as opinion mining, the target is to identify
the emotional tone or feeling underlying the data. Initially confined to text
data, sentiment analysis has evolved to encompass multiple modalities. The
task becomes challenging due to the intricate interactions of modalities,
which play a significant role in expressing sentiments. Understanding
sentiments is crucial in business intelligence, customer feedback analysis,
and social media monitoring. To evaluate InTense in sentiment analysis, we
employed CMU-MOSEI Bagher Zadeh et al. (2018), the largest dataset of
sentence-level sentiment analysis for real-world online videos, and CMU-MOSI
Zadeh et al. (2016), a collection of annotated opinion video clips.

##### Humor and Sarcasm detection.

Humor detectors identify elements that evoke amusement or comedy, while
sarcasm detection aims to discern whether a sentence is presented in a
sarcastic or sincere manner. Sarcasm and humor are often situational.
Successfully detecting them requires a comprehensive understanding of various
information sources, encompassing the utterance, contextual intricacies of the
conversation, and background of the involved entities. As this information
extends beyond textual cues, the challenge lies in learning the complex
interactions among the available modalities. To assess our approach’s
effectiveness in these tasks, we utilized UR-FUNNY Hasan et al. (2019) for
humor detection and MUStARD Castro et al. (2019) for sarcasm detection.

##### Layout Design Categorization.

Layout design categorization is about classifying graphical user interfaces
into predefined categories. Automizing this task can support designers in
optimizing the arrangement of interactive elements, ensuring the creation of
interfaces that are not only visually appealing but also functional and user-
centric. Classifiers can, e.g., assign semantic captions to elements, enable
smart tutorials, or be the foundation for advanced search engines. For this
paper, we considered the ENRICO Leiva et al. (2020) dataset as an example for
layout design categorization. ENRICO comprises 20 design categories and 1460
user interfaces with five modalities, including screenshots, wireframe images,
semantic annotations, DOM-like tree structures, and app metadata.

##### Digit Recognition.

We also include results for Audiovision-MNIST (AV-MNIST) Vielzeuf et al.
(2018), a multimodal dataset comprising images of handwritten and recordings
of spoken digits. Despite its apparent lack of immediate real-world
application, the dataset’s significance lies in its establishment as a
standard multimodal benchmark. It allows us to situate our research within the
broader context of previous research Pérez-Rúa et al. (2019); LeCun et al.
(1998).

| Baselines | Ours
---|---|---
| MultiRoute | MRO | MNL | InTense
MUStARD | $65.9$ | $66.5$ | $67.4$ | $\mathbf{69.6}$
CMU-MOSI | $76.8$ | $75.8$ | $\mathbf{80.8}$ | $79.7$
UR-FUNNY | $63.6$ | $63.4$ | $63.4$ | $\mathbf{65.1}$
CMU-MOSEI | $80.2$ | $79.7$ | $80.5$ | $\mathbf{81.5}$
AV-MNIST | 71.8 | 72.0 | 72.4 | 72.8
ENRICO | 46.7 | 49.2 | 47.1 | 50.8
Table 1: Accuracies for different baselines on the test fold. Each experiment
is carried out ten times to compute the statistics. Figure 5: Relevance scores
from InTense for audio (a), vision (v), text (t), and all their possible
interactions.

##### Baselines.

We compare the classification performance of InTense and MNL to the following
state-of-the-art interpretable multimodal learning baselines: 1) Multimodal
Residual Optimization (MRO) Wörtwein et al. (2022) and 2) Multimodal Routing
(MultiRoute) Tsai et al. (2020). Additionally, we consider three non-
interpretable baselines: 1) LF-Concat, 2) TF Network Zadeh et al. (2017), and
3) multimodal transformer (MulT) Tsai et al. (2019). These baselines have been
identified as leading in the independent comparison conducted by Liang et al.
Liang et al. (2021). The multimodal transformer was particularly highlighted
for consistently reaching some of the highest accuracy levels.

##### Results.

The results are shown in Table 1. We observe that our proposed models, MNL and
InTense, surpass all interpretable baselines in terms of classification
accuracy. InTense achieves the highest performance across all datasets except
for CMU-MOSI, where MNL excels. CMU-MOSI is the smallest dataset in our
analysis, a factor that may contribute positively to MNL’s performance.
Compared with non-interpretable multimodal learning methods (see Table 1 in
Appendix E), MNL and especially InTense demonstrate impressive performance,
almost meeting the accuracy of the non-interpretable Multimodal Transformer
(MulT). The performance of our proposed models is within a narrow 2% error
margin (and frequently much lower) compared to the MulT baseline.

Figure 5 shows the interpretable relevance scores that InTense assigns to the
various modalities. Notably, in three of the four datasets analyzed, text
emerges as the most significant modality. We identify two plausible
explanations for this phenomenon. First, several studies have reported a
strong correlation of text with sentiment Gat et al. (2021). Second, the
predominance of the text modality may be attributed to the availability of
sophisticated word embeddings obtained from large pre-trained foundation
models. However, we find an exception in the interpretability scores for the
sarcasm detection dataset (MUStARD). Sarcasm detection requires information
from multiple modalities, making sole reliance on one, especially text,
insufficient for accuracy.

##  5 Conclusion

We introduced InTense, a novel interpretable approach to multimodal learning
that offers reliable relevance scores for modalities and their interactions.
InTense achieves state-of-the-art performance in several challenging
applications, from sentiment analysis and humor detection to layout design
categorization and multimedia. We proved theoretically and validated
empirically that InTense correctly disentangles higher-order interactions from
the individual effects of each modality. The full transparency of InTense
makes it suitable for future application in safety-critical domains.

##  6 Broader Impact

As an interpretable approach, the proposed methodology naturally aids in
making multimodal learning more transparent. By attributing importance scores
to different modalities and their interactions, InTense may reveal biases in
decision-making and improve trustworthiness. For instance, consider a system
tasked with classifying loan suitability. Our approach may expose social
biases when relevance scores for certain modalities, such as gender extracted
from vision, are disproportionately high. Moreover, unlike existing
approaches, InTense has no higher-order interaction bias (see Section 3.3).
That is, it does not incorrectly assign large relevance scores to higher-order
interactions, which can create the false impression of a social bias. The full
transparency of InTense prevents the deployment of a harmful classification
model, contributing to the ethical use of AI in sensitive domains.

##  7 Acknowledgement

SV, PL, WM, and MK acknowledge support by the Carl-Zeiss Foundation, the DFG
awards KL 2698/2-1, KL 2698/5-1, KL 2698/6-1 and KL 2698/7-1, and the BMBF
awards 03|B0770E, and 01|S21010C.

## References

  * Arabacı et al. [2021] Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, and Alptekin Temizel.  Multi-modal egocentric activity recognition using multi-kernel learning.  Multimedia Tools and Applications, 80(11):16299–16328, 2021.
  * Bagher Zadeh et al. [2018] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * Baltrušaitis et al. [2016] Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency.  Openface: an open source facial behavior analysis toolkit.  In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1–10. IEEE, 2016.
  * Behravan et al. [2018] Hamid Behravan, Jaana M Hartikainen, Maria Tengström, Katri Pylkäs, Robert Winqvist, Veli-Matti Kosma, and Arto Mannermaa.  Machine learning identifies interacting genetic variants contributing to breast cancer risk: A case study in finnish cases and controls.  Scientific reports, 8(1):13149, 2018.
  * Büchel et al. [1998] Christian Büchel, Cathy Price, and Karl Friston.  A multimodal language region in the ventral visual pathway.  Nature, 394(6690):274–277, 1998.
  * Cao et al. [2020] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 565–580. Springer, 2020.
  * Castro et al. [2019] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria.  Towards multimodal sarcasm detection, 2019.
  * Chagas et al. [2020] Paulo Chagas, Luiz Souza, Ikaro Araújo, Nayze Aldeman, Angelo Duarte, Michele Angelo, Washington LC Dos-Santos, and Luciano Oliveira.  Classification of glomerular hypercellularity using convolutional features and support vector machine.  Artificial intelligence in medicine, 103:101808, 2020.
  * Chandrasekaran et al. [2018] Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh.  Do explanations make vqa models more predictable to a human?  arXiv preprint arXiv:1810.12366, 2018.
  * Chen et al. [2014] JunKai Chen, Zenghai Chen, Zheru Chi, and Hong Fu.  Emotion recognition in the wild with feature fusion and multiple kernel learning.  In Proceedings of the 16th International Conference on Multimodal Interaction, pages 508–513, 2014.
  * De la Torre and Cohn [2011] Fernando De la Torre and Jeffrey F Cohn.  Facial expression analysis.  Visual analysis of humans: Looking at people, pages 377–409, 2011.
  * Degottex et al. [2014] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer.  Covarep—a collaborative voice analysis repository for speech technologies.  In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 960–964. IEEE, 2014.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Elgart et al. [2022] Michael Elgart, Genevieve Lyons, Santiago Romero-Brufau, Nuzulul Kurniansyah, Jennifer A Brody, Xiuqing Guo, Henry J Lin, Laura Raffield, Yan Gao, Han Chen, et al.  Non-linear machine learning models incorporating snps and prs improve polygenic prediction in diverse human populations.  Communications Biology, 5(1):856, 2022.
  * Frank et al. [2021] Stella Frank, Emanuele Bugliarello, and Desmond Elliott.  Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers.  arXiv preprint arXiv:2109.04448, 2021.
  * Gat et al. [2021] Itai Gat, Idan Schwartz, and Alex Schwing.  Perceptual score: What data modalities does your model perceive?  Advances in Neural Information Processing Systems, 34:21630–21643, 2021.
  * Hasan et al. [2019] Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al.  Ur-funny: A multimodal language dataset for understanding humor.  arXiv preprint arXiv:1904.06618, 2019.
  * He et al. [2020] Zhipeng He, Zina Li, Fuzhou Yang, Lei Wang, Jingcong Li, Chengju Zhou, and Jiahui Pan.  Advances in multimodal emotion recognition based on brain–computer interfaces.  Brain sciences, 10(10):687, 2020.
  * Hessel and Lee [2020] Jack Hessel and Lillian Lee.  Does my multimodal model learn cross-modal interactions? it’s harder to tell than you might think!  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 861–877, 2020.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * Kanehira et al. [2019] Atsushi Kanehira, Kentaro Takemoto, Sho Inayoshi, and Tatsuya Harada.  Multimodal explanations by predicting counterfactuality in videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8594–8602, 2019.
  * Kloft et al. [2011] Marius Kloft, Ulf Brefeld, Sören Sonnenburg, and Alexander Zien.  Lp-norm multiple kernel learning.  The Journal of Machine Learning Research, 12:953–997, 2011.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * Leiva et al. [2020] Luis A Leiva, Asutosh Hota, and Antti Oulasvirta.  Enrico: A dataset for topic modeling of mobile ui designs.  In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 1–4, 2020.
  * Liang et al. [2018] Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion.  In First Workshop and Grand Challenge on Computational Modeling of Human Multimodal Language, volume 113, pages 116–125, 2018.
  * Liang et al. [2021] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al.  Multibench: Multiscale benchmarks for multimodal representation learning.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * Lunghi et al. [2019] Giacomo Lunghi, Raul Marin, Mario Di Castro, Alessandro Masi, and Pedro J Sanz.  Multimodal human-robot interface for accessible remote robotic interventions in hazardous environments.  IEEE Access, 7:127290–127319, 2019.
  * Micchelli et al. [2005] Charles A Micchelli, Massimiliano Pontil, and Peter Bartlett.  Learning the kernel function via regularization.  Journal of machine learning research, 6(7), 2005.
  * Park et al. [2018] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach.  Multimodal explanations: Justifying decisions and pointing to the evidence.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8779–8788, 2018.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * Pérez-Rúa et al. [2019] Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie.  Mfas: Multimodal fusion architecture search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6966–6975, 2019.
  * Poria et al. [2015] Soujanya Poria, Erik Cambria, and Alexander Gelbukh.  Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2539–2544, 2015.
  * Rakotomamonjy et al. [2008] Alain Rakotomamonjy, Francis Bach, Stéphane Canu, and Yves Grandvalet.  Simplemkl.  Journal of Machine Learning Research, 9:2491–2521, 2008.
  * Sabour et al. [2017] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton.  Dynamic routing between capsules.  Advances in neural information processing systems, 30, 2017.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * Tsai et al. [2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
  * Tsai et al. [2020] Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Multimodal routing: Improving local and global interpretability of multimodal language analysis.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2020, page 1823. NIH Public Access, 2020.
  * Vielzeuf et al. [2018] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie.  Centralnet: a multilayer approach for multimodal fusion.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * Wörtwein et al. [2022] Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency.  Beyond additive fusion: Learning non-additive multimodal interactions.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4681–4696, 2022.
  * Zadeh et al. [2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency.  Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos, 2016.
  * Zadeh et al. [2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Tensor fusion network for multimodal sentiment analysis.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, 2017.
  * Zhang et al. [2023] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue.  Meta-transformer: A unified framework for multimodal learning.  arXiv preprint arXiv:2307.10802, 2023.

Appendix

##  Appendix A Proof of Theorem 1

We begin with a few simple optimization lemmas.

######  Lemma A.1.

Let $u>0$, we have

$\displaystyle\operatorname*{arg\,min}_{x>0}\frac{u}{x}+\frac{\lambda}{p}x^{p}$
$\displaystyle=(\frac{u}{\lambda})^{\frac{1}{p+1}}$ (12)
$\displaystyle\min_{x>0}\frac{u}{x}+\frac{\lambda}{p}x^{p}$
$\displaystyle=\frac{p+1}{p}u^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}.$ (13)

###### Proof.

Let

$f(u,x)=\frac{u}{x}+\frac{\lambda}{p}x^{p},\text{we have}$ $\frac{\partial
f}{\partial x}=-\frac{u}{x^{2}}+\lambda x^{p-1},$

which cancels at $x^{*}=(\frac{u}{\lambda})^{\frac{1}{p+1}}$. Note also that
$\frac{\partial f}{\partial x}\leq 0$ for $x\leq x^{*}$ and $\frac{\partial
f}{\partial x}\geq 0$ for $x\geq x^{*}$, guaranteeing a minimum. Substituting
back into the definition of $f$, we obtain

$f(x,u^{*})=u^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}+\frac{\lambda}{p}\frac{u^{\frac{p}{p+1}}}{\lambda^{\frac{p}{p+1}}}=\frac{p+1}{p}u^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}$

∎

######  Lemma A.2.

Let $A,u,B>0$ with $u<1$, we have

$\displaystyle\operatorname*{arg\,max}_{x>0}Ax^{u}-Bx$
$\displaystyle=(\frac{Au}{B})^{\frac{1}{1-u}}$ (14)
$\displaystyle\max_{x>0}Ax^{u}-Bx$
$\displaystyle=A(\frac{Au}{B})^{\frac{u}{1-u}}-B(\frac{Au}{B})^{\frac{1}{1-u}}.$
(15)

###### Proof.

Let $f(A,B,u,x)=Ax^{u}-Bx$. We have

$\frac{\partial f}{\partial x}=Aux^{u-1}-B.$

This derivative cancels at $x=x^{*}:=(\frac{Au}{B})^{\frac{1}{1-u}}$ as
expected. Substituting back into the definition of $f$, we obtain

$f(A,B,u,x^{*})=A(\frac{Au}{B})^{\frac{u}{1-u}}-B(\frac{Au}{B})^{\frac{1}{1-u}},$

as expected. ∎

We can now write the main step in the proof of Theorem 1:

######  Proposition A.3.

Let $A_{1},A_{2},\ldots,A_{m}>0$, we have

$\displaystyle\min_{\beta>0,\|\beta\|_{p}^{p}\leq
1}\sum_{m=1}^{M}\frac{A_{m}}{\beta_{m}}=(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{p+1}{p}},$
(16)

and the $\beta_{m}$’s which satisfy the above minimization problem are given
by

$\displaystyle\beta_{m}=\frac{A_{m}^{\frac{1}{p+1}}}{(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{1}{p}}}.$
(17)

###### Proof.

Let us write $f(A,B)=\sum_{m=1}^{M}\frac{A_{m}}{\beta_{m}}$. The Lagrangian is
$\mathcal{L}(\lambda,f,\beta)=\sum_{m=1}^{M}\frac{A_{m}}{\beta_{m}}+\frac{\lambda}{p}(\|\beta\|_{p}^{p}-1)$.
By Lemma A.1, optimizing over each component of $\beta$ independently, we have

$\displaystyle\min_{\beta}\mathcal{L}(\lambda,f,\beta)=\sum_{m=1}^{M}\frac{p+1}{p}A_{m}^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}-\frac{\lambda}{p},$
(18)

realized for $\beta_{m}=(\frac{A_{m}}{\lambda})^{\frac{1}{p+1}}$. Hence, the
dual problem is

$\displaystyle\max_{\lambda}\sum_{m=1}^{M}\frac{p+1}{p}A_{m}^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}-\frac{\lambda}{p}.$
(19)

By Lemma A.2, we obtain

$\displaystyle\max_{\lambda}\sum_{m=1}^{M}\frac{p+1}{p}A_{m}^{\frac{p}{p+1}}\lambda^{\frac{1}{p+1}}-\frac{\lambda}{p}$
(20)
$\displaystyle=\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}}\frac{p+1}{p}(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{1}{p}}-\frac{(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{p+1}{p}}}{p}$
(21) $\displaystyle=(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{p+1}{p}},$
(22)

realized for $\lambda=(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{p+1}{p}}$.

Note that for $m=1,2,\ldots,M$,

$\displaystyle\beta_{m}=(\frac{A_{m}}{\lambda})^{\frac{1}{p+1}}=\frac{A_{m}^{\frac{1}{p+1}}}{(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{1}{p}}}.$
(23)

Finally, note that

$\displaystyle\sum_{m=1}^{M}\beta_{m}^{p}$
$\displaystyle=\frac{\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}}}{(\sum_{m=1}^{M}A_{m}^{\frac{p}{p+1}})^{\frac{p}{p}}}=1.$

Hence, complementary slackness is satisfied, and we have strong duality. ∎

We can now proceed with the

###### Proof of theorem 1.

First of all, similarly to Kloft et al. [2011] and Lemma 26 in Micchelli et
al. [2005] and we apply the following substitution:
$w^{m}_{L}\leftarrow\sqrt{\beta_{m}}w^{m}_{L}$, which turns the problem into
the following:

$\operatorname*{minimize}_{\begin{subarray}{c}w_{L}^{1},\cdots,w_{L}^{M},\beta,\\\
W^{1},W^{2},\cdots,W^{M},\\\ \beta\in{\mathbb{R}}^{M},\beta\geq
0,\|\beta\|_{p}\leq
1\end{subarray}}\Bigg{(}\sum_{i=1}^{n}\ell\Big{(}\sum_{m=1}^{M}\langle
w_{L}^{m},f^{m}(x_{i}^{m})\rangle+b,y_{i}\Big{)}\\\
+\Lambda\sum_{l=1}^{L-1}\sum_{m=1}^{M}\|w^{m}_{l}\|_{2}^{2}+\Lambda\sum_{m=1}^{M}\frac{\|w^{m}_{L}\|_{2}^{2}}{\beta_{m}}\Bigg{)}.$
(24)

The minimization problem from equation 24 can be then solved for $\beta$
independently, removing the dependence on $\beta$: simply apply Proposition
A.3 for $A_{m}=\|\omega_{L}^{m}\|_{2}^{2}\Lambda$ to yield the equivalence to

$\displaystyle\operatorname*{minimize}_{\begin{subarray}{c}w_{L}^{1},w_{L}^{2},\ldots,w_{L}^{M},\\\
W^{1},W^{2},\cdots,W^{M}\end{subarray}}\sum_{i=1}^{n}\ell\left(\sum_{m=1}^{M}\langle
w_{L}^{m},f^{m}(x^{m}_{i})\rangle+b,y_{i}\right)$
$\displaystyle+\Lambda\sum_{l=1}^{L-1}\sum_{m=1}^{M}\|w^{m}_{l}\|_{2}^{2}+\Lambda\left(\sum_{m=1}^{M}\|w^{m}_{L}\|_{2}^{q}\right)^{\frac{2}{q}},$
(25)

where $q=\frac{2p}{p+1}$, as expected.

∎

##  Appendix B Proof of Theorem 2

###### Proof.

The first statement is a straightforward consequence of equation 3.3.1,
collecting all the terms involving the same products of components: indeed, we
can just write

$\displaystyle\mathcal{G}_{J}=\sum_{\ell=0}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},\ldots,S^{\ell}\subset
I\atop\begin{subarray}{c}S^{1},\ldots,S^{\ell}\text{disjoint}\\\
\bigcup_{k\in\\{1,2,\ldots,\ell\\}}S^{k}=I\setminus
J\end{subarray}}\prod_{k\in\\{1,2,\ldots,\ell\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right).$ (26)

The proof of the Equation 11 is by induction over the cardinality of $I$.

##### Initial case.

If $|I|=1$, we have $\Psi(f)=f-\mathbb{E}(f)$. The only admissible $J$ is the
empty set, and we indeed have

$\mathbb{E}\left(\sum_{K:J\subset K\subset I}\mathcal{G}_{K}\prod_{m\in
K\setminus J}f^{m}\right)=\mathbb{E}\left(f-\mathbb{E}(f)\right)=0.$

##### Induction step.

We suppose that the result holds for all cardinalities strictly less than
$|I|$. Let us suppose first that $J\neq\emptyset$. Note that by the
construction of $\Psi$, we have immediately

$\displaystyle\sum_{K:J\subset K\subset I}\mathcal{G}_{K}\prod_{m\in
K\setminus J}f^{m}=\Psi_{I\setminus J}(f).$

Since $J\neq\emptyset$, we have $|I\setminus J|<|I|$, and by the induction
hypothesis, $\mathbb{E}\left(\Psi_{I\setminus J}(f)\right)=0$. Hence

$\displaystyle\sum_{K:J\subset K\subset I}\mathcal{G}_{K}\prod_{m\in
K\setminus J}f^{m}=\Psi_{I\setminus J}(Z)=0,$

which concludes the induction step in the case $J\neq\emptyset$.

Finally, if $J=\emptyset$, we have

$\mathbb{E}\left(\sum_{K\subset I}\mathcal{G}_{K}\prod_{m\in
K}f^{m}\right)=\mathbb{E}\left(\Psi_{I}(f)\right)$
$=\mathbb{E}\Biggl{(}\sum_{\ell=0}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell}\subset I\atop
S^{1},S^{2},\ldots,S^{\ell}\text{disjoint}}\prod_{m\in I\setminus(\cup
S^{k})}f^{m}\prod_{k\in\\{1,2,\ldots,\ell\\}}\\\
\mathbb{E}\biggl{(}\prod_{m\in S^{k}}f^{m}\biggl{)}\Biggl{)}$
$\displaystyle=\sum_{\ell=0}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell}\subset I\atop
S^{1},S^{2},\ldots,S^{\ell}\text{disjoint}}\mathbb{E}\left(\prod_{m\in
I\setminus(\cup S^{k})}f^{m}\right)\prod_{k\in\\{1,2,\ldots,\ell\\}}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle=\sum_{\ell=0}^{|I|-1}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell+1}\text{disjoint}\atop\cup_{k=1}^{\ell+1}S^{k}=I}\prod_{k\in\\{1,2,\ldots,\ell+1\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle\qquad\;+\sum_{\ell=1}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell}\text{disjoint}\atop\cup_{k=1}^{\ell}S^{k}=I}\prod_{k\in\\{1,2,\ldots,\ell\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle=\sum_{\ell=1}^{|I|}(-1)^{\ell-1}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell}\text{disjoint}\atop\cup_{k=1}^{\ell+1}S^{k}=I}\prod_{k\in\\{1,2,\ldots,\ell\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle\qquad+\sum_{\ell=1}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},S^{2},\ldots,S^{\ell}\text{disjoint}\atop\cup_{k=1}^{\ell}S^{k}=I}\prod_{k\in\\{1,2,\ldots,\ell\\}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$ $\displaystyle=0,$

where at the fourth and fifth lines, we have split the sum into two cases
according to whether $\cup_{k=1}^{\ell}S^{k}=I$. ∎

###  B.1 Centering step

To make the exposition clearer, we provide as an example the case
$I=\\{1,2,3\\}$ and the features $f^{m}$ are one dimensional. To do this, let
us first observe that in equation 3.3.1, several terms of the sum are
identical to each other. Indeed, if we rename the sets $S^{1},\ldots,S^{\ell}$
by $S^{\sigma(1)},\ldots,S^{\sigma(\ell)}$ where $\sigma$ is any permutation
of $\\{1,2,\ldots,\ell\\}$, then we still obtain a valid term. In particular,
collecting all of the terms which are equal to each other, we can rewrite the
formula equivalently as follows:

$\displaystyle\Psi_{I}(f)$
$\displaystyle=\sum_{\ell=0}^{|I|}(-1)^{\ell}\sum_{\emptyset\neq
S^{1},\ldots,S^{\ell}\subset I\atop
S^{1},\ldots,S^{\ell}\text{disjoint}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\quad\quad\quad\prod_{m\in
I\setminus(\cup S^{k})}f^{m}$
$\displaystyle=\sum_{\ell=0}^{|I|}(-1)^{\ell}\ell!\sum_{\emptyset\neq
S^{1}<\ldots<S^{\ell}\subset I\atop
S^{1},\ldots,S^{l}\text{disjoint}}\mathbb{E}\left(\prod_{m\in
S^{k}}f^{m}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\quad\quad\quad\prod_{m\in
I\setminus(\cup S^{k})}f^{m},$

where ‘$<$’ denotes any strict order relation between subsets fixed in
advance.

In this case, we have, using the notation
$\overline{f^{1}f^{2}}=\mathbb{E}(f^{1}f^{2})$ etc.:

$\displaystyle\hat{f}^{1,2,3}$ $\displaystyle=f^{1}\times f^{2}\times
f^{3}\;-$ $\displaystyle\quad\overline{f^{1}}\times f^{2}\times
f^{3}-f^{1}\times\overline{f^{2}}\times f^{3}-f^{1}\times
f^{2}\times\overline{f^{3}}\;-$ $\displaystyle\quad\overline{f^{1}\times
f^{2}}\times f^{3}-\overline{f^{1}\times f^{3}}\times
f^{2}-\overline{f^{2}\times f^{3}}\times f^{1}\;-$
$\displaystyle\quad\overline{f^{1}\times f^{2}\times f^{3}}\;+$
$\displaystyle\quad 2\times\overline{f^{1}}\times
f^{3}\times\overline{f^{2}}+2\times\overline{f^{2}}\times
f^{1}\times\overline{f^{3}}\;+$ $\displaystyle\quad
2\times\overline{f^{1}}\times
f^{2}\times\overline{f^{3}}+2\times\overline{f^{1}\times
f^{2}}\times\overline{f^{3}}\;+$ $\displaystyle\quad
2\times\overline{f^{1}\times
f^{3}}\times\overline{f^{2}}+2\times\overline{f^{2}\times
f^{3}}\times\overline{f^{1}}\;-$ $\displaystyle\quad
6\times\overline{f^{1}}\times\overline{f^{2}}\times\overline{f^{3}}.$

##  Appendix C Synthetic-Datasets

We describe in detail the synthetic datasets we used in our main paper. We
created two synthetic datasets with a common multimodal structure where each
modality of a datapoint is a sequence of letters chosen randomly from a
predefined set $\\{A,C,G,T\\}$. For modality $m$, each datapoint is a sequence
of length $100$. We created two specific information carrying sub-sequences of
length 7—ACGTAGC and GATGTAC. For a mulimodal datapoint, an information
subsequence is inserted at a random position with a probability $p_{m}$. We
created $10k$ multimodal samples and divided them into two equal halves. The
first half contains the first sub-sequence (ACGTAGC) inserted in each
datapoint’s modality with probability $p_{m}$, while the second half contains
the second sub-sequence (ACGTAGC) inserted in each datapoint’s modality with
the same probability. Thus, the number of samples where the subsequence is
inserted is approximately equal in the two parts across all the modalities.
Same can be said for the number of datapoints in the two halves where no
information is inserted. A high value of $p_{m}$ indicates that a vast
majority of datapoints in modality $m$ contain information specific
subsequences.

###  C.1 Dataset with Independent Modalities

We created the SynthGene dataset where the datapoints in the first half
mentioned above are labeled as positive samples, and all the datapoints in the
second half are labeled as negative. Thus, the dataset contains equal number
of positive and negative samples. SynthGene contains 10 modalities $M_{1:10}$,
where modalities $M_{2}$ is created with $p_{m}=0.2$, $M_{4}$ with
$p_{m}=0.3$, and $M_{7}$ with $p_{m}=0.5$. The other modalities are created
with $p_{m}=0$ to make them irrelevant to the classification problem. However,
there could be some informative sequences in these modalities due to
randomness. A high value of $p_{m}$ here indicates that more class specific
sequences are present in that modality. Thus, the higher the value of $p_{m}$,
the more relevant modality $m$ becomes.

###  C.2 Dataset with Modality Interactions

We also create the SynthGene-tri dataset, a trimodal version of SynthGene.
However, this time, the informative subsequence is not class-specific.
Instead, the label is defined by an exclusive-or (XOR) relationship between
the first two modalities ($M_{1}$ and $M_{2}$). The label is $0$ if both
modalities contain the subsequence or none of them does, and the label is $1$
otherwise (i.e., when one of the modalities contains the subsequence). Note
that modality $M_{3}$ does not contain any informative subsequence; thus, it
is irrelevant. As before, we generate a balanced dataset with 50% of the
samples being positive and 50% negative.

##  Appendix D Real-world Datasets

##### Motivation.

We have chosen four datasets to study how humans perceive emotions from
different data sources like sound, images, and text. These datasets are
crucial for making computers understand emotions, analyzing human behavior,
and improving AI-driven education. These datasets often provide complementary
information that becomes clear when considering different modalities. The
modalities of the datasets are intuitive for humans because humans well
perceive the modalities in their daily experiences. Thus, these datasets are a
great fit for our approach to studying and interpreting how different
modalities interact to predict human behavior.

##### Datasets Preprocessing.

To ensure reproducibility and allow performance comparison, we follow the data
preprocessing used in the MultiBench Liang et al. [2021] benchmark for all
used datasets.

###  D.1 MOSI

CMU Multimodal Opinion-level Sentiment Intensity (CMU-MOSI) dataset Zadeh et
al. [2016] comprises multimodal observations obtained from rigorously
transcribed opinion video content from the YouTube platform, focusing on video
blogs (vlogs). These videos contain individuals expressing opinions on various
subjects while primarily maintaining direct eye contact with the camera. The
transcriptions provide detailed and highly relevant sentiment-carrying
information, such as pause fillers, stresses, and speech pauses.

##### Preprocessing Details.

We adopt the preprocessing steps detailed by Liang et al. [2018]; Bagher Zadeh
et al. [2018]:

  1. 1.

Text (t): The spoken text is represented by a sequence of 300-dimensional word
embeddings obtained by Global Vectors for Word Representation (GloVe)
Pennington et al. [2014] trained on common crawl dataset. Timing of word
utterances is obtained with the Penn Phonetics Lab Forced Aligner (P2FA).

  2. 2.

Vision (v): We use visual features extracted by the Facet library De la Torre
and Cohn [2011] and OpenFace facial behavior analysis toolkit Baltrušaitis et
al. [2016]. Applying these models results in the following information: facial
action units, facial landmarks, head pose, gaze tracking, HOG features, and
facial expression features.

  3. 3.

Audio (a): As audio input to our model, we use a sequence of acoustic features
extracted by COVAREP Degottex et al. [2014]. The features include 12 Mel-
frequency cepstral coefficients, pitch tracking and voiced/unvoiced segment
features, glottal source parameters, peak slope parameters, and maxima
dispersion quotients.

###  D.2 MOSEI

CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) Bagher
Zadeh et al. [2018] is the largest dataset for sentiment analysis and emotion
recognition. The dataset consists of annotated videos collected from Youtube
of individuals discussing a topic of interest in front of the camera. Each
video is annotated for sentiment and the presence of 9 discrete emotions
(angry, excited, fearful, sad, surprised, frustrated, happy, disappointed, and
neutral) and continuous emotions (valence, arousal, and dominance).

##### Preprocessing Details.

We use a preprocessing approach similar to that employed for CMU-MOSI and UR-
FUNNY datasets.

###  D.3 MUStARD

MUStARD is a multimodal dataset that has been developed for the purpose of
advancing automated sarcasm detection research Castro et al. [2019]. This
dataset is curated from renowned television shows such as Friends, The Golden
Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. MUStARD contains
audiovisual statements that have been labeled to denote instances of sarcasm.
Each statement is accompanied by its contextual background, contributing
supplementary insights into the circumstances surrounding the statement’s
occurrence. This, in turn, poses an additional challenge in effectively
modeling multimodal information over extended periods. The deliberate
selection of sarcasm as the annotation task is motivated by the need for
intricate modeling of complementary details, especially in cases where
semantic cues from different modalities may not align.

##### Dataset Preprocessing.

We use the preprocessing steps suggested in the original paper Castro et al.
[2019], which goes as follows for each modality:

  1. 1.

Text (t): The embeddings for the text modality are obtained from pretrained
BERT Devlin et al. [2018] representations and from Common Crawl pretrained
300-dimensional GloVe word vectors for each token.

  2. 2.

Vision (v): Visual features are extracted for each frame using the pool 5
layers of ResNet-152 that is pre-trained on the Imagenet dataset. Every frame
is preprocessed by resizing, center cropping, and normalizing. The OpenFace
analysis tool has also been used to extract facial expression features.

  3. 3.

Audio (a): Low-level features from the audio data stream are extracted using
the speech processing library Librosa. Additionally, COVAREP is used to
extract acoustic features, including 12 Melfrequency cepstral coefficients,
pitch tracking and voiced/unvoiced segment features, glottal source
parameters, peak slope parameters, and maxima dispersion quotients.

###  D.4 UR-FUNNY

UR-FUNNY Hasan et al. [2019] is the first large-scale multimodal dataset of
annotated video excerpts extracted from TED talks for humor detection in human
speech. The essence of humor, as an intricate communicative tool, is
inherently multimodal, interweaving articulate language (text), complementary
gestures (visual), and expressive prosodic cues (acoustic).

##### Preprocessing Details.

We use a preprocessing approach similar to that employed for CMU-MOSI and UR-
FUNNY datasets.

##  Appendix E Experimental Setup

In our experimental Setup, we compare the baselines methods against our two
methods: 1) Multiple Neural Learning (MNL)that captures the linear
combinations of the modality representations in an interpretable manner, 2)
InTense that can capture modality combinations and their interactions in an
interpretable manner. We provide an implementation of these methods and our
appendix, which we plan to release to the public community.

###  E.1 Baselines

| Non-Interpretable | Interpretable
---|---|---
| LF-Concat | TF Network | MulT | MultiRoute | MRO | MNL | InTense
MUStARD | $65.4\pm 1.0$ | $62.1\pm 0.8$ | $71.2\pm 0.4$ | $65.9\pm 0.3$ | $66.5\pm 0.6$ | $67.4\pm 0.5$ | $\mathbf{69.6\pm 0.4}$
CMU-MOSI | $75.2\pm 0.8$ | $74.4\pm 0.2$ | $82.7\pm 0.3$ | $76.8\pm 0.8$ | $75.8\pm 0.6$ | $\mathbf{80.8\pm 0.8}$ | $79.7\pm 0.6$
UR-FUNNY | $62.2\pm 0.6$ | $61.2\pm 0.4$ | $66.2\pm 0.1$ | $63.6\pm 0.9$ | $63.4\pm 0.4$ | $63.4\pm 0.3$ | $\mathbf{65.1\pm 0.6}$
CMU-MOSEI | $79.2\pm 0.4$ | $79.4\pm 0.5$ | $82.1\pm 0.7$ | $80.2\pm 0.8$ | $79.7\pm 0.6$ | $80.5\pm 0.4$ | $\mathbf{81.5\pm 0.3}$
Table 2: Accuracies for different baselines on the test fold. Each experiment
is carried out ten times to compute the statistics.

We compare our method against three non-interpretable baselines and two
interpretable baselines, which we describe below with key points over their
implementation.

##### Non-interpretable Baselines.

We consider the following three baselines:

LF-Concat is a non-interpretable baseline, where the output of the encoder
modules is concatenated, followed by a fully-connected layer for prediction.
We chose this baseline as it is very near to our own implementations. We
provide our own implementation of this method.

TF-Network. from Zadeh et al. [2017] is non-interpretable method used as a
baseline. We chose this baseline as InTense uses a tensor fusion module to
capture modality interactions. We provide an implementation of the TF Network
in our code adopted from the original implementation of the paper.

MulT. Multimodal transformers from Tsai et al. [2019] is a non-interpretable
method used as the third baseline. We chose this as a baseline as it is the
best-performing model on the chosen four datasets according to Liang et al.
[2021]. We provide an implementation of this method in our code adopted from
the original implementation of the paper.

##### Interpretable Baselines.

We consider the following two baselines for a comparison:

MRO. Multimodal Residual Optimization (MRO) from Wörtwein et al. [2022] is
considered as a baseline as it aims at disentangling the uni-, bi-, and tri-
modal interactions. We provide an implementation of this method in our code
repository.

Multiroute. Multimodal Routing from Sabour et al. [2017] is considered a
baseline as it aims to interprect the modalities and their interactions. We
use the implementation provided from the authors to evaluate the results on
the four datasets.

###  E.2 Encoder Modules

We first apply a standard batch normalization over the input features of each
modality, followed by a GRU (Gated Recurrent Unit) module for each modality.
We use the same encoder for all experiments to test our baselines except MulT
(Multimodal Transformers), as it relies on the multimodal transformer
architecture. We report MultiRoute as well using its original implementation.

###  E.3 Network Architecture

Figure 6 summarizes the network architecture for InTense, with the tensor
fusion modules to capture modality interactions and the Iterative Batch
Normalization module to untangle modality interactions from individual
modality effects.

Figure 6: The figure shows the framework of InTense for two modalities. The
modalities and their fusion are jointly trained to obtain the representations
($f^{1}$, $f^{2}$), which then undergo through our TensorFusion module to
capture the multiplicative interactions. We apply IterBN to disentangle the
multiplicative interactions from individual modality effects. All the modality
representations and their multiplicative interactions are fused together in
the end to obtain the final prediction.

###  E.4 Performance Evaluation

We use four multimodal datasets that are evaluated in two different ways.
These are explained below.

##### 1\. MOSI, 2. MOSEI.

These two datasets are trained using the MAE (Mean Absolute Error) objective
on sentiment analysis, where the labels are integers between $-3$ and $3$. The
model with the least MAE on the validation set is chosen during training.
During testing, the accuracy is calculated with the positive/negative class
setting, where a prediction less than $0$ is considered a negative class, and
a prediction greater than $0$ is considered a positive class. The data points
with label $0$ are considered as neutral are not considered while evaluating
accuracy on the test set.

##### 3\. SARCASM, 4. UR-FUNNY.

These are the binary-classification datasets with labels $0$ and $1$. The
training objective is Cross-Entropy Loss. The model with the least loss on the
validation set is chosen during training. The accuracy is computed on the test
fold using the best model obtained on the validation set.

##### 5\. AV-MNIST, 6. ENRICO.

These are the multi-class classification datasets containing $10$ classes for
AV-MNIST and $23$ classes for ENRICO daaset The training objective is Cross-
Entropy Loss. The model with the least loss on the validation set is chosen
during training. The accuracy is computed on the test fold using the best
model obtained on the validation set.

For all the experiments, we report the mean accuracy and its standard
deviation on ten random seeds.

|  | lr | batch_size | hidden_dims | tf_indices | tf_latent_dim
---|---|---|---|---|---|---
MOSEI | MNL | $1e-3$ | $32$ | $[64,64,768]$ | None | None
InTense | $1e-3$ | $32$ | $[64,64,1024]$ | [’13’] | 8
MOSI | MNL | $5e-3$ | $32$ | $[64,64,768]$ | None | None
InTense | $1e-3$ | $32$ | $[64,64,768]$ | [’13’] | 8
UR-FUNNY | MNL | $5e-3$ | $32$ | $[64,64,768]$ | None | None
InTense | $1e-3$ | $32$ | $[64,64,768]$ | [’13’] | 8
MUStARD | MNL | $5e-3$ | $32$ | $[64,64,768]$ | None | None
InTense | $1e-3$ | $32$ | $[64,64,768]$ | [’13’] | 8
Table 3: The table shows the best performing hyperparamaters for MNL and
InTense across all four real-world dataset

###  E.5 Hyperparameters

Below, we will delve into the key hyperparameters used to obtain the reported
performance in InTense. In our implementation, we have used the same
nomenclature for these hyperparameters to make our code easier to follow.
Table 3 shows a list of the hyperparameter settings we used to achieve the
best results across all baseline models.

  1. 1.

learninng rate (lr): For all the experiments, we chose the initial learning
rate between $0.0005$ to $0.005$. We used an exponential learning rate decay
that decays the learning rate of each parameter group by gamma every epoch.
The value of gamma is set between $0.9$ and $1$.

  2. 2.

epochs: We ran all our experiments for $100$ epochs and chose the model with
the best validation metrics. The validation metrics for the datasets are shown
in Table 3

  3. 3.

batch_size: Batch size denotes the number of datapoints in a mini-batch. We
chose a batch size of either $16$ or $32$ for all the experiments.

  4. 4.

weight_decay: As per our optimization framework defined in Equation A of the
main paper, a weight decay is applied to every modality encoder. The value of
weight decay was set to $0.01$ for all the experiments.

  5. 5.

reg_param: It is the hyperparameter $\Lambda$ in Equation A of our framework
that tunes the block-norm regularization for the fusion layer. We set its
value to $0.05$ for all the experiments.

  6. 6.

p: It defines the value of the $q-$norm over the weights of the fusion layer
(described in the Equation A of the main paper). The value can be used to tune
the sparsity of relevance scores with $1\leq q\leq 2$. We set the value of
this hyperparameter to be $1$ for all our experiments.

  7. 7.

hidden_dims: This hyperparameter is an array with dimensions matching the
count of distinct modalities in the dataset. Each individual element within
the array determines the resultant dimensionality of a corresponding modality
encoder module. Table 3 shows a list of the hyperparameter settings.

  8. 8.

tf_indices: This hyperparameter is an array of modality indices that defines
the interactions captured by InTense. E.g. a value
$[^{\prime}12^{\prime},^{\prime}23^{\prime}]$ would indicate an interaction
between modalities at index $1$,$2$ and an interaction between modalities at
index $2$,$3$ is learned.

  9. 9.

tf_latent_dim: This hyperparameter is an integer value that defines the
dimension of the modality representations undergoing the tensor product. To
alleviate computation overhead, the dimension of the output from the modality
encoders is mapped to a size of tf_latent_dim.

  10. 10.

scheduler_gamma: Decays the learning rate of each parameter group by gamma
every epoch. The default value for all the experiments is set to $0.9$.

  11. 11.

optimizer: Set the optimizer for learning. By default, we used PyTorch
implementation of the AdamW algorithm.

  12. 12.

task: Defines the task for which the model is trained. The task is set
regression for the dataset “MOSI” and “MOSEI” where the labels are regression
labels. For other datasets, the default value of task is set to
“classification.”

  13. 13.

act_before_vbn: In our VBN normalization scheme, the output from the modality
encoders are passed through ReLU activation when this parameter is set to
True. Otherwise, the VBN normalization is directly applied to the outputs of
the modality encoders. We found through experiments that setting this
parameter doesn’t improve the performace, so the value is set to False for all
the experiments.

##  Appendix F Ablation Studies

| GRUwithBN | Transformers | MLP
---|---|---|---
| MNL | InTense | MNL | InTense | MNL | InTense
MOSI | $80.8\pm 0.8$ | $79.7\pm 0.6$ | $78.2\pm 0.2$ | $79.5\pm 0.4$ | $77.4\pm 0.6$ | $78.2\pm 0.4$
MOSEI | $80.5\pm 0.4$ | $81.5\pm 0.3$ | $80.2\pm 0.5$ | $81.2\pm 0.2$ | $78.8\pm 0.7$ | $79.7\pm 0.2$
UR-FUNNY | $64.4\pm 0.3$ | $65.1\pm 0.6$ | $65.6\pm 0.3$ | $65.8\pm 0.3$ | $62.6\pm 0.3$ | $63.9\pm 0.5$
MUStARD | $67.4\pm 0.5$ | $69.6\pm 0.4$ | $67.8\pm 0.5$ | $69.8\pm 0.2$ | $65.8\pm 0.4$ | $66.4\pm 0.3$
Table 4: The table shows the Accuracy of our methods MNL and InTense on the test set. Here we use three different encoders to perform an evaluation of our method. We report the mean accuracy and its standard deviation over $10$ experiments performed on random seeds. | MOSI | MOSEI | MUStARD | UR-FUNNY
---|---|---|---|---
| Acc. | Rel. | Acc. | Rel. | Acc. | Rel. | Acc. | Rel.
text(t) | 74.2 | 0.75 | 78.8 | 0.85 | 68.6 | 0.78 | 58.3 | 0.77
audio(a) | 65.5 | 0.11 | 66.4 | 0.05 | 64.9 | 0.07 | 57.2 | 0.15
video(v) | 66.3 | 0.14 | 67.2 | 0.10 | 65.7 | 0.16 | 57.3 | 0.07
Table 5: The table shows that the modality with highest accuracy individually
gets a high relevance score when trained through InTense. Acc. stands for the
Accuracy of the model on the test set, and Rel. stands for the Relevance
scores obtained through InTense

###  F.1 What can go wrong

We conducted experiments where we found that without proper normalization, the
interpretability scores are biased toward higher-order interactions, even when
those interactions do not add any useful information as compared to the lower-
order modality interactions among fewer modalities.

To conduct our experiments, we considered the SynthGene-tri dataset. Within
this dataset, we have established that while labels are influenced solely
through modality interaction between modalities $1$ and $2$, no valuable
information for label determination is present in modality $3$. Therefore, an
interaction involving fewer modalities ($1\;\text{and}\;2$) should receive the
highest relevance score.

The figure 7 illustrates that in the absence of iterative batch normalization
(IterBN), our methods exhibit biased scores towards the modality interaction
among modalities $1,2,\text{and};3$ (depicted by the orange bars). Notably,
the figure demonstrates that IterBN successfully mitigates this bias,
correctly assigning the highest relevance score to the modality interaction
between modalities $1\;\text{and}\;2$.

Figure 7: An illustration of our method when higher-order modality-
interactions are involved. Naive normalization with VBN leads to biased
results (orange bars) where the relevance scores are concentrated toward
higher-order interactions $M_{1\otimes 2\otimes 3}$. In contrast, InTense
(blue bars) correctly assigns a high relevance score only to the interaction
$M_{1\otimes 2}$, which contains all class-specific signals.

###  F.2 Modality Encoder Modules

We tried different encoder modules on the top of the preprocessed features for
the approaches MNL, and InTense. The table 4 shows the accuracy obtained on
the test set.

##### GRUwithBN.

These encoder modules are used to compare and report the results in the main
paper. The archtecture of the encoder modules is explained in Appendix E.2

##### Tranformer.

We also comapare our results when a Tranformer encoder is used for each
modality. We trained a Transformer from scratch using its PyTorch
implementation with $4$ layers and $5$ heads. Table 4 shows that these encoder
performs slightly better on an average than GRUwithBN on two out of four
datasets.

##### MLP.

We used a Multi Layer Perceptron as an encoder for all the three modalities.

###  F.3 MultiRoute global interpretabilty scores

We compare the interpretability scores of our method against the global
interpretability scores obtained from MultiRoute Tsai et al. [2020]. Figure 4
in the main paper depicts the biased interpretability scores from MultiRoute.

In this section, we explain the process of obtaining the interpretability
scores from MultiRoute. At first, we used the same process as described in the
paper to obtain the relevance scores. At first, we trained a multimodal
routing based capsule model on our synthetic dataset SynthGene-tri. Next, as
desribed in the paper, we computed the average routing coefficient
$\overline{r_{ij}}$ for all three modalities and their interactions. We
normalized the routing coefficients so that they sum to $1$. This brings them
in the same scale as our interpretability scores.

###  F.4 Correlation between Relevance scores and Unimodal Accuracy

In the main paper, we show through our synthetic dataset that the relevance
scores from InTense correlate with the accuracies when trained on individual
modalities. Table 5 shows that the same correlation can be seen on the real-
world multimodal datasets when each modality is trained independently.

##  Appendix G Source Code for Reproducibility

We provide an implementation of our methods in PyTorch as a zip file. In order
to run the code, unzip the file’s content and navigate into the root
directory. The code has been tested on Python 3.10+, and PyTorch 1.12+ on
Ubuntu 22.10 and Red Hat Enterprise Linux Server. We recommend using and
Linux-based operating system and a virtual environment management tool of your
choice to install the packages. We provide step-by-step instructions to set up
our code through Conda.

###  G.1 Code Setup

##### Step 1:

Create a conda environment for InTense.

⬇

1conda create -n intense python=3.10

##### Step 2:

Install a compatible PyTorch version inside the environment. Check your
compatible version at the official webpage: https://pytorch.org/

⬇

1conda activate intense

2conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia

##### Step 3:

Set up the code to install all the required packages by running the following
commands. We use pip for dependency management:

⬇

1pip install -U -r requirements.txt

2pip install -e .

We are ready to run the experiments now!

###  G.2 Running Experiments

We describe the scripts here to run the experiments for synthetic and real-
world datasets.

##### Synthetic Datasets.

In order to run different experiments, use the following scripts:

  1. 1.

SythGene To Run the experiment for the SythGene dataset with independent
modalities, run the following command:

⬇

1python intense/training_modules/synthetic/syn_mnl.py \--scheduler-gamma 0.9
\--epochs 20 \--experiment-name "synthgene" \--batch-size 32 \--latent-dim 8
\--lr 1e-3 \--reg-param 0.05

  2. 2.

SythGene-tri Run experiment for modality interactions, using:

⬇

1python intense/training_modules/synthetic/syn_xor.py \--scheduler-gamma 0.9
\--epochs 20 \--experiment-name "synthgene" \--batch-size 32 \--latent-dim 8
\--lr 1e-3 \--reg-param 0.05

##### MOSI

In order to run different experiments, use the follwing scripts:

  1. 1.

MNL (Multiple Neural Learning)

⬇

1python intense/training_modules/regression/train_mnl.py \--scheduler-gamma 1
\--epochs 100 \--is-packed \--experiment-name "mnl_mosi" \--hidden-dims 64 64
768 \--lr 1e-3 \--dataset mosi

  2. 2.

InTense

⬇

1python intense/training_modules/regression/train_intense.py \--scheduler-
gamma 1 \--epochs 5 \--is-packed \--experiment-name
"mkl_intensfuse_mosi_eval_2" \--hidden-dims 64 64 768 \--dataset mosi \--lr
1e-3 \--tf-latent-dim 8 \--tf-indices ’13’

  3. 3.

MRO

⬇

1python intense/training_modules/regression/train_mro.py \--scheduler-gamma
0.9 \--epochs 10 \--is-packed \--experiment-name "mro_mosi_eval" \--affine
\--hidden-dims 64 64 768 \--dataset mosi \--lr 5e-3 \--tf-latent-dim 8

  4. 4.

MulT

⬇

1python intense/training_modules/regression/train_mult.py \--scheduler-gamma
0.9 \--epochs 100 \--experiment-name "mosi_mult" \--lr 1e-3 \--dataset mosi
\--num-workers 2

##### MOSEI

In order to run different experiments, use the following scripts:

  1. 1.

MNL (Multiple Neural Learning)

⬇

1python intense/training_modules/regression/train_mnl.py \--scheduler-gamma
0.9 \--epochs 100 \--is-packed \--experiment-name "mnl_mosei" \--hidden-dims
64 64 1024 \--lr 1e-3 \--dataset mosei_senti

  2. 2.

InTense

⬇

1python intense/training_modules/regression/train_intense.py \--scheduler-
gamma 0.9 \--epochs 200 \--is-packed \--experiment-name "intense_mosei"
\--hidden-dims 64 64 1024 \--dataset mosei_senti \--lr 1e-3 \--tf-latent-dim 8
\--tf-indices ’13’

  3. 3.

MRO

⬇

1python intense/training_modules/regression/train_mro.py \--scheduler-gamma
0.9 \--epochs 100 \--is-packed \--experiment-name "mro_mosei" \--affine
\--hidden-dims 32 32 128 \--dataset mosei_senti \--lr 1e-3 \--tf-latent-dim 8
\--num-workers 4

  4. 4.

MulT

⬇

1 python intense/training_modules/regression/train_mult.py \--scheduler-gamma
0.9 \--epochs 5 \--experiment-name "mult_mosei" \--lr 1e-3 \--dataset
mosei_senti \--num-workers 2

##### UR-FUNNY

In order to run different experiments, use the following scripts:

  1. 1.

MNL (Multiple Neural Learning)

⬇

1python intense/training_modules/classification/train_mnl.py \--scheduler-
gamma 1 \--epochs 100 \--is-packed \--experiment-name "humor_mkl" \--hidden-
dims 32 32 128 \--lr 1e-3 \--reg-param 0.01 \--batch-size 16 \--num-workers 4
\--dataset humor

  2. 2.

InTense

⬇

1python intense/training_modules/classification/train_intense.py \--scheduler-
gamma 0.9 \--epochs 20 \--is-packed \--experiment-name "humor_intense"
\--hidden-dims 32 32 64 \--tf-latent-dim 4 \--lr 1e-3 \--scheduler-gamma 1
\--reg-param 0.01 \--batch-size 32 \--num-workers 2 \--dataset humor \--tf-
indices ’23’

  3. 3.

MRO

⬇

1python intense/training_modules/classification/train_mro.py \--scheduler-
gamma 0.9 \--epochs 200 \--is-packed \--experiment-name "humor_mro_eval_2"
\--hidden-dims 32 32 256 \--lr 1e-3 \--scheduler-gamma 0.9 \--reg-param 0.01
\--batch-size 16 \--num-workers 2 \--tf-latent-dim 16 \--dataset humor

  4. 4.

MulT

⬇

1python intense/training_modules/classification/train_mnl.py \--scheduler-
gamma 1 \--epochs 100 \--is-packed \--experiment-name "humor_mkl" \--hidden-
dims 32 32 128 \--lr 1e-3 \--reg-param 0.01 \--batch-size 16 \--num-workers 4
\--dataset humor

##### MUStARD

In order to run different experiments, use the following scripts:

  1. 1.

MNL (Multiple Neural Learning)

⬇

1python intense/training_modules/classification/train_mnl.py \--scheduler-
gamma 0.9 \--epochs 100 \--is-packed \--experiment-name "sarcasm_mnl"
\--hidden-dims 32 32 128 \--lr 5e-3 \--scheduler-gamma 0.9 \--reg-param 0.01
\--batch-size 16 \--num-workers 4 \--dataset sarcasm

  2. 2.

InTense

⬇

1python intense/training_modules/classification/train_intense.py \--scheduler-
gamma 0.9 \--epochs 200 \--is-packed \--experiment-name
"sarcasm_mkl_intensfuse_eval" \--hidden-dims 32 32 128 \--tf-latent-dim 8
\--lr 1e-3 \--scheduler-gamma 0.9 \--reg-param 0.01 \--batch-size 16 \--num-
workers 2 \--dataset sarcasm \--tf-indices ’13’

  3. 3.

MRO

⬇

1intense/training_modules/classification/train_mro.py \--scheduler-gamma 0.9
\--epochs 200 \--is-packed \--experiment-name "sarcasm_mro_eval_3" \--hidden-
dims 32 32 256 \--lr 1e-3 \--scheduler-gamma 0.9 \--reg-param 0.01 \--batch-
size 16 \--num-workers 2 \--tf-latent-dim 8 \--dataset sarcasm

  4. 4.

MulT

⬇

1python intense/training_modules/classification/train_mnl.py \--scheduler-
gamma 1 \--epochs 100 \--is-packed \--experiment-name "humor_mkl" \--hidden-
dims 32 32 128 \--lr 1e-3 \--reg-param 0.01 \--batch-size 16 \--num-workers 4
\--dataset humor

#####

All information about the training arguments is provided in Appendix E.5. For
further information, use the ”- -help” argument while running the command.

##### Note.

We use the MLFlow library for experiment tracking; you can run the following
command to start the mlflow server from the shell. The relevance scores are
automatically computed and tracked during the training.

⬇

1mlflow server \--backend-store-uri file:experiments \--no-serve-artifacts