# Exchanging-based Multimodal Fusion with Transformer

Renyu Zhu1 Chengcheng Han211footnotemark: 1 Yong Qian3 Qiushi Sun4
Xiang Li2 Ming Gao2 Xuezhi Cao3 Yunsen Xian3
1NetEase Fuxi AI Lab, Hangzhou, Zhejiang, China
2School of Data Science and Engineering, East China Normal University,
Shanghai, China
3Meituan Inc., Beijing, China 4National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>{caoxuezhi<EMAIL_ADDRESS>
Equal contribution Corresponding author

###### Abstract

We study the problem of multimodal fusion in this paper. Recent exchanging-
based methods have been proposed for vision-vision fusion, which aim to
exchange embeddings learned from one modality to the other. However, most of
them project inputs of multimodalities into different low-dimensional spaces
and cannot be applied to the sequential input data. To solve these issues, in
this paper, we propose a novel exchanging-based multimodal fusion model MuSE
for text-vision fusion based on Transformer. We first use two encoders to
separately map multimodal inputs into different low-dimensional spaces. Then
we employ two decoders to regularize the embeddings and pull them into the
same space. The two decoders capture the correlations between texts and images
with the image captioning task and the text-to-image generation task,
respectively. Further, based on the regularized embeddings, we present
CrossTransformer, which uses two Transformer encoders with shared parameters
as the backbone model to exchange knowledge between multimodalities.
Specifically, CrossTransformer first learns the global contextual information
of the inputs in the shallow layers. After that, it performs inter-modal
exchange by selecting a proportion of tokens in one modality and replacing
their embeddings with the average of embeddings in the other modality. We
conduct extensive experiments to evaluate the performance of MuSE on the
Multimodal Named Entity Recognition task and the Multimodal Sentiment Analysis
task. Our results show the superiority of MuSE against other competitors. Our
code and data are provided at https://github.com/RecklessRonan/MuSE.

##  1 Introduction

Humans perceive the world by simultaneously processing various modalities,
such as visions, sounds and feelings, and further fusing them. Therefore,
multimodal fusion has become a key research problem Baltrušaitis et al. (2018)
in the area of artificial intelligence. With the recent success of deep
learning, a series of deep multimodal fusion methods Bayoudh et al. (2021);
Guo et al. (2019) have been proposed and applied on various multimodal tasks,
such as Multimodal Named Entity Recognition (MNER) Sun et al. (2021); Wang et
al. (2020) and Multimodal Sentiment Analysis (MSA) Colombo et al. (2021); Wu
et al. (2021). Early methods mainly can be divided into two categories:
aggregation-based and alignment-based. Aggregation-based methods first
represent each modality by a sub-network and then aggregate different
representations with various operators, such as concatenation Zeng et al.
(2019), averaging Hazirbas et al. (2016), and self-attention Valada et al.
(2020). Further, alignment-based methods Colombo et al. (2021); Song et al.
(2020) adopt a regularization loss to align the embeddings of different sub-
networks without explicit combination.

Figure 1: A sketched comparison between existing fusion methods and ours.

Recently, a novel exchanging-based method CEN Wang et al. (2020) was proposed
to handle the trade-off between intra-modal processing and inter-modal fusion.
By utilizing the scaling factor of Batch Normalization (BN) Ioffe and Szegedy
(2015) as the importance measurement of each channel, CEN replaces those
channels with close-to-zero factor values in one modality with the average of
channels in the other modality. However, the method is specially designed for
vision-vision fusion and the channel exchange cannot be directly applied in
other multimodal scenarios, such as text-vision fusion. There are two main
challenges. On the one hand, CEN implicitly assumes that the two modalities
are represented in the same low-dimensional embedding space, while the
modalities of texts and images are distant from each other and generally
correspond to different spaces. On the other hand, the exchanging mode based
on CNN channels of images is inapplicable for texts, since texts are sequences
of words. Therefore, a research question arises: Can we develop an effective
exchanging-based neural network model that fuses modalities of text and
vision?

In this paper, we present a novel Multimodal fuSion method based on
Exchanging, namely, MuSE, which bridges the gap of exchanging-based methods in
the field of text-vision multimodal fusion. We summarize the comparison
between existing multimodal fusion methods and ours in Figure 1. To handle the
problem of different low-dimensional spaces corresponding to various data
modalities, we first perform separate low-dimensional projections for texts
and images, and then propose two tasks to pull their embeddings into the same
space. Specifically, inspired by the fact that texts can be captions of images
and images can also be generated from texts, we specially design an _image
captioning_ task and a _text-to-image generation_ task to capture the
correlations between texts and images. We implement the process with an
architecture of two encoder-decoders, where encoders are used for text (image)
projection and decoders are for image (text) generation. The two decoders
jointly regularize the embeddings generated by the encoders and pull them into
the same space. Further, we investigate information exchange for text-vision
modalities and propose _CrossTransformer_ , which uses two Transformer Vaswani
et al. (2017) encoders with shared parameters as the backbone model.
Specifically, we insert cls111The special token in Transformer that can be
used to derive the sentence-level embedding. into the beginning of the
generated embedding vectors of texts and images by the encoder-decoder module,
which are then fed into the two Transformer encoders for both modalities.
Intuitively, we first need to learn the global contextual information of the
input vectors in the shallow layers of the Transformer encoders and then
exchange the well-learned knowledge between modalities. After the multimodal
fusion completes, the exchange can stop. Therefore, in CrossTransformer, we
introduce two hyper-parameters $\mu$ and $\eta$, which denote the start layer
and the end layer for information exchanging, respectively. In each exchanging
layer, we select the tokens in one modality with the smallest attention scores
to cls and replace their embeddings with the average of all token embeddings
in the other modality. In this way, text-vision fusion can be automatically
performed. Finally, we summarize our contributions as follows.

  * •

We generalize the exchanging-based methods from vision-vision fusion to text-
vision fusion and propose a novel exchanging-based model MuSE.

  * •

We employ the image captioning task and the text-to-image generation task to
capture the correlations between texts and images, which jointly regularize
multimodal embeddings and pull them into the same space. We further design
CrossTransformer, which exchanges knowledge between texts and images.

  * •

We conduct extensive experiments to compare MuSE with other state-of-the-arts
on the MNER and MSA tasks. Experimental results show the effectivenss of our
method.

##  2 Related Works

###  2.1 Deep Multimodal Fusion

Deep multimodal fusion has recently received significant attention in deep
multimodal learning Bayoudh et al. (2021); Guo et al. (2019). Early methods
can be mainly divided into two categories Baltrušaitis et al. (2018), namely
aggregation-based methods Hazirbas et al. (2016); Zeng et al. (2019); Valada
et al. (2020) and alignment-based methods Colombo et al. (2021); Song et al.
(2020). For the former, they learn separate representations in each modality
and directly aggregate the learned representations from different modalities,
which lacks the inter-modal communication. In contrast, alignment-based
methods use regularization to align embedding distributions in different
modalities, such as mutual information maximization Colombo et al. (2021) and
maximum mean discrepancy Gretton et al. (2012); Shankar (2021). However, these
methods omit the intra-modal characteristics with simple distribution aligning
Song et al. (2020). While some works Han et al. (2021); Ju et al. (2021)
combine aggregation and alignment, they need fine-grained hierarchical design
that could introduce the additional cost of computation and engineering.
Another taxonomy of multimodal fusion De Vries et al. (2017) concentrates on
the fusion stage and a detailed analysis is given in Nagrani et al. (2021).
Different from all these methods, an exchanging-based method CEN Wang et al.
(2020) is proposed for multi-vision modalities, which can effectively handle
the trade-off between inter-modal processing and intra-modal fusion. While
there exist some extensions Wang et al. (2021b); Jiang et al. (2022) to CEN,
all these methods are limited in channel-exchanging and vision-vision fusion.

Due to the space limitation, we move the related works of MNER and MSA to the
Appendix A.

Figure 2: The overall architecture of MuSE.

##  3 Methodology

In this section, we propose a deep multimodal fusion model MuSE. As shown in
Figure 2, MuSE is mainly composed of four functional components and we tag
them by ①-④. The component ① is used to project the input texts and images
into low-dimensional spaces, which includes a text encoder and an image
encoder. Considering that multimodal data might be mapped into different
spaces, we further present two embedding regularizers (see components ② and ③)
to pull embeddings of multimodal inputs into the same space. These two
regularizers are taken as decoders, which implement the text-to-image
generation task and the image captioning task, respectively. After the
embeddings of multimodal inputs are generated, we feed them into component ④,
which is a Transformer-encoder-based module called CrossTransformer.
CrossTransformer performs multimodal information exchanging and finally
generates fused embeddings from multimodality. Details of each component will
be introduced in the following.

###  3.1  Low-dimensional Projection and Embedding Regularization

We use an architecture of two encoder-decoders to project the inputs of
textual ($T$) and visual ($I$) modalities into the same embedding space before
exchanging-based fusion.

#### Low-dimensional Projection.

We first employ two separate encoders to encode the input texts and images
into low-dimensional embeddings, respectively:

$\displaystyle\mathbf{T_{e}}$ $\displaystyle=\text{TextEncoder}(T),$ (1)
$\displaystyle\mathbf{I_{e}}$ $\displaystyle=\text{ImageEncoder}(I),$ (2)

where TextEncoder can be a typical text representation model (e.g., BERT
Kenton and Toutanova (2019)) and ImageEncoder can be a typical CNN model
(e.g., ResNet He et al. (2016)). Specifically, we set
$\mathbf{T_{e}},\mathbf{I_{e}}\in\mathbb{R}^{n\times d}$, where $n$ is the
text length and $d$ is the embedding dimensionality.

#### Embedding Regularization.

Since the inputs are in two modalities, they are generally projected into
different spaces by Equation 2. Therefore, for information exchange from
multimodalities, we first need to pull these embeddings into the same space.
Empirically, we observe that texts can be captions of images while images can
be generated from texts, as shown in Figure 3. To capture the correlations
between texts and images, we specially design an image captioning task and a
text-to-image generation task. These two tasks are implemented by two
decoders, which jointly regularize the embeddings from the encoders. For the
image captioning task, the encoder takes the image embedding as input and
generates the caption text, while we do the opposite for the text-to-image
generation task. The overall procedure is summarized as follows. First,
similar as in Zhang et al. (2021b), to enhance the generalization ability of
the model, we add random noise to the embeddings of texts and images:

$\displaystyle\mathbf{T_{n}}$
$\displaystyle=\text{MLP}(\mathbf{T_{e}}+\mathcal{N}_{t}(0,1)),$ (3)
$\displaystyle\mathbf{I_{n}}$
$\displaystyle=\text{MLP}(\mathbf{I_{e}}+\mathcal{N}_{i}(0,1)),$ (4)

where $\mathcal{N}_{t}(0,1)$ and $\mathcal{N}_{i}(0,1)$ are Gaussian random
noise for textual and visual modalities, respectively. Then we use two
decoders to generate images $\hat{I}$ and texts $\hat{T}$, respectively:

$\displaystyle\hat{I}$ $\displaystyle=\text{ImageDecoder}(\mathbf{T_{n}}),$
(5) $\displaystyle\hat{T}$ $\displaystyle=\text{TextDecoder}(\mathbf{I_{n}}),$
(6)

where ImageDecoder can be a typical text-to-image generation model (e.g.,
PixelCNN Van den Oord et al. (2016)), and TextDecoder can be a typical image
captioning model (e.g., NIC Vinyals et al. (2015)). Based on the generated
images and texts, we compare them with the input ones and construct the text-
to-image generation loss $\mathcal{L}_{ti}$ and image captioning loss
$\mathcal{L}_{it}$, respectively. These two losses regularize the embeddings
of texts and images generated by the encoders and can be considered as
auxiliary tasks to the main prediction task that will be introduced later.

Figure 3: Toy examples on the correlations between texts and images, which are
extracted from the Twitter15 (for the MNER task) and MVSA-Single (for the MSA
task) datasets, respectively.

###  3.2 Multimodal Exchanging

After we derive embeddings $\mathbf{T_{e}}$ and $\mathbf{I_{e}}$ of textual
and visual modalities, we next present the multimodal exchanging process. We
start with the introduction of Transformer basics.

#### Transformer basics.

Transformer Vaswani et al. (2017) is introduced for sequential data modeling
with an encoder-decoder structure. In Transformer, the inputs are linearly
mapped to three matrices: query $\mathbf{Q}$, key $\mathbf{K}$, and value
$\mathbf{V}$. The multi-head self-attention is then computed by:

$\displaystyle\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}})\mathbf{V},$
(7)

where $\sqrt{d_{k}}$ is the scaling factor. Here,
$\text{Softmax}(\mathbf{Q}\mathbf{K}^{T}/\sqrt{d_{k}})$ generates an attention
map matrix, which characterizes the attention scores between tokens.
Generally, Transformer adds a special token cls to the beginning of the input
sequences, which is used to derive the sentence-level embedding. Therefore,
the first row of the attention map represents the attention scores of cls to
all the tokens in the input. In this paper, we only use the Transformer
encoder as in Kenton and Toutanova (2019), where each layer first computes the
multi-head self-attention, followed by a feed-forward network (FFN) with
residual connection He et al. (2016) and layer normalization Ba et al. (2016).

#### CrossTransformer.

Based on Transformer, we propose CrossTransformer, which uses two Transformer
encoders with shared parameters to learn the embeddings of textual and visual
modalities, and exchanges information between multimodalities. The overall
pipeline of CrossTransformer is shown in component ④ of Figure 2. We first add
_cls_ to the beginning of the embeddings generated by the text encoder and the
image encoder, which are taken as the input of the CrossTransformer. After
that, considering that the global contextual information of both input vectors
should be first learned and then exchanged, CrossTransformer sets its shallow
layers as the regular Transformer encoder layer, followed by a number of
exchanging layers. When multimodal fusion finishes, the exchanging process
stops. To achieve this, CrossTransformer introduces two hyper-parameters $\mu$
and $\eta$ to control the start layer and the end layer for information
exchanging, respectively. In each exchanging layer, inspired by Caron et al.
(2021); Liang et al. (2022), we select tokens in one modality with a
proportion of the smallest attention scores to _cls_ and replace their
embeddings vectors with the average embedding of all the tokens in the other
modality. Here, _cls_ is taken as the reference because it generates the
sentence-level embedding.

Specifically, given the embeddings $\mathbf{T_{e}}$ and $\mathbf{I_{e}}$
generated from the text and image encoders, we first add cls to the beginning
of them and obtain the inputs
$\mathbf{T}_{e}(0),\mathbf{I}_{e}(0)\in\mathbb{R}^{(n+1)\times d}$ for
CrossTransformer:

$\displaystyle\mathbf{T}_{e}(0)=\texttt{Concat}\left(\mathbf{v}_{cls}(t_{0}),\mathbf{T}_{e}\right),$
(8)
$\displaystyle\mathbf{I}_{e}(0)=\texttt{Concat}\left(\mathbf{v}_{cls}(i_{0}),\mathbf{I}_{e}\right),$
(9)

where $\mathbf{v}_{cls}(t_{0})$ and $\mathbf{v}_{cls}(i_{0})$ denote the
initial embeddings of $cls$ for textual and visual modalities, respectively.
Here, $n$ is the input text length and $d$ is the embedding dimensionality.
After $\mu$ regular Transformer encoder layers, we can obtain the updated
embeddings $\mathbf{T}_{e}(\mu)$ and $\mathbf{I}_{e}(\mu)$. At layer $\mu+1$,
we step into the exchanging layer, which consists of three sub-modules. The
first sub-module calculates the multi-head self-attention in Equation 7, which
generates intermediate embeddings $\tilde{\mathbf{T}}_{e}(\mu+1)$ for texts
and $\tilde{\mathbf{I}}_{e}(\mu+1)$ for images. The second sub-module selects
a $\theta$-proportion of tokens with the smallest attention scores to _cls_
for both modalities and performs information exchanging. Suppose the token
corresponding to the $k$-th row of $\tilde{\mathbf{T}}_{e}(\mu+1)$ is
selected, whose embedding vector is then updated by:

$\displaystyle\begin{split}\tilde{\mathbf{T}}_{e}(\mu+1)[k,:]&=\frac{1}{n}\sum^{n}_{j=1}\tilde{\mathbf{I}}_{e}(\mu+1)[j,:]\\\
&+\tilde{\mathbf{T}}_{e}(\mu+1)[k,:],\end{split}$ (10)

where we employ residual connection He et al. (2016) to reduce the information
loss caused by replacement. Similarly, we can update the embedding vector in
$\tilde{\mathbf{I}}_{e}(\mu+1)$ by:

$\displaystyle\begin{split}\tilde{\mathbf{I}}_{e}(\mu+1)[k,:]&=\frac{1}{n}\sum_{j=1}^{n}\tilde{\mathbf{T}}_{e}(\mu+1)[j,:]\\\
&+\tilde{\mathbf{I}}_{e}(\mu+1)[k,:].\end{split}$ (11)

After information exchange, the third sub-module inputs the updated embedding
matrices into FFN with layer normalization to obtain the output embeddings at
layer $\mu+1$:

$\displaystyle\mathbf{T}_{e}(\mu+1)=\text{FFN}(\tilde{\mathbf{T}}_{e}(\mu+1)),$
(12)
$\displaystyle\mathbf{I}_{e}(\mu+1)=\text{FFN}(\tilde{\mathbf{I}}_{e}(\mu+1)).$
(13)

The exchanging process continues until reaching the preset end layer $\eta$.
Finally we concatenate the output embeddings of the two Transformers and feed
them into a fully connected network to derive the final fusion embedding
matrix $\mathbf{F}_{e}$.

###  3.3 Training Objective

The overall optimization objective function of MuSE is summarized as

$\displaystyle\mathcal{L}=\mathcal{L}_{task}+\alpha\mathcal{L}_{it}+\beta\mathcal{L}_{ti},$
(14)

where $\mathcal{L}_{task}$, $\mathcal{L}_{it}$, $\mathcal{L}_{ti}$ are the
loss of the main task (MNER or MSA), the image captioning task and the text-
to-image generation task, respectively. Here, $\alpha$ and $\beta$ are two
hyper-parameters to balance the importance of the three terms.

##  4 Experiments

In this section, we evaluate the performance of MuSE by comparing it with
other multimodal fusion methods on two tasks: Multimodal Named Entity
Recognition (MNER) and Multimodal Sentiment Analysis (MSA). Given an input
pair of text $T$ and image $I$, MNER aims to extract a set of entities from
$T$ and classify each entity into one of the pre-defined types; the target of
MSA is to classify each pair to one of the pre-defined sentiment types.

#### Common Implementation.

Before diving into the details of experiments, we introduce the common
implementation in two tasks. We use ResNet He et al. (2016), BERT Kenton and
Toutanova (2019), PixelCNN++ Salimans et al. (2017), NIC-Att Xu et al. (2015)
for ImageEncoder, TextEncoder, ImageDecoder, TextDecoder, respectively. The
batch size is set to $40$, and the learning rate is selected via a small grid
search [1e-4, 5e-5, 1e-5] on the validation set. In ImageEncoder, the input
image is resized to [224, 224] and the encoded image size is set to [8, 8]. In
TextEncoder, the max sequence length is set to $64$. In ImageDecoder, the
input image is resized to [32, 32]. In CrossTransformer, we keep the same
setting with Transformer encoder, except that the dropout rate is selected via
a small grid search [0.1, 0.2, 0.3, 0.4] on the validation set. For loss
selection, we use CrossEntropy for MNER prediction, MSA prediction and image
captioning, and use discretized mix logistic loss for text-to-image generation
as in PixelCNN++. For initialization of cls, we use Kaiming Initialization He
et al. (2015). Our framework is implemented based on PyTorch Paszke et al.
(2019). We run all experiments on $6$ Tesla V100 GPUs.

Due to the space limitation, we move the description of datasets to Appendix
B.

###  4.1 Multimodal Named Entity Recognition

#### Implementation.

Following previous works Yu et al. (2020), we feed the generated fusion
embedding $\mathbf{F}_{e}$ into a CRF Lafferty et al. (2001) layer to make
prediction for MNER. The learning rate and the dropout rate of CRF are set to
$0.0001$ and $0.5$, respectively.

#### Results.

We compare our method with three types of competitors: aggregation-based
methods (GVATT Lu et al. (2018), AdaCAN Zhang et al. (2018), UMT Yu et al.
(2020)), alignment-based methods (ITA Wang et al. (2021a)), and task-specific
methods (RIVA Sun et al. (2020), RpBERT Sun et al. (2021), UMGF Zhang et al.
(2021a)). Details of these approaches can be found in Section A.1. We take
precision, recall and F1 score as the evaluation measures. For all the
measures, the larger value indicates the better model performance. Table 1
summarizes the comparison results. From the table, we see that MuSE achieves
the best performance w.r.t. all the evaluation metrics on all the datasets. In
particular, MuSE leads the runner-up w.r.t. the F1 score by percents of $1.29$
and $1.22$ on Twitter15 and Twitter17, respectively. These results demonstrate
the effectiveness of our exchanging-based method for multimodal fusion.
Further, we notice that in MT-product, our method is the only one that can
beat BERT-CRF, which also justifies the generalizability of MuSE.

Due to the space limitaion, we move the efficiency analysis on MNER datasets
to Appendix C.

Table 1: MNER results (%) w.r.t. precision (P), recall (R) and F1 scores. We highlight the best results in bold. * indicates that the improvements are statistically significant for $p<0.01$ with paired t-test. Methods | Twitter15 | Twitter17 | MT-Product
---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
BERT-CRF | 69.22 | 74.59 | 71.81 | 83.32 | 83.57 | 83.44 | 45.13 | 42.94 | 44.01
GVATT Lu et al. (2018) | 73.96 | 67.90 | 70.80 | 83.14 | 80.38 | 81.87 | - | - | -
AdaCAN Zhang et al. (2018) | 72.75 | 68.75 | 70.69 | 84.16 | 80.24 | 82.15 | 40.15 | 41.66 | 40.89
UMT Yu et al. (2020) | 71.67 | 75.23 | 73.41 | 85.28 | 85.34 | 85.31 | 40.86 | 43.69 | 42.22
RIVA Sun et al. (2020) | - | - | 73.80 | - | - | 87.40 | - | - | -
RpBERT Sun et al. (2021) | - | - | 74.40 | - | - | 87.40 | - | - | -
UMGF Zhang et al. (2021a) | 74.49 | 75.21 | 74.85 | 86.54 | 84.50 | 85.51 | 41.67 | 43.55 | 42.58
ITA Wang et al. (2021a) | - | - | 75.52 | - | - | 85.96 | - | - | -
MuSE | 76.13 | 77.51 | 76.81 | 88.92 | 88.32 | 88.62 | 49.17 | 40.34 | 44.32∗
Table 2: MSA results (%) w.r.t. accuracy and F1 scores. We highlight the best results in bold. * indicates that the improvements are statistically significant for $p<0.01$ with paired t-test. Methods | MVSA-Single | MVSA-Multiple
---|---|---
Accuracy | F1 | Accuracy | F1
CNN-Multi Cai and Xia (2015) | 61.20 | 58.37 | 66.39 | 64.19
DNN-LR Yu et al. (2016) | 61.42 | 61.03 | 67.86 | 66.33
HSAN Xu (2017) | - | 66.90 | - | 67.76
MultiSN Xu and Mao (2017) | 69.84 | 69.63 | 68.86 | 68.11
CoMN Xu et al. (2018) | 70.51 | 70.01 | 69.92 | 69.83
MVAN Yang et al. (2020) | 72.98 | 72.98 | 72.36 | 72.30
ITIN Zhu et al. (2022) | 75.19 | 74.97 | 73.52 | 73.49
MuSE | 75.80∗ | 75.58∗ | 74.10∗ | 73.93∗

###  4.2 Multimodal Sentiment Analysis

#### Implementation.

We apply a one-layer MLP on the fusion embedding of the _cls_ token to
classify the sentiments in MSA. The dropout rate of the MLP is set to $0.5$.

#### Results.

We compare MuSE with two types of approaches: aggregation-based methods (CNN-
Multi Cai and Xia (2015), DNN-LR Yu et al. (2016), HSAN Xu (2017), MultiSN Xu
and Mao (2017), MVAN Yang et al. (2020)), and alignment-based methods (ITIN
Zhu et al. (2022), CoMN Xu et al. (2018)), whose details are provided in
Section A.2. We choose accuracy and F1 score as evaluation metrics. For both
metrics, the larger the value, the better the model performance. The results
are given in Table 2. From the table, we observe that MuSE outperforms other
competitors on both datasets. In particular, compared with the runner-up model
ITIN that is based on inter-modal alignment, MuSE utilizes both the intra-
modal processing and inter-modal exchanging-based fusion, which explains its
superiority.

###  4.3 Ablation Study

We further conduct an ablation study to verify the importance of the main
components in MuSE. Specifically, one variant takes only the textual input. We
call this variant MuSE-only-$\mathbf{T}$. Similarly, we define another variant
MuSE-only-$\mathbf{I}$ that inputs only the vision modality. These two
variants help us recognize the importance of multimodal fusion for downstream
tasks. To further show the significance of CrossTransformer, we remove it from
MuSE and directly employ the embeddings generated from TextEncoder and
ImageEncoder. We call this variant MuSE-w/o-CT. Finally, To show the necessity
of the image captioning task and the text-to-image generation task for
embedding regularization, we propose three variants, in which MuSE-
only-$\mathbf{\mathcal{L}_{task}}$ only optimizes the task loss,
MuSE-w/o-$\mathbf{\mathcal{L}_{it}}$ removes the image captioning loss, and
MuSE-w/o-$\mathbf{\mathbf{\mathcal{L}_{ti}}}$ removes the text-to-image
generation loss.

Table 3 summarizes the results w.r.t. the F1 score on both MNER and MSA tasks.
From the table, we see that: 1) MuSE clearly outperforms MuSE-only-$T$ and
MuSE-only-$I$, which shows the importance of multimodal fusion for textual and
visual representation in the MNER and MSA tasks. 2) MuSE achieves better
performance than MuSE-w/o-CT. This is because CrossTransformer performs
multimodal knowledge exchanging, which boosts the quality of the learned
embeddings for each modality. 3) The outperformance of MuSE over the remaining
three variants shows the necessity of embedding regularization. Since
multimodal data could be projected into different spaces, both the image
captioning task and the text-to-image generation task are used to pull
embeddings of multimodalities into the same space, based on which multimodal
fusion can then be performed. To conclude, all the components in MuSE are
important.

Table 3: Ablation study results (%) on MuSE w.r.t. the F1 score. We highlight the best results in bold.  Methods | Twitter15 | Twitter17 | MT-Product | MVSA-Single | MVSA-Multiple
---|---|---|---|---|---
MuSE-only-$T$ | 72.05 | 84.32 | 43.65 | 63.85 | 61.97
MuSE-only-$I$ | - | - | - | 64.99 | 62.39
MuSE-w/o-CT | 73.26 | 85.46 | 43.78 | 74.24 | 72.35
MuSE-only-$\mathcal{L}_{task}$ | 74.39 | 87.63 | 44.20 | 74.66 | 72.95
MuSE-w/o-$\mathcal{L}_{it}$ | 75.43 | 88.41 | 44.02 | 75.23 | 73.46
MuSE-w/o-$\mathcal{L}_{ti}$ | 75.27 | 88.10 | 44.13 | 75.44 | 73.65
MuSE | 76.81 | 88.62 | 44.32 | 75.58 | 73.93

(a) $\theta$

(b) $\mu$

(c) $\eta$

Figure 4:  Hyper-parameter sensitivity analysis on the exchange proportion
$\theta$, the start layer $\mu$ and the end layer $\eta$ for information
exchanging in CrossTransformer. We short Twitter15, Twitter17, MT-Product,
MVSA-Single, and MVSA-Multiple as T15, T17, MT-P, M-S, and M-M, respectively.

###  4.4 Hyper-parameter Sensitivity Analysis

We end this section with a sensitivity analysis of hyper-parameters in our
model. In particular, we investigate three key hyper-parameters in
CrossTransformer: the exchange proportion $\theta$ of tokens, the start layer
$\mu$ and the end layer $\eta$ for multimodal exchanging. We study one hyper-
parameter with others fixed. Figure 4 shows the experimental results w.r.t.
the F1 score on all the datasets. From the figure, our observations can be
summarized as follows.

#### The exchange proportion $\theta$.

As shown in Figure 4(a), the hyper-parameter $\theta$ plays an important role
in MuSE. We see that the model performance first increases and then drops,
with the increase of $\theta$. When the $\theta$ is too small, the inter-modal
fusion could be inadequate, which adversely affects the model performance.
Further, when $\theta$ is too large, a large number of tokens will be
replaced, which might attenuate the raw intra-modal knowledge. Our results
empirically suggest that setting $\theta$ to be about $10\%$ to $20\%$ could
be a reasonable choice. In our experiments, the default setting of $\theta$ is
$10\%$.

#### The start layer $\mu$.

Since the default number of layers in the regular Transformer is 6, we vary
the value of $\mu$ from $\\{1,2,3,4\\}$. For any $\mu$ value, we fix the end
layer $\eta$ to $4$. From Figure 4(b), we observe that the model performance
improves quickly when $\mu$ increases from $1$ to $2$. A possible explanation
is that the randomly initialized embeddings for cls might not capture the
sentence-level semantics and better contextual representations can be learned
with few layers. This further demonstrates the necessity to use the regular
Transformer layer without exchanging for contextual learning in the shallow
layers of CrossTransformer.

#### The end layer $\eta$.

We fix the start layer $\mu$ to 2 and show the results w.r.t. the end layer
$\eta$ in Figure 4(c). Similar as in Figure 4(b), we also see that the model
performance first rises and then degenerates, as $\eta$ increases. Within
exchanging layers, MuSE fuses knowledge extracted from one modality to the
other, which improves the quality of the learned representations and explains
the performance increase. However, after the fusion completes, more exchanging
layers could degrade the original intra-modal knowledge, which could lead to
performance degeneration. We empirically set the default setting of $\eta$ to
$4$ in the experiments.

##  5 Conclusion

In this paper, we proposed a novel exchanging-based model MuSE for multimodal
fusion. MuSE first uses two encoders to project inputs of texts and images
into separate low-dimensional spaces. Then it regularizes the embeddings and
pulls them into the same space with two generative decoders, which capture the
correlations between texts and images by two tasks: image captioning and text-
to-image generation. After that, based on the regularized embeddings, we
proposed CrossTransformer for multimodal fusion by exchanging token embeddings
from different modalities. We conducted extensive experiments to evaluate the
performance of MuSE on MNER and MSA tasks. Experimental results show that our
method performs favorably against other SOTAs.

##  6 Limitation

In this paper, we propose a new exchange-based multimodal fusion method that
does better on the MNER and MSA tasks. Moreover, the hyperparameters of the
exchange module are analyzed in detail. In the future, we will extend this
method to more tasks.

##  7 Ethical Considerations

The proposed method has no obvious potential risks. All the scientific
artifacts used/created are properly cited/licensed, and the usage is
consistent with their intended use. Also, we open up our codes and hyper-
parameters to facilitate future reproduction without repeated energy cost.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443.
  * Bayoudh et al. (2021) Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtibaa. 2021.  A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets.  _The Visual Computer_ , pages 1–32.
  * Cai and Xia (2015) Guoyong Cai and Binbin Xia. 2015.  Convolutional neural networks for multimedia sentiment analysis.  In _Natural language processing and chinese computing_ , pages 159–167. Springer.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9650–9660.
  * Colombo et al. (2021) Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chloé Clavel. 2021.  Improving multimodal fusion via mutual dependency maximisation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 231–245.
  * De Vries et al. (2017) Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. 2017.  Modulating early visual processing by language.  _Advances in Neural Information Processing Systems_ , 30.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773.
  * Guo et al. (2019) Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019.  Deep multimodal representation learning: A survey.  _IEEE Access_ , 7:63373–63394.
  * Han et al. (2021) Wei Han, Hui Chen, and Soujanya Poria. 2021.  Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9180–9192.
  * Hazirbas et al. (2016) Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. 2016.  Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture.  In _Asian conference on computer vision_ , pages 213–228. Springer.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR.
  * Jiang et al. (2022) Zijie Jiang, Hajime Taira, Naoyuki Miyashita, and Masatoshi Okutomi. 2022.  Self-supervised ego-motion estimation based on multi-layer fusion of rgb and inferred depth.  _arXiv preprint arXiv:2203.01557_.
  * Ju et al. (2021) Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, and Guodong Zhou. 2021.  Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4395–4405.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Liang et al. (2022) Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. 2022\.  Not all patches are what you need: Expediting vision transformers via token reorganizations.  _arXiv preprint arXiv:2202.07800_.
  * Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018.  Visual attention model for name tagging in multimodal social media.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1990–1999.
  * Moon et al. (2018) Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018.  Multimodal named entity recognition for short social media posts.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 852–860.
  * Nagrani et al. (2021) Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021.  Attention bottlenecks for multimodal fusion.  _Advances in Neural Information Processing Systems_ , 34.
  * Niu et al. (2016) Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Saddik. 2016.  Sentiment analysis on multi-view social data.  In _International Conference on Multimedia Modeling_ , pages 15–27. Springer.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Salimans et al. (2017) Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. 2017.  Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications.  _arXiv preprint arXiv:1701.05517_.
  * Shankar (2021) Shiv Shankar. 2021.  Neural dependency coding inspired multimodal fusion.  _arXiv preprint arXiv:2110.00385_.
  * Song et al. (2020) Sijie Song, Jiaying Liu, Yanghao Li, and Zongming Guo. 2020.  Modality compensation network: Cross-modal adaptation for action recognition.  _IEEE Transactions on Image Processing_ , 29:3957–3969.
  * Sun et al. (2020) Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, and Yuanyi Chen. 2020.  Riva: a pre-trained tweet multimodal model based on text-image relation for multimodal ner.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1852–1862.
  * Sun et al. (2021) Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021.  Rpbert: A text-image relation propagation-based bert model for multimodal ner.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13860–13868.
  * Valada et al. (2020) Abhinav Valada, Rohit Mohan, and Wolfram Burgard. 2020.  Self-supervised model adaptation for multimodal semantic segmentation.  _International Journal of Computer Vision_ , 128(5):1239–1285.
  * Van den Oord et al. (2016) Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016.  Conditional image generation with pixelcnn decoders.  _Advances in neural information processing systems_ , 29.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164.
  * Wang et al. (2021a) Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a.  Ita: Image-text alignments for multi-modal named entity recognition.  _arXiv preprint arXiv:2112.06482_.
  * Wang et al. (2021b) Yikai Wang, Wenbing Huang, Fuchun Sun, Fengxiang He, and Dacheng Tao. 2021b.  Channel exchanging networks for multimodal and multitask dense image prediction.  _arXiv preprint arXiv:2112.02252_.
  * Wang et al. (2020) Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. 2020\.  Deep multimodal fusion by channel exchanging.  _Advances in Neural Information Processing Systems_ , 33:4835–4845.
  * Wu et al. (2021) Yang Wu, Zijie Lin, Yanyan Zhao, Bing Qin, and Li-Nan Zhu. 2021.  A text-centered shared-private framework via cross-modal prediction for multimodal sentiment analysis.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4730–4738.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057. PMLR.
  * Xu (2017) Nan Xu. 2017.  Analyzing multimodal public sentiment based on hierarchical semantic attentional network.  In _2017 IEEE International Conference on Intelligence and Security Informatics (ISI)_ , pages 152–154. IEEE.
  * Xu and Mao (2017) Nan Xu and Wenji Mao. 2017.  Multisentinet: A deep semantic network for multimodal sentiment analysis.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pages 2399–2402.
  * Xu et al. (2018) Nan Xu, Wenji Mao, and Guandan Chen. 2018.  A co-memory network for multimodal sentiment analysis.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 929–932.
  * Yang et al. (2020) Xiaocui Yang, Shi Feng, Daling Wang, and Yifei Zhang. 2020.  Image-text multimodal emotion classification via multi-view attentional network.  _IEEE Transactions on Multimedia_ , 23:4014–4026.
  * Yu et al. (2020) Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020.  Improving multimodal named entity recognition via entity span detection with unified multimodal transformer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3342–3352.
  * Yu et al. (2016) Yuhai Yu, Hongfei Lin, Jiana Meng, and Zhehuan Zhao. 2016.  Visual and textual sentiment analysis of a microblog using deep convolutional neural networks.  _Algorithms_ , 9(2):41.
  * Zeng et al. (2019) Jin Zeng, Yanfeng Tong, Yunmu Huang, Qiong Yan, Wenxiu Sun, Jing Chen, and Yongtian Wang. 2019.  Deep surface normal estimation with hierarchical rgb-d fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6153–6162.
  * Zhang et al. (2021a) Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021a.  Multi-modal graph fusion for named entity recognition with targeted visual guidance.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14347–14355.
  * Zhang et al. (2021b) Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. 2021b.  Cross-modal contrastive learning for text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 833–842.
  * Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018.  Adaptive co-attention network for named entity recognition in tweets.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Zhu et al. (2022) Tong Zhu, Leida Li, Jufeng Yang, Sicheng Zhao, Hantao Liu, and Jiansheng Qian. 2022\.  Multimodal sentiment analysis with image-text interaction network.  _IEEE Transactions on Multimedia_.

##  Appendix A Related Works

###  A.1 Multimodal Named Entity Recognition (MNER)

Existing works for MNER can be mainly categorized into three types:
aggregation-based, alignment-based and task-specific. For example, AttMNER
Moon et al. (2018), GVATT Lu et al. (2018) and AdaCAN Zhang et al. (2018) are
pioneering works that employ the attention mechanism to aggregate text and
image features. UMT Yu et al. (2020) further uses Transformer to enhance the
aggregation performance. The alignment-based method ITA Wang et al. (2021a)
introduces three types of alignments between texts and images to improve the
classification results. There are also some task-specific works, such as RIVA
Sun et al. (2020) and RpBERT Sun et al. (2021), which aim to solve the
mismatch problem between texts and images. Further, UMGF Zhang et al. (2021a)
proposes a unified multimodal graph fusion based on fine-grained graph
modeling.

###  A.2 Multimodal Sentiment Analysis (MSA)

MSA has recently attracted growing attention for its capability in learning
cross-domain knowledge for sentiment analysis. Similar as in MNER, there are
also aggregation-based methods for MSA. For example, CNN-Multi Cai and Xia
(2015) first uses three CNNs to learn visual and textual features and then
aggregates them for sentiment prediction. Further, DNN-LR Yu et al. (2016)
introduces pre-trained CNNs to improve the performance. There also exist
approaches Xu (2017); Xu and Mao (2017); Yang et al. (2020) that present
hierarchical designs to enhance aggregation networks. Another type of methods
for MSA is based on aligning techniques. For example, CoMN Xu et al. (2018)
introduces mutual information to align the influences of texts and images;
ITIN Zhu et al. (2022) utilizes cross-modal alignment to capture the region-
word correspondence with the adaptive gating module, which has been shown to
achieve superior results.

##  Appendix B Datasets

###  B.1 MNER

We evaluate our method on two public datasets Twitter15 Zhang et al. (2018)
and Twitter17 Lu et al. (2018), and we keep the same Train/Validation/Test
splits. The two datasets are comprised of multimodal user posts, which are
collected from Twitter during 2014-2015 and 2016-2017, respectively. Each
dataset contains four types of entities, namely person, location, organization
and miscellaneous. We use the released version222Twitter15 and Twitter17
download url: https://github.com/jefferyYu/UMT. from UMT Yu et al. (2020).
Additionally, we collected a product dataset MT-Product from a well-known
E-commercial platform, which has only one type of entity, namely product. In
this dataset, we only keep one sample for each product to ensure that products
in the training and test sets do not overlap, to further test the model
generalizability. We randomly split the dataset into the
Training/Validation/Test sets with the split ratio 6/2/2. We will release this
dataset later. The statistics of the three datasets are shown in Table 4.

###  B.2 MSA

We evaluate our method on two public datasets MVSA-Single and MVSA-
Multiple333MVSA-Single and MVSA-Multiple download url:
https://mcrlab.net/research/mvsa-sentiment-analysis-on-multi-view-social-
data/., which are released in Niu et al. (2016). The two datasets are both
collected from Twitter. For fair comparison, we preprocess the datasets
following works Xu and Mao (2017); Zhu et al. (2022): 1) Remove the pairs
whose image label and text label are inconsistent. 2) Split the datasets into
the Training/Validation/Test sets randomly with the split ratio 8/1/1. Each
dataset has three sentiment types, namely _positive_ , _neutral_ , and
_negative_. The statistics of the two datasets are shown in Table 4.

Table 4: The statistics of datasets. Task | Dataset | Training | Validation | Test
---|---|---|---|---
Multimodal Named Entity Recognition | Twitter15 | 4000 | 1000 | 3257
Twitter17 | 3373 | 723 | 723
MT-Product | 2680 | 893 | 895
Multimodal Sentiment Analysis | MVSA-Single | 3600 | 440 | 471
MVSA-Multiple | 13600 | 1720 | 1707

##  Appendix C Efficiency Analysis

To evaluate the efficiency of MuSE, we compare it to several representative
cross-attention-based text-vision fusion methods from the number of parameters
(#Params), performance (Perf), training time in one epoch (TT), and evaluation
time (ET). Table 5 gives the detailed comparsion results. The results show
that MuSE significantly improves performance on both datasets but also comes
with higher time costs. Further, we can observe that the time cost of MuSE
mainly comes from the exchanging module. We have rewritten the transformer
layer of PyTorch but have not optimized it well. In the future, we will
further optimize the code implementation.

Table 5: Efficiency comparison. Models(#Params) | T15 | T17
---|---|---
Perf | TT | ET | Perf | TT | ET
ViLBERT + bilstm-crf (210M) | 73.88 | 90s | 12s | 87.23 | 78s | 8s
UMT (208M) | 73.41 | 86s | 10s | 85.31 | 74s | 7s
MuSE w/o two tasks (204M) | 74.39 | 145s | 28s | 87.63 | 118s | 27s
MuSE (236M) | 76.13 | 180s | 32s | 88.62 | 161s | 30s