# ScaleVLAD: Improving Multimodal Sentiment Analysis via
Multi-Scale Fusion of Locally Descriptors

Huaishao Luo1 , Lei Ji2, Yanyong Huang3, Bin Wang4, Shenggong Ji5, Tianrui Li1
1Southwest Jiaotong University, Chengdu, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
2Microsoft Research Asia, Beijing, China
3Southwestern University of Finance and Economics, Chengdu, China
4Ocean University of China, Qingdao, China
5Tencent, Shenzhen, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>This work was done during the
first author’s internship in MSR Asia

###### Abstract

Fusion technique is a key research topic in multimodal sentiment analysis. The
recent attention-based fusion demonstrates advances over simple operation-
based fusion. However, these fusion works adopt single-scale, i.e., token-
level or utterance-level, unimodal representation. Such single-scale fusion is
suboptimal because that different modality should be aligned with different
granularities. This paper proposes a fusion model named ScaleVLAD to gather
multi-Scale representation from text, video, and audio with shared Vectors of
Locally Aggregated Descriptors to improve unaligned multimodal sentiment
analysis. These shared vectors can be regarded as shared topics to align
different modalities. In addition, we propose a self-supervised shifted
clustering loss to keep the fused feature differentiation among samples. The
backbones are three Transformer encoders corresponding to three modalities,
and the aggregated features generated from the fusion module are feed to a
Transformer plus a full connection to finish task predictions. Experiments on
three popular sentiment analysis benchmarks, IEMOCAP, MOSI, and MOSEI,
demonstrate significant gains over baselines.

##  1 Introduction

Multimodal Sentiment Analysis (MSA) has been a hot research direction with the
increasing number of user-generated videos available on online platforms such
as YouTube and Facebook in recent years Poria et al. (2020); Tsai et al.
(2019); Zadeh et al. (2017).

(a) Single-scale alignment

(b) Multi-scale alignment

Figure 1: Illustration of alignment between text, video, and audio.
(LABEL:sub@fig:singlescale_alignment) single-scale alignment.
(LABEL:sub@fig:scale_alignment) our multi-scale alignment.

Its main objective is to identify sentiment and emotion with multimodal
signals such as textual, visual, and acoustic information. Compared with
unimodal sentiment analysis, multimodal fusion can provide more comprehensive
information and capture more emotional characteristics, which leads to robust
and salient improvements Yang, Xu, and Gao (2020); Yu et al. (2021). For
example, to judge the sentiment of _this movie is sick_ is a non-trivial task
due to the existing language ambiguity only from this sentence, and if given
the acoustic and visual modalities, e.g., a loud voice and a smile, this
sentence will certainly be predicted as positive Zadeh et al. (2017); Wang,
Wan, and Wan (2020).

There are two main components in multimodal sentiment analysis: unimodal
representation and information fusion. For the unimodal representation, there
are some off-the-shelf methods. These methods are elaborate and specialized
for each modality or can be improved with pretraining on extra pure datasets,
e.g., MFCC for audio and BERT encoding for text Devlin et al. (2019). Thus,
multimodal information fusion is the key to affect performance Poria et al.
(2020); Zhang et al. (2020). Most of the works focus on investigating
effective multimodal fusion. These fusion methods can be categorized into
including but not limited to simple operation-based Poria et al. (2016),
attention-based Zadeh et al. (2018c); Gu et al. (2018); Akhtar et al. (2019);
Han et al. (2021); Rahman et al. (2020), tensor-based Zadeh et al. (2017),
translation-based Pham et al. (2019); Wang, Wan, and Wan (2020); Mai, Hu, and
Xing (2020), GANs-based Peng and Qi (2019), graph-based Yang et al. (2021),
and routing-based methods Tsai et al. (2020). The fusion target is to learn a
modality-invariant embedding space, then use the modality-invariant feature or
integrate the modality-invariant with modality-specific features to finish the
final prediction.

However, most of the fusion methods either adopt the token-level or the
utterance-level unimodal representation. Such a single-scale fusion is
suboptimal because different modalities need to align with different
granularities. For example, the ‘really really good’ shown in Figure 1. The
single-scale alignment of the three tokens can not capture the intense
emotion. Instead, they should be regarded as an entirety shown in Figure 1(b).
Besides, the visual and acoustic features do not have apparent semantic
boundaries due to variable sampling rates, leading to inherent data non-
alignment for each modality Tsai et al. (2019). Although the attention-based
methods can make each token in one modality cover long-range contexts in other
modalities, they are still single-scale alignment and can not capture _many
tokens-to-many tokens_ relationship.

To this end, we propose a multi-scale fusion method called ScaleVLAD to gather
multi-Scale representation from text, video, and audio with shared Vectors of
Locally Aggregated Descriptors to address the _unaligned_ multimodal sentiment
analysis. Instead of detecting the boundary of different semantic scales in
each modality, ScaleVLAD utilizes learnable shared latent semantic vectors to
select and aggregate the modality features automatically. These latent
semantic vectors, regarded as different semantic topics, are shared across
different modalities and scales. Thus they can reduce the semantic gap between
modalities and align various scale features naturally. In our implementation,
we use three Transformer-based modules Vaswani et al. (2017) to extract
unimodal representation from text, video, and audio, respectively. Then, the
unimodal feature sequences are fed to the ScaleVLAD module with different
scales of shifted windows. The aggregated features from the ScaleVLAD module
are used to predict the final output via a Transformer and a full connection
layer. Figure 2 shows the main structure of the proposed ScaleVLAD. Besides,
to keep the differentiation of the fused feature among samples and leverage
label information effectively, we propose a self-supervised shifted clustering
loss to train the model jointly. This loss will pull clusters of samples
belonging to the same category (or close score) together in embedding space.
The contribution of this paper can be summarized as follows:

1) We propose a multi-scale fusion method ScaleVLAD to address the unaligned
multimodal sentiment analysis. It is a flexible approach to fuse unimodal
representation with a multi-scale perspective.

2) We propose a self-supervised shifted clustering loss to keep the fused
feature differentiation among samples and leverage label information
effectively.

3) We report new records on three benchmark datasets, including IEMOCAP Busso
et al. (2008), CMU-MOSI Zadeh et al. (2016), and CMU-MOSEI Zadeh et al.
(2018b). Extensive experiments validate the effectiveness of ScaleVLAD.

##  2 Related Works

###  2.1 Multimodal Sentiment Analysis

In recent years, multimodal sentiment analysis has become a popular research
topic as the increasing of user-generated multimedia data on online
communities, blogs, and multimedia platforms. It mainly focuses on integrating
multiple heterogeneous resources, such as textual, visual, and acoustic
signals to comprehend varied human emotions Morency, Mihalcea, and Doshi
(2011); Poria et al. (2020). Previous researchers mainly focus on unimodal
representation learning and multimodal fusion. For the unimodal
representation, Hazarika, Zimmermann, and Poria (2020) attempted to factorize
modality features in joint spaces and presented modality-invariant and
modality-specific representations across different modalities. Yu et al.
(2021) designed a unimodal label generation strategy based on the self-
supervised approach to acquire information-rich unimodal representations by
learning one multimodal task and three unimodal subtasks. Wang et al. (2019)
constructed a recurrent attended variation embedding network to model the
fine-grained structure of nonverbal sub-word sequences and dynamically shift
word representations based on nonverbal cues.

For the multimodal fusion, the previous methods can be divide into simple
operation-based Poria et al. (2016), attention-based Zadeh et al. (2018c); Gu
et al. (2018); Akhtar et al. (2019); Han et al. (2021); Rahman et al. (2020),
tensor-based Zadeh et al. (2017); Verma et al. (2019, 2020), translation-based
Pham et al. (2019); Wang, Wan, and Wan (2020); Mai, Hu, and Xing (2020), GANs-
based Peng and Qi (2019), graph-based Yang et al. (2021); Mai et al. (2020),
and routing-based methods Tsai et al. (2020), etc. Some works assumed the
given multimodal sequences are aligned with each word’s boundary Pham et al.
(2019); Gu et al. (2018); Dumpala et al. (2019); Rahman et al. (2020).
However, some modalities, e.g., video and audio, exist inherent data non-
alignment due to variable sampling rates. Thus, modeling unaligned multimodal
sequences is more flexible and practical. Tsai et al. (2019); Yang, Xu, and
Gao (2020); Siriwardhana et al. (2020) used multiple cross-modal Transformers
to model unaligned multimodal language sequences. Yang et al. (2021) proposed
a parameter-efficient and interpretable graph-based neural model by
integrating an efficient trimodal-temporal graph fusion operation and dynamic
pruning technique.

Figure 2: The main structure of our ScaleVLAD, which comprises four
components, including three unimodal encoders, and a fusion module. The model
is trained with a task-related loss and an extra clustering loss.

This paper aims at unaligned multimodal sentiment analysis. Unlike previous
studies adopting the token-level or the utterance-level unimodal
representation, we propose a multi-scale fusion method to align different
granularity information from multiple modalities.

###  2.2 VLAD, Vector of Locally Aggregated Descriptors

The Vector of Locally Aggregated Descriptors (VLAD) Jégou et al. (2010);
Arandjelovic and Zisserman (2013) has achieved great impacts in aggregating
discriminative features for various scenarios, including video retrieval and
video classification. NetVLAD Arandjelovic et al. (2016) extending from the
VLAD is an end-to-end differentiable layer that could be readily plugged into
many existing neural models. This paper borrows the idea of VLAD and NetVLAD
to align different modalities, e.g., text, video, and audio, instead of using
to be as a discriminative feature learner. Wang, Zhu, and Yang (2021) has a
similar motivation that leverages NetVLAD to reduce the gap of locally learned
features from texts and videos. However, their objective is for text-video
local similarity matching, and we have a different target. Besides, we
introduce multi-scale features for enhanced fusion performance. Hausler et al.
(2021) also presents a multi-scale fusion by deriving patch-level features
from NetVLAD residuals. However, it is designed for place recognition and only
on visual modality. We focus on unaligned multimodal sentiment analysis and
involves text, video, and audio modalities.

##  3 Framework

Given a set of multimodal signals including text $\mathcal{T}$, video clips
$\mathcal{V}$, and audios $\mathcal{A}$, the target is to predict their
sentiment. Specifically, these signals can be regarded as a set of triplets
$(T_{i},V_{i},A_{i})$, where $T_{i}\in\mathcal{T}$, $V_{i}\in\mathcal{V}$ and
$A_{i}\in\mathcal{A}$. The $T_{i}$, $V_{i}$, and $A_{i}$ contain a sequence of
tokens, respectively, such that
$T_{i}=\big{\\{}t_{i}^{j}|j\in[1,|T_{i}|]\big{\\}}$,
$V_{i}=\big{\\{}\boldsymbol{v}_{i}^{j}|j\in[1,|V_{i}|]\big{\\}}$, and
$A_{i}=\big{\\{}\boldsymbol{a}_{i}^{j}|j\in[1,|A_{i}|]\big{\\}}$, where
$t_{i}^{j}$ is word token, $\boldsymbol{v}_{i}^{j}$ is visual feature, and
$\boldsymbol{a}_{i}^{j}$ denotes acoustic feature. We regard the visual
features and acoustic features as tokens for a consistent description with the
word tokens. Multimodal sentiment analysis aims to learn a function
$f(T_{i},V_{i},A_{i})$ to get the sentiment score or emotion category. The
function learning can be regarded as either a regression or a classification
task.

Figure 2 demonstrates our framework. We focus on the multi-scale fusion module
and a training loss, S3C loss, in this paper. Besides, three unimodal
encoders, a text encoder, a video encoder, and an audio encoder, are also
introduced in detail in this section.

###  3.1 Modality Representation Learning

The unimodality representation is the footstone of this model and will affect
the performance of the subsequential fusion module. We use Transformer Vaswani
et al. (2017) with different layers to encode original text $T_{i}$, raw video
feature sequence $V_{i}\in\mathbb{R}^{|V_{i}|\times\hat{d}_{v}}$, and raw
audio feature sequence $A_{i}\in\mathbb{R}^{|A_{i}|\times\hat{d}_{a}}$, where
$\hat{d}_{v}$ and $\hat{d}_{a}$ are the dimensions of the raw feature. The raw
video feature and raw audio feature are extracted with pretrained toolkits
following previous works Zadeh et al. (2017); Yu et al. (2021). For the text
encoder, we use the pretrained 12-layers BERT Devlin et al. (2019) and
12-layers T5 Raffel et al. (2020) to extract text representation
$\mathcal{F}_{T_{i}}\in\mathbb{R}^{|T_{i}|\times d_{t}}$ since the tremendous
success of the pre-trained language model on many downstream NLP tasks, where
$d_{t}=768$ is the dimension of the text representation.

$\displaystyle\mathcal{F}_{T_{i}}=\texttt{Transformer}_{\texttt{T}}(T_{i}),$
(1)

where $\texttt{Transformer}_{\texttt{T}}$ means the Transformer-based text
encoder, e.g., BERT and T5 in our implementation.

Similarly, the video feature sequence
$\mathcal{F}_{V_{i}}\in\mathbb{R}^{|V_{i}|\times d_{v}}$ and audio feature
sequence $\mathcal{F}_{A_{i}}\in\mathbb{R}^{|A_{i}|\times d_{a}}$ can be
calculated with $V_{i}$ and $A_{i}$ respectively as follows,

$\displaystyle\mathcal{F}_{V_{i}}=\texttt{Transformer}_{\texttt{V}}(V_{i}),$
(2)
$\displaystyle\mathcal{F}_{A_{i}}=\texttt{Transformer}_{\texttt{A}}(A_{i}),$
(3)

where $\texttt{Transformer}_{\texttt{V}}$ and
$\texttt{Transformer}_{\texttt{A}}$ are Transformer-based video encoder and
Transformer-based audio encoder, respectively, both of them are randomly
initialized. $d_{v}$ and $d_{a}$ are the dimension of the video feature and
audio feature, respectively.

###  3.2 ScaleVLAD Module

After generating the unimodality representation, the framework comes to the
fusion module. We propose a multi-scale fusion method to cover different
granularities of unimodality representation in this paper. Different full
connection layers are used for the generated $\mathcal{F}_{T_{i}}$,
$\mathcal{F}_{V_{i}}$, and $\mathcal{F}_{A_{i}}$ to map the hidden size to a
common size $d_{s}$ before the following modules if their current hidden sizes
are not equal to this value. When considering the fusion of the three
unimodality features, especially with different granularities, a core problem
is aligning different semantic units. However, the semantic unit of each
unimodality has no clear alignment boundary and can not be fused directly. A
feasible approach is to assume some shared semantic vectors among these
unimodality features and align them to these shared anchors. Such shared
vectors can be regarded as shared topics and can also be shared across
different unimodality scales.

Motivated by this spirit and Inspired by the VLAD and NetVLAD, we propose a
ScaleVLAD module to fuse different unimodality representations. The different
scale information of unimodality is generated by mean pooling with different
kernel size (the stride size is the same as the kernel size) in our
implementation. Specifically, for $m$-scale unimodality representation
$\mathcal{F}_{M_{i}},M\in\\{T,V,A\\}$, the scaled features can be denoted as
$\mathcal{F}_{M_{i}}^{(m)}=\\{\boldsymbol{f}_{j}^{(m)}|j\in[1,|\mathcal{F}_{M_{i}}^{(m)}|]\\}$,
where $\boldsymbol{f}_{j}^{(m)}$ is generated via mean pooling with kernel
size $m$. The $\mathcal{F}_{M_{i}}^{(m)}$ is equal to $\mathcal{F}_{M_{i}}$
when $m=1$. Assuming there are $K$ shared semantic vectors
$\\{\boldsymbol{c}_{k}|k\in[1,K]\\}$ with $d_{s}$ dimension. The similarity
between the $m$-scale feature $\boldsymbol{f}_{j}^{(m)}$ and the shared
vectors can be calculated by dot-product operation following Arandjelovic et
al. (2016),

$\displaystyle
w_{ij}^{(m)}=\frac{\exp(\boldsymbol{f}_{i}^{(m)}\boldsymbol{c}_{j}^{\top}+b_{j})}{\sum_{k=1}^{K}\exp(\boldsymbol{f}_{i}^{(m)}\boldsymbol{c}_{k}^{\top}+b_{k})},$
(4)

where $b_{j}$ and $b_{k}$ are learnable biases, the shared semantic vectors
are jointly learned with the whole model. Then the aggregated feature on each
vector can be generated as follows,

$\displaystyle\hat{\boldsymbol{r}}_{j}^{(m)}=$
$\displaystyle\sum\nolimits_{i=1}^{|\mathcal{F}_{M_{i}}^{(m)}|}w_{ij}^{(m)}(\boldsymbol{f}_{i}^{(m)}-\hat{\boldsymbol{c}}_{j}),$
(5) $\displaystyle\boldsymbol{r}_{j}^{(m)}=$
$\displaystyle\hat{\boldsymbol{r}}_{j}^{(m)}/{\lVert\hat{\boldsymbol{r}}_{j}^{(m)}\rVert_{2}},$
(6)

where $\hat{\boldsymbol{c}}_{j}$ has the same size as $\boldsymbol{c}_{j}$,
and using two groups of similar vectors increases the adaptation capability as
described in Arandjelovic et al. (2016). The output $\boldsymbol{r}_{j}^{(m)}$
can be regarded as the aligned feature for unimodality with $m$-scale. Thus,
the aggregated feature corresponding to $\mathcal{F}_{M_{i}}$ can be generated
as follows,

$\displaystyle\hat{\boldsymbol{u}}=\texttt{stack}([\boldsymbol{r}_{1}^{(m)},\boldsymbol{r}_{2}^{(m)},\cdots,\boldsymbol{r}_{K}^{(m)}]),$
(7)
$\displaystyle\boldsymbol{u}_{M_{i}}^{(m)}=\texttt{LN}(\texttt{GELU}(\hat{\boldsymbol{u}}\mathbf{W}\\!_{M}+\boldsymbol{b}_{M})),$
(8)

where stack is a stack operation and
$\hat{\boldsymbol{u}}\in\mathbb{R}^{Kd_{s}}$,
$\mathbf{W}\\!_{M}\in\mathbb{R}^{Kd_{s}\times d_{s}}$ and
$\boldsymbol{b}_{M}\in\mathbb{R}^{d_{s}}$ ($M\in\\{T,V,A\\}$) are learnable
weights and biases, GELU and LN are GELU activate function Hendrycks and
Gimpel (2016) and Layer Normalization operation Ba, Kiros, and Hinton (2016),
respectively.

The fusion and prediction are conducted on the multi-scale aggregated features
$\boldsymbol{u}_{M_{i}}^{(m)}$. We stack all the representation with different
scales $m_{1},m_{2},\dots$ together to get representation matrix,
$R_{i}=[\boldsymbol{u}_{T_{i}}^{(m_{1})},\boldsymbol{u}_{V_{i}}^{(m_{1})},\boldsymbol{u}_{A_{i}}^{(m_{1})},\boldsymbol{u}_{T_{i}}^{(m_{2})},\dots,\bar{\boldsymbol{f}}_{T_{i}},\bar{\boldsymbol{f}}_{V_{i}},\bar{\boldsymbol{f}}_{A_{i}}]\in\mathbb{R}^{(3\cdot|m|+3)\times
d_{s}}$, where $|m|$ means the number of scales,
$\bar{\boldsymbol{f}}_{M_{i}}$ $(M\in\\{T,V,A\\})$ is the mean pooling result
on $\mathcal{F}_{M_{i}}$. After obtaining $R_{i}$, a randomly initialized
Transformer encoder $\texttt{Transformer}_{\texttt{F}}$ is utilized to
interact the learned multi-scale representation:

$\displaystyle\hat{R}_{i}=\texttt{Transformer}_{\texttt{F}}(R_{i}).$ (9)

Finally, the score or probability can be calculated as,

$\displaystyle\hat{\boldsymbol{r}}=\texttt{max-pooling}(\hat{R}_{i}),$ (10)
$\displaystyle\boldsymbol{o}_{i}=\hat{\boldsymbol{r}}\mathbf{W}\\!_{r}+\boldsymbol{b}_{r},$
(11)

where $\hat{\boldsymbol{r}}\in\mathbb{R}^{d_{s}}$ is the max pooling result of
$\hat{R}_{i}$, $\mathbf{W}_{r}\in\mathbb{R}^{d_{s}\times c}$ and
$\boldsymbol{b}_{r}\in\mathbb{R}^{c}$ are learnable weights and biases, $c$ is
the number of categories for classification task or 1 for regression task.

###  3.3 S3C Loss, Self-supervised Shifted Clustering Loss

Beyond proposing the ScaleVLAD module to capture and align different
granularities of unimodality representation, we proposed an extra self-
supervised shifted clustering loss (S3C Loss) to keep the differentiation of
the fused feature among samples and to leverage label information effectively.

For the fusion feature $\hat{\boldsymbol{r}}$ of each sample from Eq. (10), we
first perform $k$-means to obtain $C$ clusters111We use the Faiss
(https://github.com/facebookresearch/faiss) to finish clustering in our
implementation.. We refer to the $i$-th cluster center as
$\boldsymbol{z}_{i}\in\mathbb{R}^{d_{s}}$ and refer to all cluster centers as
a matrix $Z\in\mathbb{R}^{C\times d_{s}}$. The clustering operation is
calculated on all representations of training samples at each epoch beginning.
For the same sample in the running epoch, we assign its cluster center index
$i$ as a classified label. The S3C Loss can be obtained as follows,

$\displaystyle\boldsymbol{p}=\texttt{softmax}(Z\hat{\boldsymbol{r}}),$ (12)
$\displaystyle\mathcal{L}_{s3c}=-\frac{1}{N}\sum_{i=1}^{N}\left(\mathbb{I}(i)(\log(\boldsymbol{p}))^{\top}\right),$
(13)

where $\mathbb{I}(i)$ means the one-hot vector with length $C$ and its $i$-th
value is 1, $N$ is the number of training samples.

This loss is self-supervised but the clustering centers are not stable at the
beginning of the training stage. So we set a start epoch $s_{s3c}$ to train
with $\mathcal{L}_{s3c}$ instead of optimizing it from training beginning.
Such a setting makes the features used for clustering semantically relate to
the group-truth labels. To make the cluster centers stable, we adopt a shifted
update with a momentum parameter $\alpha$ as $Z^{(t)}=\alpha
Z^{(t-1)}+(1-\alpha)Z$ and use $Z^{(t)}$ to replace $Z$ at each iteration. The
$\alpha$ is set as a constant of 0.99 in our experiments. The clustering loss
makes the fusion features differentiate in the embedding space.

To improve the weak robustness caused by the unknown ground-truth cluster
number of the fusion space, we design multiple clustering, e.g., with $C_{1}$
clusters and $C_{2}$ clusters. Thus, the $\mathcal{L}_{s3c}$ will be replaced
by
$\mathcal{L}_{s3c}=\mathcal{L}_{s3c}^{(C_{1})}+\mathcal{L}_{s3c}^{(C_{2})}+\dots$,
where $\mathcal{L}_{s3c}^{(C_{i})}$ means $\mathcal{L}_{s3c}$ with $C_{i}$
clusters.

###  3.4 Training Objectives

The overall objective of the model is to minimize:

$\displaystyle\mathcal{L}=\mathcal{L}_{task}+\mathcal{L}_{s3c},$ (14)

where $\mathcal{L}_{s3c}$ is the S3C Loss, and $\mathcal{L}_{task}$ is the
task loss. The task loss has different formulations for the classification
task and regression task. For the classification task, we use cross-entropy
error with $\boldsymbol{o}_{i}$ in Eq. (11) as
$\mathcal{L}_{task}=-\frac{1}{N}\sum_{i=1}^{N}(\mathbb{I}(y_{i})(\log(\boldsymbol{o}_{i}))^{\top})$,
where $\mathbb{I}(y_{i})$ means the one-hot vector of $y_{i}$. For the
regression task, we use mean MSE as the training objective as
$\mathcal{L}_{task}=\frac{1}{N}\sum_{i=1}^{N}(\lVert
y_{i}-\boldsymbol{o}_{i}\rVert_{2}^{2})$. $y_{i}$ is the category for
classification or the score for regression, and $N$ is the number of training
samples.

##  4 Experiments

We conduct experiments to evaluate the effectiveness of the proposed
framework. The datasets, experimental settings, and results are introduced in
this section.

###  4.1 Datasets

We evaluate our framework on three benchmark datasets, IEMOCAP Busso et al.
(2008), CMU-MOSI Zadeh et al. (2016), and CMU-MOSEI Zadeh et al. (2018b).
These datasets provide unaligned language, visual, and acoustic signals for
multimodal sentiment analysis.

IEMOCAP IEMOCAP Busso et al. (2008) consists of 10,000 videos for human
emotion analysis. We follow Wang et al. (2019) and select four emotions
(happy, sad, angry, and neutral) for emotion recognition. The task of this
dataset is a multilabel task (e.g., a person can be sad and angry
simultaneously). The metric used on this dataset is the binary classification
accuracy (Acc) and the F1 score of the predictions.

CMU-MOSI Multimodal Opinion Sentiment and Emotion Intensity Zadeh et al.
(2016) is sentence-level sentiment analysis and emotion recognition in online
videos. CMU-MOSI contains 2,199 opinion video clips, each annotated with real-
valued sentiment intensity annotations in the range [-3, +3]. We evaluate the
model performances using various metrics following prior works: binary
accuracy (BA), F1 score, mean absolute error (MAE) of the score, and the
correlation of the prediction with humans (Corr).

CMU-MOSEI The CMU-MOSEI dataset Zadeh et al. (2018b) improves over MOSI with a
higher number of utterances, greater variety in samples, speakers, and topics.
The dataset contains 23,453 annotated video segments (utterances), from 5,000
videos, 1,000 distinct speakers and 250 different topics. The metrics are the
same as the CMU-MOSI.

Pretrained | IEMOCAP | CMU-MOSI | CMU-MOSEI
---|---|---|---
$\overline{\text{Acc}}\uparrow$ | $\overline{\text{F1}}\uparrow$ | BA$\uparrow$ | F1$\uparrow$ | BA$\uparrow$ | F1$\uparrow$
BERT-Base | 82.9 | 82.6 | 85.0/86.9 | 84.9/86.9 | 82.9/86.1 | 83.3/86.1
T5-Base | 82.6 | 82.4 | 87.2/89.3 | 87.3/89.3 | 84.5/86.4 | 84.7/86.3

Table 1: Text Encoder. T5-Base has better performance than BERT-Base
summarily. $\overline{\text{Acc}}$ and $\overline{\text{F1}}$ of IEMOCAP are
the average values of Acc and F1, respectively.

Scale | IEMOCAP | CMU-MOSI | CMU-MOSEI
---|---|---|---
$\overline{\text{Acc}}\uparrow$ | $\overline{\text{F1}}\uparrow$ | BA$\uparrow$ | F1$\uparrow$ | BA$\uparrow$ | F1$\uparrow$
1 | 82.2 | 82.1 | 86.4/88.8 | 86.2/88.8 | 83.3/85.9 | 83.3/85.9
1,2 | 82.5 | 82.2 | 86.3/88.9 | 86.3/88.8 | 83.2/86.2 | 83.6/86.2
1,3 | 81.9 | 81.8 | 86.4/88.9 | 86.4/88.9 | 83.5/86.3 | 83.6/86.2
1,2,3 | 82.0 | 81.9 | 86.7/89.0 | 86.6/89.0 | 84.3/86.4 | 84.0/86.2
1,2,10 | 82.6 | 82.4 | 86.7/89.0 | 86.8/89.1 | 84.0/86.2 | 84.2/86.3
1,2,3,10 | 82.1 | 81.9 | 87.2/89.3 | 87.3/89.3 | 84.5/86.4 | 84.7/86.3

Table 2: Multi-scale Fusion. Fusing different scale features improve the
performance.

Cluster | IEMOCAP | CMU-MOSI | CMU-MOSEI
---|---|---|---
$\overline{\text{Acc}}\uparrow$ | $\overline{\text{F1}}\uparrow$ | BA$\uparrow$ | F1$\uparrow$ | BA$\uparrow$ | F1$\uparrow$
10 | 82.1 | 82.1 | 86.3/88.9 | 86.2/88.8 | 83.8/86.4 | 84.1/86.3
15 | 82.3 | 82.1 | 86.4/88.7 | 86.4/88.7 | 83.6/86.5 | 83.9/86.4
20 | 82.5 | 82.2 | 86.3/88.6 | 86.2/88.6 | 84.1/86.5 | 84.4/86.5
10,15 | 82.6 | 82.4 | 87.2/89.3 | 87.3/89.3 |  84.5/86.3 |  84.7/86.2
15,20 | 82.3 | 82.1 | 85.4/87.9 | 85.3/87.9 |  84.5/86.4 |  84.7/86.3
10,15,20 | 82.0 | 82.0 | 85.8/88.2 | 85.8/88.2 | 83.3/86.5 | 83.8/86.5

Table 3: Cluster NO. in S3C Loss. Cluster number is an important impact to
affect the performance.

Following previous works Tsai et al. (2019); Rahman et al. (2020) and the CMU-
MultimodalSDK222https://github.com/A2Zadeh/CMU-MultimodalSDK, the video
feature is extracted via Facet333iMotions. Facial expression analysis, 2017.
and the acoustic feature is extracted using COVAREP Degottex et al. (2014).
The video feature mainly contains 35 facial action units, e.g., facial muscle
movement. The acoustic feature mainly includes Mel-frequency cepstral
coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features,
glottal source parameters, peak slope parameters, and maxima dispersion
quotients. The video feature dimension $\hat{d}_{v}$ is 35 for IEMOCAP and
CMU-MOSEI, and 47 for CMU-MOSI. The acoustic feature dimension $\hat{d}_{a}$
is 74 for all three benchmarks. We refer to this version of the feature as
_Facet &COVAREP_.

For the IEMOCAP, we also compare the video feature extracted by
OpenFace444https://github.com/TadasBaltrusaitis/OpenFace and the acoustic
feature extracted by librosa555https://github.com/librosa/librosa to
investigate the influence of the unimodality representation. Compared with
CMU-MOSI and CMU-MOSEI, each frame of IEMOCAP has two people in the scenario
simultaneously, making the judgment difficult. We partition two people
according to the layout of the frame and extract the feature separately. The
video feature dimension $\hat{d}_{v}$ is 709 and the acoustic feature
dimension $\hat{d}_{a}$ is 33. We refer to this version of the feature as
_OpenFace &Librosa_.

Modality | CMU-MOSI
---|---
BA$\uparrow$ | F1$\uparrow$
T | 86.4/88.6 | 86.4/88.6
V | 53.1/54.1 | 52.9/54.0
A | 54.7/55.0 | 54.1/54.4
T,V | 86.6/88.9 | 86.5/88.9
T,A | 87.0/89.3 | 87.0/89.3
V,A | 54.9/55.4 | 54.9/55.6
T,V,A | 87.2/89.3 | 87.3/89.3

Table 4: Multi-modality Fusion. Combining different unimodality can improve
model performance. T, V, and A mean text, video, and audio modality,
respectively. For the BA and F1 of CMU-MOSI and CMU-MOSEI, we report two
values: the left side of “/” is calculated following Zadeh et al. (2018c), and
the right side is following Tsai et al. (2019).

Features | IEMOCAP
---|---
$\overline{\text{Acc}}\uparrow$ | $\overline{\text{F1}}\uparrow$
_Facet &COVAREP_ | 82.6 | 82.4
_OpenFace &Librosa_ | 85.1 | 85.0

Table 5: Nonverbal Feature. Stronger nonverbal features can improve
performance.

Methods | Happy | Sad | Angry | Neutral | Average
---|---|---|---|---|---
Acc$\uparrow$ | F1$\uparrow$ | Acc$\uparrow$ | F1$\uparrow$ | Acc$\uparrow$ | F1$\uparrow$ | Acc$\uparrow$ | F1$\uparrow$ | $\overline{\text{Acc}}\uparrow$ | $\overline{\text{F1}}\uparrow$
CTC + EF-LSTM Tsai et al. (2019) | 76.2 | 75.7 | 70.2 | 70.5 | 72.7 | 67.1 | 58.1 | 57.4 | 69.3 | 67.7
LF-LSTM Tsai et al. (2019) | 72.5 | 71.8 | 72.9 | 70.4 | 68.6 | 67.9 | 59.6 | 56.2 | 68.4 | 66.6
CTC + RAVEN Wang et al. (2019) | 77.0 | 76.8 | 67.6 | 65.6 | 65.0 | 64.1 | 62.0 | 59.5 | 67.9 | 66.5
CTC + MCTN Pham et al. (2019) | 80.5 | 77.5 | 72.0 | 71.7 | 64.9 | 65.6 | 49.4 | 49.3 | 66.7 | 66.0
MulT Tsai et al. (2019) | 84.8 | 81.9 | 77.7 | 74.1 | 73.9 | 70.2 | 62.5 | 59.7 | 74.7 | 71.5
PMR Lv et al. (2021) | 86.4 | 83.3 | 78.5 | 75.3 | 75.0 | 71.3 | 63.7 | 60.9 | 75.9 | 72.7
MTAG Yang et al. (2021) | - | 86.0 | - | 79.9 | - | 76.7 | - | 64.1 | - | 76.7
ScaleVLAD | 86.7 | 85.9 | 84.8 | 84.6 | 86.8 | 86.9 | 72.1 | 72.1 | 82.6 | 82.4
\- w/o multi-scale | 86.6 | 85.7 | 84.1 | 84.2 | 86.7 | 86.9 | 71.5 | 71.3 | 82.2 | 82.0
\- w/o S3C loss | 85.1 | 84.9 | 84.3 | 84.4 | 88.5 | 88.3 | 69.4 | 68.5 | 81.8 | 81.5

Table 6: Sentiment prediction on IEMOCAP (unaligned) dataset.
$\overline{\text{Acc}}$ and $\overline{\text{F1}}$ are the average values. CTC
Graves et al. (2006) denotes connectionist temporal classification. The
results of CTC + EF-LSTM, LF-LSTM, CTC + RAVEN and CTC + MCTN are from Tsai et
al. (2019).

###  4.2 Experimental Details

We initial the text encoder with T5 Base Encoder Raffel et al. (2020) in this
paper due to its advantages after training with an extensive corpus. We also
conduct an ablation study to compare with BERT Base uncased version Devlin et
al. (2019). The rest of the parameters, e.g., Video Transformer, Audio
Transformer, and Fusion module, are initialized randomly. The fusion dimension
$d_{s}$ is set to 128. We train the model with the Adam optimizer Kingma and
Ba (2015) with a linear schedule. The warmup rate is set to 0.1 based on the
total epoch 50. The learning rate is set from {1e-3, 1e-4, 5e-5, 1e-5}. The
Video Transformer and Audio Transformer are set from {4, 6} layers with {128,
768} hidden size. The fusion Transformer in Eq. (9) is set with layer 2. The
multi-scale parameter $m$ and the number of shared semantic vectors $K$ in Eq.
(4) is set from {1, 2, 3, 10} and {8, 10}, respectively. The cluster $C$ is
set from {10, 15, 20}. Note these candidate choices are not exact and also can
not set with a grid search strategy, so we set them through empirical testing
on validation set. The start epoch $s_{s}3c$ for loss $\mathcal{L}_{s3c}$ is
set to 5, the same as the warmup epochs. All hyper-parameters are set
according to the performance from the validation set. The batch size is 64
across three datasets. All experiments are carried out on 4 NVIDIA Tesla V100
GPUs.

Methods | BA$\uparrow$ | F1$\uparrow$ | MAE$\downarrow$ | Corr$\uparrow$
---|---|---|---|---
MV-LSTM Rajagopalan et al. (2016) | 73.9/- | 74.0/- | 1.019 | 0.601
TFN Zadeh et al. (2017) | 73.9/- | 73.4/- | 1.040 | 0.633
MARN Zadeh et al. (2018c) | 77.1/- | 77.0/- | 0.968 | 0.625
MFN Zadeh et al. (2018a) | 77.4/- | 77.3/- | 0.965 | 0.632
RMFN Liang et al. (2018) | 78.4/- | 78.0/- | 0.922 | 0.681
RAVEN Wang et al. (2019) | 78.0/- | -/- | 0.915 | 0.691
MulT Tsai et al. (2019) | -/81.1 | -/81.0 | 0.889 | 0.686
ICCN Sun et al. (2020) | -/83.1 | -/83.0 | 0.862 | 0.714
PMR Lv et al. (2021) | -/82.4 | -/82.1 | - | -
FMT Zadeh et al. (2019) | 81.5/83.5 | 81.4/83.5 | 0.837 | 0.744
UniVL Luo et al. (2020) | 83.2/84.6 | 83.3/84.6 | 0.781 | 0.767
MISA (Hazarika et al. 2020) | 81.8/83.4 | 81.7/83.6 | 0.783 | 0.761
MAG-BERT Rahman et al. (2020) | 84.2/86.1 | 84.1/86.0 | 0.712 | 0.796
MAG-XLNet Rahman et al. (2020) | 85.7/87.9 | 85.6/87.9 | 0.675 | 0.821
Self-MM Yu et al. (2021) | 84.0/86.0 | 84.4/86.0 | 0.713 | 0.798
MTAG Yang et al. (2021) | -/82.3 | -/82.1 | 0.866 | 0.722
ScaleVLAD | 87.2/89.3 | 87.3/89.3 | 0.684 | 0.819
\- w/o multi-scale | 86.3/88.6 | 86.2/88.6 | 0.713 | 0.807
\- w/o S3C loss | 86.0/88.0 | 85.9/88.0 | 0.727 | 0.810
Human | 85.7/- | 87.5/- | 0.710 | 0.820

Table 7: Sentiment prediction on CMU-MOSI dataset. For BA and F1, we report
two values: the left side of “/” is calculated following Zadeh et al. (2018c),
and the right side is following Tsai et al. (2019).

###  4.3 Ablation Studies

We conduct comprehensive ablation studies on text encoder, key hyper-
parameters settings, and features in this section.

Text Encoder. In Table 1, we compare the BERT-Base with the T5-Base. The
T5-Base wins on CMU-MOSI and CMU-MOSEI. Besides, it has comparable results on
IEMOCAP. Thus, we use T5-Base as our text encoder in our work. We suppose that
a larger pretrained model, e.g., T5-Large, can achieve better performance but
needs more computational resources.

Multi-scale Fusion. In Table 2, we ablate the scale setting of the ScaleVLAD
module. The table lists a part of combinations, and we find {1,2,10} and
{1,2,3,10} can achieve better results than others. It proves that fusing
different granularities of representation can achieve better performance.

Cluster NO. in S3C Loss. In Table 3, we ablate the cluster setting of S3C
loss. The table lists a part of combinations, and we find {10,15} and {15,20}
can achieve better results than others. It indicates an appropriate choice of
the cluster will keep the feature clustering and thus improve the results.

Multi-modality Fusion. The results in Table 4 prove that multimodal fusion can
provide more comprehensive information and capture more emotional
characteristics than unimodality.

Nonverbal Feature. In Table 5, different nonverbal features are conducted on
IEMOCAP. It shows that more sophisticated features can obtain better results.
Further, we suppose that end-to-end training from raw signals instead of the
features extracted by off-the-shelf tools can improve more, like video
retrieval from Luo et al. (2021).

Methods | BA$\uparrow$ | F1$\uparrow$ | MAE$\downarrow$ | Corr$\uparrow$
---|---|---|---|---
MV-LSTM Rajagopalan et al. (2016) | 76.4/- | 76.4/- | - | -
MFN Zadeh et al. (2018a) | 76.0/- | 76.0/- | - | -
RAVEN Wang et al. (2019) | 79.1/- | 79.5/- | 0.614 | 0.662
PMR Lv et al. (2021) | -/83.1 | -/82.8 | - | -
MAG-BERT Rahman et al. (2020) | -/84.7 | -/84.5 | - | -
MAG-XLNet Rahman et al. (2020) | -/85.6 | -/85.7 | - | -
TFN Zadeh et al. (2017) | -/82.5 | -/82.1 | 0.593 | 0.700
MulT Tsai et al. (2019) | -/81.6 | -/81.6 | 0.591 | 0.694
ICCN Sun et al. (2020) | -/84.2 | -/84.2 | 0.565 | 0.713
MISA (Hazarika et al. 2020) | 83.6/85.5 | 83.8/85.3 | 0.555 | 0.756
Self-MM Yu et al. (2021) | 82.8/85.2 | 82.5/85.3 | 0.530 | 0.765
ScaleVLAD | 84.5/86.4 | 84.7/86.3 | 0.527 | 0.781
\- w/o multi-scale | 83.1/85.8 | 83.3/85.7 | 0.541 | 0.779
\- w/o S3C loss | 82.7/86.1 | 83.1/86.1 | 0.548 | 0.773

Table 8: Sentiment prediction on CMU-MOSEI dataset. For BA and F1, the values
on the both sides of “/” have the same calculations as Table 7.

###  4.4 Comparison to State-of-the-art

We compare ScaleVLAD with state-of-the-art methods on IEMOCAP, CMU-MOSI, and
CMU-MOSEI, and the results are shown in Table 6, Table 7, and Table 8,
respectively. In summary, 1) the proposed ScaleVLAD outperforms all baselines
in all datasets; 2) The ablation on multi-scale fusion and S3C loss proves
their effectiveness in all metrics and datasets. Our BERT-based results shown
in Table 1 can also have advantages over the BERT feature-based models, e.g.,
UniVL Luo et al. (2020), MAG-BERT Rahman et al. (2020), and Self-MM Yu et al.
(2021) in Table 7. The T5 based feature can improve the performance of IEMOCAP
by a significant margin shown in Table 6, which proves the strong capability
of the pretrained model after training with an extensive corpus in a self-
supervised manner.

###  4.5 Qualitative Analysis

(a)

(b)

Figure 3: Visualization of the ScaleVLAD w/o and w/ S3C loss in the training
set of MOSI using t-SNE projections van der Maaten and Hinton (2008).

Figure 3 displays the visualization of fusion features calculated by Eq. (11)
on training with S3C loss or not. For a clear observation, we regard the
continuous labels as six groups, each having width 1, e.g., $[-3.0,-2.0)$.
Figure 3(b) illustrates a tight clustering and clearer boundary, e.g., the
samples in blue color, when using S3C loss. It proves the effectiveness of the
S3C loss on representation learning. Figure 4 shows the similarity calculated
by Eq. (4). The alignment patterns of text, video, and audio with different
scales are different and are dynamically learned by the model. In this case,
the ‘really really loved’ with the yellow box can be regarded entirely to
align with the latent shared semantic vectors. Besides, the video and audio
with red boxes, which have bigger scales, i.e., 3 and 10, show consistently
shared vectors (NO. 2 and 6) with the text. Through the shared vectors, the
model can align and fuse the video and audio representation despite their
fuzzy semantic boundaries. We suppose the improvement of the ScaleVLAD is
benefits from such a multi-scale alignment.

Figure 4: Visualization of the similarity weights from ScaleVLAD module (Eq.
(4)). The tokens are processed by T5 tokenization. The y-axis means ten shared
semantic vectors. The x-axis denotes three blocks: text, video, and audio.
Each block has four scales: {1, 2, 3, 10}.

##  5 Conclusion

We proposed a multi-scale fusion method ScaleVLAD and a self-supervised
shifted clustering loss to address unaligned multimodal sentiment analysis in
this paper. The proposed method considers different granularities of
representation through aligning different modalities to trainable latent
semantic vectors. Thus, it can remit the fuzzy semantic boundary of
unimodality. The proposed self-supervised shifted clustering loss keeps the
differentiation of the fusion features via maintaining a momentum updated
cluster centers. The extensive experiments on three publicly available
datasets demonstrate the effectiveness of the proposed model.

## References

  * Akhtar et al. (2019) Akhtar, M. S.; Chauhan, D. S.; Ghosal, D.; Poria, S.; Ekbal, A.; and Bhattacharyya, P. 2019.  Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis.  In _NAACL-HLT_ , 370–379.
  * Arandjelovic et al. (2016) Arandjelovic, R.; Gronát, P.; Torii, A.; Pajdla, T.; and Sivic, J. 2016.  NetVLAD: CNN Architecture for Weakly Supervised Place Recognition.  In _CVPR_ , 5297–5307.
  * Arandjelovic and Zisserman (2013) Arandjelovic, R.; and Zisserman, A. 2013.  All About VLAD.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 1578–1585.
  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Busso et al. (2008) Busso, C.; Bulut, M.; Lee, C.-c.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S. 2008.  IEMOCAP: interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_.
  * Degottex et al. (2014) Degottex, G.; Kane, J.; Drugman, T.; Raitio, T.; and Scherer, S. 2014.  COVAREP—A collaborative voice analysis repository for speech technologies.  In _ICASSP_ , 960–964.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_ , 4171–4186.
  * Dumpala et al. (2019) Dumpala, S. H.; Sheikh, I.; Chakraborty, R.; and Kopparapu, S. K. 2019.  Audio-visual fusion for sentiment classification using cross-modal autoencoder.  In _NIPS_ , 1–4.
  * Graves et al. (2006) Graves, A.; Fernández, S.; Gomez, F. J.; and Schmidhuber, J. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _ICML_ , volume 148, 369–376.
  * Gu et al. (2018) Gu, Y.; Yang, K.; Fu, S.; Chen, S.; Li, X.; and Marsic, I. 2018.  Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment.  In _ACL_ , 2225–2235.
  * Han et al. (2021) Han, W.; Chen, H.; Gelbukh, A.; Zadeh, A.; Morency, L.-p.; and Poria, S. 2021.  Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis.  _ICMI_.
  * Hausler et al. (2021) Hausler, S.; Garg, S.; Xu, M.; Milford, M.; and Fischer, T. 2021.  Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition.  In _CVPR_ , 14141–14152.
  * Hazarika, Zimmermann, and Poria (2020) Hazarika, D.; Zimmermann, R.; and Poria, S. 2020.  MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis.  In _ACM MM_ , 1122–1131.
  * Hendrycks and Gimpel (2016) Hendrycks, D.; and Gimpel, K. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Jégou et al. (2010) Jégou, H.; Douze, M.; Schmid, C.; and Pérez, P. 2010.  Aggregating local descriptors into a compact image representation.  In _CVPR_ , 3304–3311.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  _ICLR_.
  * Liang et al. (2018) Liang, P. P.; Liu, Z.; Zadeh, A.; and Morency, L. 2018.  Multimodal Language Analysis with Recurrent Multistage Fusion.  In _EMNLP_ , 150–161.
  * Luo et al. (2020) Luo, H.; Ji, L.; Shi, B.; Huang, H.; Duan, N.; Li, T.; Li, J.; Bharti, T.; and Zhou, M. 2020.  UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.  _arXiv preprint arXiv:2002.06353_.
  * Luo et al. (2021) Luo, H.; Ji, L.; Zhong, M.; Chen, Y.; Lei, W.; Duan, N.; and Li, T. 2021.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.  _arXiv preprint arXiv:2104.08860_.
  * Lv et al. (2021) Lv, F.; Chen, X.; Huang, Y.; Duan, L.; and Lin, G. 2021.  Progressive Modality Reinforcement for Human Multimodal Emotion Recognition From Unaligned Multimodal Sequences.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2554–2562.
  * Mai, Hu, and Xing (2020) Mai, S.; Hu, H.; and Xing, S. 2020.  Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion.  In _AAAI_ , volume 34, 164–172.
  * Mai et al. (2020) Mai, S.; Xing, S.; He, J.; Zeng, Y.; and Hu, H. 2020.  Analyzing unaligned multimodal sequence via graph convolution and graph pooling fusion.  _arXiv preprint arXiv:2011.13572_.
  * Morency, Mihalcea, and Doshi (2011) Morency, L.; Mihalcea, R.; and Doshi, P. 2011.  Towards multimodal sentiment analysis: harvesting opinions from the web.  In _ICMI_ , 169–176.
  * Peng and Qi (2019) Peng, Y.; and Qi, J. 2019.  CM-GANs: Cross-modal generative adversarial networks for common representation learning.  _TOMM_ , 15(1): 1–24.
  * Pham et al. (2019) Pham, H.; Liang, P. P.; Manzini, T.; Morency, L.; and Póczos, B. 2019.  Found in Translation: Learning Robust Joint Representations by Cyclic Translations between Modalities.  In _AAAI_ , 6892–6899.
  * Poria et al. (2016) Poria, S.; Chaturvedi, I.; Cambria, E.; and Hussain, A. 2016.  Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis.  In _ICDM_ , 439–448.
  * Poria et al. (2020) Poria, S.; Hazarika, D.; Majumder, N.; and Mihalcea, R. 2020.  Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research.  _IEEE Transactions on Affective Computing_.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140): 1–67.
  * Rahman et al. (2020) Rahman, W.; Hasan, M. K.; Lee, S.; Zadeh, A. B.; Mao, C.; Morency, L.; and Hoque, M. E. 2020.  Integrating Multimodal Information in Large Pretrained Transformers.  In _ACL_ , 2359–2369.
  * Rajagopalan et al. (2016) Rajagopalan, S. S.; Morency, L.; Baltrusaitis, T.; and Goecke, R. 2016.  Extending Long Short-Term Memory for Multi-View Structured Learning.  In _ECCV_ , 338–353.
  * Siriwardhana et al. (2020) Siriwardhana, S.; Reis, A.; Weerasekera, R.; and Nanayakkara, S. 2020.  Jointly Fine-Tuning ”BERT-Like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition.  In _Interspeech_ , 3755–3759.
  * Sun et al. (2020) Sun, Z.; Sarma, P.; Sethares, W.; and Liang, Y. 2020.  Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis.  In _AAAI_ , volume 34, 8992–8999.
  * Tsai et al. (2019) Tsai, Y. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.; and Salakhutdinov, R. 2019.  Multimodal Transformer for Unaligned Multimodal Language Sequences.  In _ACL_ , 6558–6569.
  * Tsai et al. (2020) Tsai, Y. H.; Ma, M.; Yang, M.; Salakhutdinov, R.; and Morency, L. 2020.  Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis.  In _EMNLP_ , 1823–1833.
  * van der Maaten and Hinton (2008) van der Maaten, L.; and Hinton, G. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ , 9(86): 2579–2605.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_ , 5998–6008.
  * Verma et al. (2019) Verma, S.; Wang, C.; Zhu, L.; and Liu, W. 2019.  DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis.  In _IJCAI_ , 3627–3634.
  * Verma et al. (2020) Verma, S.; Wang, J.; Ge, Z.; Shen, R.; Jin, F.; Wang, Y.; Chen, F.; and Liu, W. 2020\.  Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis.  In _ICDM_ , 561–570.
  * Wang, Zhu, and Yang (2021) Wang, X.; Zhu, L.; and Yang, Y. 2021.  T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval.  In _CVPR_ , 5079–5088.
  * Wang et al. (2019) Wang, Y.; Shen, Y.; Liu, Z.; Liang, P. P.; Zadeh, A.; and Morency, L. 2019.  Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors.  In _AAAI_ , 7216–7223.
  * Wang, Wan, and Wan (2020) Wang, Z.; Wan, Z.; and Wan, X. 2020.  TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis.  In _WWW_ , 2514–2520.
  * Yang et al. (2021) Yang, J.; Wang, Y.; Yi, R.; Zhu, Y.; Rehman, A.; Zadeh, A.; Poria, S.; and Morency, L. 2021.  MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences.  In _NAACL-HLT_ , 1009–1021.
  * Yang, Xu, and Gao (2020) Yang, K.; Xu, H.; and Gao, K. 2020.  CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , 521–528.
  * Yu et al. (2021) Yu, W.; Xu, H.; Yuan, Z.; and Wu, J. 2021.  Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis.  In _AAAI_ , 10790–10797.
  * Zadeh et al. (2017) Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L. 2017.  Tensor Fusion Network for Multimodal Sentiment Analysis.  In _EMNLP_ , 1103–1114.
  * Zadeh et al. (2018a) Zadeh, A.; Liang, P. P.; Mazumder, N.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018a.  Memory Fusion Network for Multi-view Sequential Learning.  _AAAI_.
  * Zadeh et al. (2018b) Zadeh, A.; Liang, P. P.; Poria, S.; Cambria, E.; and Morency, L. 2018b.  Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph.  In _ACL_ , 2236–2246.
  * Zadeh et al. (2018c) Zadeh, A.; Liang, P. P.; Poria, S.; Vij, P.; Cambria, E.; and Morency, L.-P. 2018c.  Multi-attention recurrent network for human communication comprehension.  In _AAAI_.
  * Zadeh et al. (2019) Zadeh, A.; Mao, C.; Shi, K.; Zhang, Y.; Liang, P. P.; Poria, S.; and Morency, L. 2019.  Factorized Multimodal Transformer for Multimodal Sequential Learning.  _arXiv preprint arXiv:1911.09826_.
  * Zadeh et al. (2016) Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6): 82–88.
  * Zhang et al. (2020) Zhang, C.; Yang, Z.; He, X.; and Deng, L. 2020.  Multimodal Intelligence: Representation Learning, Information Fusion, and Applications.  _IEEE J. Sel. Top. Signal Process._ , 14(3): 478–493.


# Distilling Missing Modality Knowledge from Ultrasound for Endometriosis
Diagnosis with Magnetic Resonance Images

###### Abstract

Endometriosis is a common chronic gynecological disorder that has many
characteristics, including the pouch of Douglas (POD) obliteration, which can
be diagnosed using Transvaginal gynecological ultrasound (TVUS) scans and
magnetic resonance imaging (MRI). TVUS and MRI are complementary non-invasive
endometriosis diagnosis imaging techniques, but patients are usually not
scanned using both modalities and, it is generally more challenging to detect
POD obliteration from MRI than TVUS. To mitigate this classification
imbalance, we propose in this paper a knowledge distillation training
algorithm to improve the POD obliteration detection from MRI by leveraging the
detection results from unpaired TVUS data. More specifically, our algorithm
pre-trains a teacher model to detect POD obliteration from TVUS data, and it
also pre-trains a student model with 3D masked auto-encoder using a large
amount of unlabelled pelvic 3D MRI volumes. Next, we distill the knowledge
from the teacher TVUS POD obliteration detector to train the student MRI model
by minimizing a regression loss that approximates the output of the student to
the teacher using unpaired TVUS and MRI data. Experimental results on our
endometriosis dataset containing TVUS and MRI data demonstrate the
effectiveness of our method to improve the POD detection accuracy from MRI.

Index Terms—  Endometriosis, Knowledge Distillation, Masked Auto-Encoder,
Pouch of Douglas Obliteration

##  1 Introduction

Endometriosis is a gynecological disorder associated with the growth of
endometrial glands and stroma outside the uterine cavity [1, 2]. The clinical
manifestations of endometriosis include infertility and endometriosis-related
pain [3]. As a common chronic gynecological disease, it affects approximately
1.5 million women worldwide [4]. There is currently no known way to prevent or
cure endometriosis, but early diagnosis, intervention and management may slow
or stop the natural disease progression. However, the diagnosis of
endometriosis can take about 7 years on average after the appearance of
initial symptoms [5]. Laparoscopy used to be the diagnostic gold standard [6],
but with the improvement in the quality and availability of imaging modalities
for endometriosis diagnosis, there has been evidence suggesting that accurate
endometriosis diagnosis can be achieved with the analysis of TVUS sequences
and MRI volumes [7, 8].

Many of the symptomatic endometriosis cases can be associated with the pouch
of Douglas (POD) obliteration, which can be diagnosed from complementary TVUS
and MRI data, as shown in Fig. 1. However, in clinical practice, it is
difficult to access clinicians who can diagnose endometriosis with one of
these modalities, not to mention those who are proficient in both modalities.
For TVUS, POD obliteration can be accurately detected manually [9] and
automatically [10] via the ultrasound ‘sliding sign’ [11], which is classified
as positive (i.e. normal) or negative (i.e. abnormal), where a negative
sliding sign is considered when the anterior rectum or bowel glides cannot
freely slide over the posterior upper uterine fundus. For MRI, POD can be
classified as obliterated or normal, where the POD obliteration can be
characterized by endometrial plaques and dense adhesions between uterosacral
ligaments, uterine serosa, ovaries, rectum and vagina on T2-weighted and
T1-weighted images [12]. However, differently from TVUS, the manual POD
obliteration detection from MRI can only reach 61.4-71.9% accuracy [13]. There
has been some effort to propose methods that can automatically diagnose deep
pelvic endometriosis classification from MRI111Deep infiltrating endometriosis
(DIE) can lead to a partial or complete obliteration of the pouch of Douglas
(POD) [14]., but we are not aware of methods that can detect POD obliteration
from MRI.

Fig. 1: Examples of POD obliteration on MRI and sliding sign on TVUS. (a) and
(b) represent negative and positive POD obliteration sign on sagittal plane
MRI, respectively. (c) and (d) represent positive and negative sliding sign on
TVUS, respectively.

Leveraging the TVUS POD obliteration detection to improve the automated
detection accuracy from MRI using an unpaired training set containing scans
from both modalities is the main goal of this paper. We achieve this goal by
proposing a new knowledge distillation algorithm based on two stages: 1) pre-
training a teacher model to detect POD obliteration from TVUS data, and pre-
training a student model with 3D masked auto-encoder (MAE) using a large
amount of unlabelled pelvic 3D MRI volumes; and 2) knowledge distillation from
the teacher TVUS detector to train the student MRI model by minimizing a
regression loss that approximates the output of the student to the teacher
using unpaired TVUS and MRI data. The testing is realised using only MRI data.
The main innovations of this work are:

  * •

To the best of our knowledge, this is the first POD obliteration detection
method that distills knowledge from TVUS to MRI using unpaired data, with the
objective of improving the accuracy of diagnosing endometriosis from MRI;

  * •

It is also the first machine learning method that can automatically detect POD
obliteration from MRI data with the goal of diagnosing endometriosis.

Experimental results on a private endometriosis training set containing
unpaired TVUS and MRI data show the effectiveness of our method to increase
the POD detection accuracy from testing MRI volumes.

##  2 Related Work

The automated detection of endometriosis from medical imaging has received
some attention lately. Using ultrasound images, Guerriero et al. [15] compared
the ability of six machine learning algorithms and neural networks for the
diagnosis of endometriosis in the rectosigmoid, where the neural network
achieved the highest classification accuracy of 0.73. Maicas et al. [10]
constructed a deep learning model based on a temporal residual network to
classify POD obliteration from TVUS videos, achieving an AUC of 96.5% and an
accuracy of 88.8%. The methods above use TVUS data alone, but recently, Yang
et al. [16] built a bi-model method with one model for MRI and another for
TVUS, but they do not explore the relationship between MRI and TVUS, like we
propose in this paper. Furthermore, as mentioned above, it is unlikely that
patients will have access to both modalities in clinical practice, which
justifies the need for single-modality approaches that have highly accurate
endometriosis detection.

Knowledge distillation is a general framework to extract the knowledge learned
from a teacher model to a student model by soft-label supervision. The
original purpose of knowledge distillation is to compress deep learning
models, so they can run on resource-constrained devices [17], but in this
paper we focus on the transfer of knowledge from a teacher network trained on
source modalities to a student network that is trained on different target
modalities [18]. In medical image analysis, data from different modalities can
provide rich and complementary information about diseases, so multimodal
knowledge distillation is suitable for scenarios where data or labels for some
modalities are missing during training or testing. Inspired by knowledge
distillation, Dou et al. [19] tackled the task of unpaired multi-modal image
segmentation. Guan et al. [20] leverage additional supervision distilled from
clinical data to improve MRI-based Alzheimer’s disease prediction models.
Cheng et al. [21] utilise both a pixel-level and an image-level distillation
scheme to transfer knowledge from a multimodal-MRI teacher network to a
unimodal segmentation student network. However, most unpaired multi-modal
studies above focus on MRI and CT scans, which is arguably easier than
focusing on MRI and TVUS, which is the case considered in this paper.

##  3 Method

An overview of our proposed TVUS to MRI knowledge distillation model is shown
in Figure 2, which consists of two models: a teacher model pre-trained with a
TVUS dataset and a student model pre-trained on an MRI dataset, and then
trained on an unpaired dataset of TVUS and MRI data by distilling the
knowledge in the representation learned by the teacher model to the student
model.

Formally, let
$\mathcal{D}_{M}=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i=1}^{N}$ denote the
endometriosis MRI dataset, with $N$ T2 SPC images
$\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{H\times W\times D}$ and
corresponding POD obliteration one-hot label $\mathbf{y}\in\\{0,1\\}^{2}$,
where $H$, $W$ and $D$ are height, width and depth of the MRI, respectively.
For TVUS dataset, let
$\mathcal{D}_{U}^{s}=\\{(\mathbf{x}_{i}^{s},\mathbf{y}_{i}^{s})\\}_{i=1}^{M}$
be the video clips dataset, where
$\mathbf{x}^{s}\in\mathcal{X}^{s}\subset\mathbb{R}^{H\times W\times T}$ ($H$,
$W$ and $T$ are height, width and number of frames), and
$\mathbf{y}^{s}\in\\{0,1\\}^{2}$ denotes the POD obliteration one-hot label.
For the self-supervised pre-training of the MRI POD obliteration detector, we
have $\mathcal{D}_{P}=\\{\mathbf{x}_{i}^{p}\\}_{i=1}^{K}$, which contains $K$
unlabeled MRI volumes, where the number of unlabeled images is much larger
than the labeled images (i.e., $K>>N$ and $K>>M$). The teacher model
$f_{\theta_{U}}:\mathcal{X}^{s}\to\Delta$ (with $\Delta\subset[0,1]^{2}$ being
the probability simplex) is trained with dataset $\mathcal{D}^{s}_{U}$, the
student model $f_{\theta_{M}}:\mathcal{X}\to\Delta$ is pre-trained with
dataset $\mathcal{D}_{P}$ and fine-tuned using $\mathcal{D}_{M}$. The final
knowledge distillation model is initialised by the pre-trained student model
$f_{\theta_{M}}(.)$, which is then trained from $\mathcal{D}_{M}$ and
$\mathcal{D}^{s}_{U}$. The testing to classify POD obliteration uses the
trained $f_{\theta_{M}}(.)$ on the MRI testing images from $\mathcal{D}_{M}$.

Fig. 2: Proposed POD obliteration detector trained by distilling knowledge to
MRI from unpaired TVUS. (a) MRI pre-training with 3D masked auto-encoder, (b)
TVUS pre-training with ResNet(2+1)D, (c) MRI Knowledge Distillation from the
frozen teacher model pretrained on TVUS.

###  3.1 Pre-training

For pre-training the MRI encoder, we explore the self-supervised masked auto-
encoder (MAE) method [22] using the large dataset of un-annotated T2 MRI
images $\mathcal{D}_{P}$. 3D-MAE has an asymmetric encoder-decoder
architecture based on 3D Vision Transformer (3D ViT) [23], as shown in Figure
2. During pre-training, each volume is cropped into $8\times 8\times 8$
patches, then 50% of the patches are randomly masked. The encoder, denoted by
$f_{\theta_{E}}:\mathcal{X}\to\mathcal{E}$, takes visible patches embedded by
a linear projection (of dimension 768) with additional positional embeddings,
and processes via the 3D ViT with 12 Transformer blocks whose output is then
fed into the decoder, denoted by $f_{\theta_{D}}:\mathcal{D}\to\mathcal{X}$,
together with the masked volume tokens, to reconstruct the original volume at
the voxel level. After pre-training, the decoder is discarded and the pre-
trained encoder is applied to extract MRI features for the downstream
classification task. During MRI pre-training, we minimize the mean squared
error (MSE) on the masked patches between the reconstructed and original
volumes in the voxel space, defined as follows:

$\displaystyle\ell_{MAE}(\mathcal{D}_{P};\theta_{E},\theta_{D})=\frac{1}{L\times
K}\sum_{i=1}^{K}\sum_{l=1}^{L}\|\mathbf{x}_{i}(l)-\hat{\mathbf{x}}_{i}(l)\|_{2}^{2},$
(1)

where $K$ is the size of the unlabelled MRI dataset, $L$ indicates the number
of masked patches, $\mathbf{x}_{i}(l)$ and $\hat{\mathbf{x}}_{i}(l)$ represent
the voxel values of the $l^{th}$ masked patch in the original and
reconstructed volumes, respectively, with the reconstructed volume obtained
from $\hat{\mathbf{x}}_{i}=f_{\theta_{D}}(f_{\theta_{E}}(\mathbf{x}_{i}))$.

After this self-supervised pre-training, we take the encoder
$f_{\theta_{E}}(.)$, add a linear layer to change the size from 768 (output
size of the MRI encoder) to 512 (output size of the TVUS encoder), and add a
final classification layer with a 2-dimensional output activated by softmax to
form the student network, denoted by $f_{\theta_{M}}(.)$. We fine-tune all
transformer blocks in the encoder and fully connected layer to classify POD
obliteration by minimizing the cross-entropy (CE) loss:

$\displaystyle\ell_{PTM}(\mathcal{D}_{M};\theta_{M})=-\sum_{(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{D}_{M}}\ell_{CE}(\mathbf{y}_{i},f_{\theta_{M}}(\mathbf{x}_{i})).$
(2)

For the TVUS pre-training, we adopted the ResNet (2+1)D model proposed in
[10]. This model contains 18 modules of R(2+1) convolutional layers, with each
convolution being followed by batch normalization. During this TVUS pre-
training, we also minimize the CE loss, as follows:

$\displaystyle\ell_{PTU}(\mathcal{D}^{s}_{U};\theta_{U})=-\sum_{(\mathbf{x}^{s}_{i},\mathbf{y}^{s}_{i})\in\mathcal{D}_{M}}\ell_{CE}(\mathbf{y}^{s}_{i},f_{\theta_{U}}(\mathbf{x}^{s}_{i})).$
(3)

###  3.2 Knowledge Distillation

In the knowledge distillation (KD) stage, we consider the pre-trained TVUS
model $f_{\theta_{U}}(.)$ as the teacher model, and the pre-trained MRI model
$f_{\theta_{M}}(.)$ as the student model. Given that the pre-trained TVUS
model tends to produce superior classification accuracy than the pre-trained
MRI model, we aim to fine-tune the MRI model to match the predictions produced
by the TVUS model. The goal is to use this knowledge distillation procedure to
improve the classification accuracy of the MRI model, which uses only the MRI
volume during testing. However, recall that we do not have matched TVUS and
MRI data for this knowledge distillation procedure, so we match the data based
only on their classification labels, i.e., an MRI sample with positive POD
obliteration is matched with a random TVUS sample with positive POD
obliteration, and similarly for the negative POD obliteration. Then, the KD
training minimises the following loss

$\displaystyle\ell_{KD}(\mathcal{D}_{M},\mathcal{D}_{U}^{s};\theta_{M})=-\sum_{\begin{subarray}{c}(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{D}_{M}\\\
(\mathbf{x}^{s}_{j},\mathbf{y}^{s}_{j})\in\mathcal{D}_{U}^{s}\\\
\mathbf{y}_{i}=\mathbf{y}^{s}_{j}\end{subarray}}\|f_{\theta_{U}}(\mathbf{x}^{s}_{j})-f_{\theta_{M}}(\mathbf{x}_{i}))\|_{1}.$
(4)

The loss in (4) is added to the $\ell_{PTM}(.)$ loss from (2) to form the
final loss that encourages the model to pull the TVUS and the MRI outputs
closer to distill the TVUS classification information to the student network,
as follows

$\displaystyle\ell(\mathcal{D}_{M},\mathcal{D}_{U}^{s};\theta_{M})=$
$\displaystyle\alpha^{epoch}\times\ell_{KD}(\mathcal{D}_{M},\mathcal{D}_{U}^{s};\theta_{M})+$
(5)
$\displaystyle(1-\alpha^{epoch})\times\ell_{PTM}(\mathcal{D}_{M};\theta_{M}),$

which is used to estimate the optimal value of $\theta_{M}$, where
$\alpha^{epoch}$ is a parameter to dynamically balance the contributions of
the two loss terms during training.

##  4 Experiments

###  4.1 Dataset

Our private dataset contains: an MRI endometriosis dataset, a TVUS
endometriosis dataset, and a female pelvic MRI dataset for self-supervised
pre-training.

The MRI endometriosis dataset contains 88 T2 SPACE MRI scans from women aged
18-45, including 19 POD obliteration cases. These examinations were performed
in several clinical sites in Australia. The scans contain the whole pelvic
area, but we focus on the region around the uterus that can display the POD
obliteration. We performed 3D contrast limited adaptive histogram equalization
(CLAHE) to improve the local contrast and enhance the definitions of edges in
each region of a sequence. For the experiments, we use stratified random
sampling to split the dataset into 5 folds, each with 17-18 subjects, with 4
folds used for training and 1 for testing, which are rotated to produce a
cross-validated result.

The TVUS endometriosis dataset has 749 video clips of the ’sliding sign’,
including 103 negative (POD obliterated) cases. We follow the data preparation
and pre-processing protocol as well as model parameter settings proposed in
[10] to pre-train the TVUS model. For the knowledge distillation phase, we
divide the dataset into negative and positive groups, then use stratified
random sampling to split each group into 5 folds and use 4 folds in each group
as the training set.

The female pelvic MRI dataset contains 8,984 volumes. In the context of
endometriosis research, we accept all scans when the patient is aged between
18 to 45, the physical examination site is pelvis, and the sequence
description contains ’T2’. It is worth noting that most of these volumes were
scanned in different settings, so they may contain signs of other diseases and
the scanned area may or may not overlap with the diagnostic area of
endometriosis. We re-sampled all scans by SimpleITK with an output spacing of
$1\times 1\times 1mm$, and filtered out volumes if the number of slices in any
dimension is less than 65.

###  4.2 Training and Evaluation

In the pre-training phase, the 3D MAE network is trained for 200 epochs on the
female pelvic MRI dataset using a batch size of 3 with AdamW optimizer and a
learning rate of 1e-3. Then we fine-tuned the pre-trained checkpoint saved
from epoch 195 for 25 epochs on the endometriosis MRI dataset with a “5-fold
cross validation” strategy. The ResNet(2+1)D network was pre-trained on the
Kinetics-400 dataset then fine-tuned for 30 epochs on the TVUS endometriosis
dataset using a batch size of 30 with Adam optimizer and a learning rate of
1e-5. For knowledge distillation, we loaded the weights of the $10^{th}$
checkpoint fine-tuned from each fold as the MRI feature extractor of the MRI
model and the pre-trained TVUS feature extractor. The knowledge distillation
network is trained for 10 epochs on the endometriosis MRI dataset using a
batch size of 7 with AdamW optimizer and a learning rate of 1e-3 with the same
“5-fold cross validation” strategy. We set the hyper-parameter $\alpha=0.85$
based on the cross-validation results. We evaluate our method by computing
Area Under the Receiver Operating Characteristic Curve (ROC AUC) for five
folds and calculate the mean and standard deviation of them.

###  4.3 Results

The classification results with our proposed method is shown in Table 1. The
ResNet(2+1)D model shows an outstanding classification AUC of 96.9% from TVUS
data. The small amount of training samples limited the generalisation of 3D
ViT to classify POD obliteration from MRI volumes with an AUC of 65.0%, but
the MAE pre-training (PT) partially mitigates the issue, improving the AUC to
87.2%. With knowledge distillation (KD), we observe that training a 3D ViT
from scratch on such a small dataset is still challenging, with an AUC of
66.7%. Also, the KD performance of 3D ViT with MAE pre-training (PT) reaches
AUC=77.2%), which is worse than without KD, with AUC=87.2%, which may be due
to the excessive domain shift between the pre-training dataset and TVUS
dataset. By fine-tuning (FT) the model from MAE pre-training, the model
improves accuracy from AUC=87.2% (without KD and FT) to AUC=90.6% (with KD and
FT).

Table 1:  POD obliteration classification results.

Method | Training | Testing | AUC
---|---|---|---
| Modality | Modality | mean$\pm$stddev
ResNet(2+1)D | TVUS | TVUS | 0.969$\pm$0.012
3D ViT | MRI | MRI | 0.650$\pm$0.102
| 3D ViT: MAE PT
---
MRI | MRI | 0.872$\pm$0.094
| 3D ViT: KD
---
MRI,TVUS | MRI | 0.667$\pm$0.107
| 3D ViT: MAE PT + KD
---
MRI,TVUS | MRI | 0.772$\pm$0.087
| 3D ViT: MAE PT + KD + FT
---
MRI,TVUS | MRI | 0.906$\pm$0.099

##  5 Conclusion

In this paper, we proposed a two-stage algorithm to distill the knowledge from
a TVUS to an MRI classifier, thereby improving the POD obliteration
classification accuracy of the MRI classifier. Through the MAE pre-training,
knowledge distillation and fine-tuning, we are able to significantly reduce
the distance between the two domains and accomplish a promising knowledge
distillation from TVUS to MRI. The efficacy and superiority of our proposed
approach are demonstrated by experimental results on our endometriosis
datasets. In the future, we will introduce a missing modality deep learning
approach and expand our proposed method to perform weakly-supervised lesion
segmentation, thereby improving the interpretability of the model, so it can
be widely applied in future clinical trials.

##  6 Compliance with ethical standards

This study was performed in line with the principles of the Declaration of
Helsinki. Approval was granted by Human Research Ethics Committee (HREC) of
University of Adelaide(Date 01-03-2020/No. H-2020-051) and the Southern
Adelaide Clinical Human Research Ethics Committee (SAC HREC) (Date
01-11-2021/No. 111.20).

##  7 Acknowledgments

This work received funding from the Australian Government through the Medical
Research Futures Fund: Primary Health Care Research Data Infrastructure Grant
2020 and from Endometriosis Australia.

## References

  * [1] Antonio Simone Lagana et al.,  “Evaluation of m1 and m2 macrophages in ovarian endometriomas from women affected by endometriosis at different stages of the disease,”  Gynecological Endocrinology, vol. 36, no. 5, pp. 441–444, 2020\.
  * [2] KM Moss, J Doust, H Homer, IJ Rowlands, R Hockey, and GD Mishra,  “Delayed diagnosis of endometriosis disadvantages women in art: a retrospective population linked data study,”  Human Reproduction, vol. 36, no. 12, pp. 3074–3082, 2021.
  * [3] Paolo Vercellini et al.,  “Endometriosis: pathogenesis and treatment,”  Nature Reviews Endocrinology, vol. 10, no. 5, pp. 261–275, 2014\.
  * [4] Krina T. Zondervan et al.,  “Endometriosis,”  New England Journal of Medicine, vol. 382, no. 13, pp. 1244–1256, 2020,  PMID: 32212520.
  * [5] Australian Institute of Health and Welfare,  “Endometriosis in australia: prevalence and hospitalisations,” 2019,  [Online; accessed 18-02-2022].
  * [6] Christian M Becker et al.,  “Eshre guideline: endometriosis,”  Human reproduction open, vol. 2022, no. 2, pp. hoac009, 2022.
  * [7] Alison Deslandes et al.,  “Current status of transvaginal ultrasound accuracy in the diagnosis of deep infiltrating endometriosis before surgery: a systematic review of the literature,”  Journal of Ultrasound in Medicine, vol. 39, no. 8, pp. 1477–1490, 2020.
  * [8] T Indrielle-Kelly et al.,  “Diagnostic accuracy of ultrasound and mri in the mapping of deep pelvic endometriosis using the international deep endometriosis analysis (idea) consensus,”  BioMed research international, vol. 2020, 2020.
  * [9] Le Chi Chiu et al.,  “Predicting pouch of douglas obliteration using ultrasound and laparoscopic video sets: an interobserver and diagnostic accuracy study,”  Journal of Ultrasound in Medicine, vol. 38, no. 12, pp. 3155–3161, 2019.
  * [10] Gabriel Maicas et al.,  “Deep learning to diagnose pouch of douglas obliteration with ultrasound sliding sign,”  Reproduction and Fertility, vol. 2, no. 4, pp. 236–243, 2021.
  * [11] Stefano Guerriero et al.,  “Systematic approach to sonographic evaluation of the pelvis in women with suspected endometriosis, including terms, definitions and measurements: a consensus opinion from the international deep endometriosis analysis (idea) group,”  Ultrasound in Obstetrics & Gynecology, vol. 48, no. 3, pp. 318–332, 2016.
  * [12] Anitha L Thalluri et al.,  “Mri findings in deep infiltrating endometriosis: a pictorial essay,”  Journal of Medical Imaging and Radiation Oncology, vol. 61, no. 6, pp. 767–773, 2017.
  * [13] Milliam L Kataoka et al.,  “Posterior cul-de-sac obliteration associated with endometriosis: Mr imaging evaluation,”  Radiology, vol. 234, no. 3, pp. 815–823, 2005.
  * [14] Kristina Arion et al.,  “Prediction of pouch of douglas obliteration: point-of-care ultrasound versus pelvic examination,”  Journal of Minimally Invasive Gynecology, vol. 26, no. 5, pp. 928–934, 2019.
  * [15] Stefano Guerriero et al.,  “Artificial intelligence (ai) in the detection of rectosigmoid deep endometriosis,”  European Journal of Obstetrics & Gynecology and Reproductive Biology, vol. 261, pp. 29–33, 2021.
  * [16] Minmin Yang et al.,  “Diagnostic efficacy of ultrasound combined with magnetic resonance imaging in diagnosis of deep pelvic endometriosis under deep learning,”  The Journal of Supercomputing, vol. 77, no. 7, pp. 7598–7619, 2021\.
  * [17] Geoffrey Hinton et al.,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015.
  * [18] Nikolaos Passalis et al.,  “Learning deep representations with probabilistic knowledge transfer,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 268–284.
  * [19] Qi Dou et al.,  “Unpaired multi-modal segmentation via knowledge distillation,”  IEEE transactions on medical imaging, vol. 39, no. 7, pp. 2415–2425, 2020.
  * [20] Hao Guan et al.,  “Mri-based alzheimer’s disease prediction via distilling the knowledge in multi-modal data,”  NeuroImage, vol. 244, pp. 118586, 2021.
  * [21] Cheng Chen et al.,  “Learning with privileged multimodal knowledge for unimodal segmentation,”  IEEE Transactions on Medical Imaging, vol. 41, no. 3, pp. 621–632, 2021.
  * [22] Kaiming He et al.,  “Masked autoencoders are scalable vision learners,”  arXiv preprint arXiv:2111.06377, 2021.
  * [23] Alexey Dosovitskiy et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.