# Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach
for Multimodal Sentiment Analysis with Missing Modality

Haozhe Chi Minghua Yang Junhao Zhu Guanhong Wang Gaoang Wang
Zhejiang University-University of Illinois at Urbana-Champaign Institute,
Zhejiang University, China
{haozhe.20, minghua.20, junhao.20<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Multimodal sentiment analysis (MSA) is an important way of observing mental
activities with the help of data captured from multiple modalities. However,
due to the recording or transmission error, some modalities may include
incomplete data. Most existing works that address missing modalities usually
assume a particular modality is completely missing and seldom consider a
mixture of missing across multiple modalities. In this paper, we propose a
simple yet effective meta-sampling approach for multimodal sentiment analysis
with missing modalities, namely Missing Modality-based Meta Sampling (M3S). To
be specific, M3S formulates a missing modality sampling strategy into the
modal agnostic meta-learning (MAML) framework. M3S can be treated as an
efficient add-on training component on existing models and significantly
improve their performances on multimodal data with a mixture of missing
modalities. We conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets, and
superior performance is achieved compared with recent state-of-the-art
methods.

##  1 Introduction

Multimodal sentiment analysis (MSA) aims to estimate human mental activities
by multimodal data, such as a combination of audio, video, and text. Though
much progress has been made recently, there still exist challenges, including
missing modality problem. In reality, missing modality is a common problem due
to the errors in data collection, storage, and transmission. To address the
issue with missing modality in MSA, many approaches have been proposed Ma et
al. (2021c); Zhao et al. (2021); Ma et al. (2021b); Parthasarathy and Sundaram
(2020); Ma et al. (2021a); Tran et al. (2017).

In general, methods that address the missing modality issue usually only
consider the situation where a certain input modality is severely damaged. The
strategies of these proposed methods can be divided into three categories: 1)
Designing new architectures with a reconstruction network to recover missing
modality with the information from other modalities Ma et al. (2021c); Ding et
al. (2014); 2) Formulating innovative and efficient loss functions to tackle
missing modality Ma et al. (2021a, 2022); 3) Improving the encoding and
embedding strategies from existing models Tran et al. (2017); Cai et al.
(2018).

Figure 1: M3S helps MMIN model achieve superior performance.

In the MSA tasks, most of the proposed methods focus on the situation where
certain modalities are completely missing and the other modalities are
complete. However, due to the transmission or collection errors, each modality
may contain partial information based on a certain missing rate, while
existing methods seldom consider this type of scenario and they are not
suitable to be applied directly in this situation. Besides, our experiments
also verify the inefficacy of existing methods in such a challenging
situation, which is demonstrated in Section 5.

To address the aforementioned problem, in this paper, we propose a simple yet
effective solution to the Missing Modality problem with Meta Sampling in the
MSA task, namely M3S. To be specific, M3S combines the augmented missing
modality transform in sampling, following the model-agnostic meta-learning
(MAML) framework Finn et al. (2017). M3S maintains the advantage of meta-
learning and makes models easily adapt to data with different missing rates.
M3S can be treated as an efficient add-on training component on existing
models and significantly improve their performances on multimodal data with a
mixture of missing modalities. We conduct experiments on IEMOCAP Busso et al.
(2008), SIMS Yu et al. (2020) and CMU-MOSI Zadeh et al. (2016) datasets and
superior performance is achieved compared with recent state-of-the-art (SOTA)
methods. A simple example is shown in Figure 1, demonstrating the
effectiveness of our proposed M3S compared with other methods. More details
are provided in the experiment section.

The main contributions of our work are as follows:

  * •

We formulate a simple yet effective meta-training framework to address the
problem of a mixture of partial missing modalities in the MSA tasks.

  * •

The proposed method M3S can be treated as an efficient add-on training
component on existing models and significantly improve their performances on
dealing with missing modality.

  * •

We conduct comprehensive experiments on widely used datasets in MSA, including
IEMOCAP, SIMS, and CMU-MOSI. Superior performance is achieved compared with
recent SOTA methods.

##  2 Related Work

###  2.1 Emotion Recognition

Emotion recognition aims to identify and predict emotions through these
physiological and behavioral responses. Emotions are expressed in a variety of
modality forms. However, early studies on emotion recognition are often single
modality. Shaheen et al. Shaheen et al. (2014) and Calefato et al. Calefato et
al. (2017) present novel approaches to automatic emotion recognition from
text. Burkert et al. Burkert et al. (2015) and Deng et al. Deng et al. (2020)
conduct researches on facial expressions and the emotions behind them.
Koolagudi and Rao Koolagudi and Rao (2012) and Yoon et al. Yoon et al. (2019)
exploit acoustic data in different types of speeches for emotional recognition
and classification tasks. Though much progress has been made for emotion
recognition with single modality data, how to combine information from diverse
modalities has become an interesting direction in this area.

###  2.2 Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is a popular area of research in the
present since the world we live in has several modality forms. When the
dataset consists of more than one modality information, traditional single
modality methods are difficult to deal with. MSA mainly focuses on three
modalities: text, audio, and video. It makes use of the complementarity of
multimodal information to improve the accuracy of emotion recognition.
However, the heterogeneity of data and signals bring significant challenges
because it creates distributional modality gaps. Hazarika et al. Hazarika et
al. (2020) propose a novel framework, MISA, which projects each modality to
two distinct subspaces to aid the fusion process. And Hori et al. Hori et al.
(2017) introduce a multimodal attention model that can selectively utilize
features from different modalities. Since the performance of a model highly
depends on the quality of multimodal fusion, Han et al. Han et al. (2021b)
construct a framework named MultiModal InfoMax (MMIM) to maximize the mutual
information in unimodal input pairs as well as obtain information related to
tasks through multimodal fusion process. Besides, Han et al. Han et al.
(2021a) make use of an end-to-end network Bi-Bimodal Fusion Network (BBFN) to
better utilize the dynamics of independence and correlation between
modalities. Due to the unified multimodal annotation, previous methods are
restricted in capturing differentiated information. Yu et al. Yu et al. (2021)
design a label generation module based on the self-supervised learning
strategy. Then, joint training the multimodal and unimodal tasks to learn the
consistency and difference. However, limited by the pre-processed features,
the results show that the generated audio and vision labels are not
significant enough.

###  2.3 Missing Modality Problem

Compared with unimodal learning method, multimodal learning has achieved great
success. It improves the performance of emotion recognition tasks by
effectively combining the information from different modalities. However, the
multimodal data may have missing modalities in reality due to a variety of
reasons like signal transmission error and limited bandwidth. To deal with
this problem, Ma et al. Ma et al. (2021b) propose an efficient approach based
on maximum likelihood estimation to incorporate the knowledge in the modality-
missing data. Nonetheless, the more complex scenarios like missing modalities
exist in both training and testing phases are not involved. What’s more,
recent studies aim to capture the common information in different types of
training data and leverage the relatedness among different modalities Ma et
al. (2021a); Tran et al. (2017); Parthasarathy and Sundaram (2020); Wagner et
al. (2011). To solve the problem that modalities will be missing is uncertain,
Zhao et al. Zhao et al. (2021) put forward a unified model: Missing Modality
Imagination Network (MMIN). Ma et al. Ma et al. (2021c) utilize a new method
named SMIL that leverages Bayesian meta-learning to handle the problem that
modalities are partially severely missing, e.g., 90% training examples may
have incomplete modalities.

##  3 Methodology

###  3.1 Problem Description

The multimodal sentiment analysis aims at predicting the sentiment labels
$\mathcal{Y}$ based on the model $f(\mathcal{X};\bm{\theta})$ given the
multimodal data $\mathcal{X}$. We consider the input data with three
modalities, i.e. $\mathcal{X}=(\mathcal{A},\mathcal{V},\mathcal{L})$, where
$\mathcal{A}$, $\mathcal{V}$ and $\mathcal{L}$ represents audio, video and
linguistic data, respectively. In this paper, we tackle the missing modality
issue, where each modality can include missing data.

Algorithm 1 Meta-Sampling Training

1:Multimodal dataset
$(\mathcal{X}=(\mathcal{A},\mathcal{V},\mathcal{L}),\mathcal{Y})$; number of
iterations $K$ for inner loop; inner learning rate $\alpha$; outer learning
rate $\beta$; estimation model $f(\cdot\,;\bm{\theta})$; model’s loss function
${l}\left(f,\mathcal{Y}\right)$.

2:while not converged do

3: Sample batch of data $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$ from
$\mathcal{X}$.

4: Get $\tilde{\mathcal{X}}_{1}=\mathcal{T}(\mathcal{X}_{1}\,;\mathcal{F})$
and $\tilde{\mathcal{X}}_{2}=\mathcal{T}(\mathcal{X}_{2}\,;\mathcal{F})$.

5: Set $\bm{\theta}_{0}\leftarrow\bm{\theta}$

6: Meta-train:

7: for $n=0$ to $K-1$ do

8:
$\bm{\theta}_{n+1}\leftarrow\bm{\theta}_{n}-\alpha\nabla_{\theta_{n}}\,l\left(f(\tilde{\mathcal{X}}_{1};\bm{\theta}_{n}),\mathcal{Y}_{1}\right)$

9: end for

10: $\bm{\theta}^{*}\leftarrow\bm{\theta}_{K}$

11: Meta-update:

12:
$\bm{\theta}\leftarrow\bm{\theta}-\beta\nabla_{\theta^{*}}\,{l}\left(f(\tilde{\mathcal{X}}_{2};\bm{\theta}^{*}),\mathcal{Y}_{2}\right)$

13:end while

###  3.2 Augmented Missing Modality Transform

Given a sample $\bm{X}_{i}=(\bm{A}_{i},\bm{V}_{i},\bm{L}_{i})$ from
$\mathcal{X}$, we use a augmented transform
$\mathcal{T}(\bm{X}_{i}\,;\mathcal{F})$ to generate a random sample with
missing data based on a distribution $\mathcal{F}$. Specifically, for each
modality $m\in\\{a,v,l\\}$, we define a missing ratio $r_{m}\in[0,1]$, where
$a$, $v$ and $l$ stands for audio, video and linguistic modality,
respectively. For the encoded feature in each modality $m$, we replace the
values between $[\lambda_{m},\lambda_{m}+k_{m}-1]$ with zeros, where $k_{m}$
represents the number of missing values with $k_{m}=\lfloor T_{m}\cdot
r_{m}\rfloor$ and $T_{m}$ is the dimension of the encoded feature.
$\lambda_{m}$ is sampled from the uniform distribution, i.e.,
$\lambda_{m}\sim\mathcal{U}(0,T_{m}-k_{m})$. As a result, the augmented sample
with missing modality can be obtained by
$\tilde{\bm{X}}_{i}=\mathcal{T}(\bm{X}_{i}\,;\mathcal{F})$, where
$\mathcal{F}$ represents the composition of uniform distributions for each
individual modality.

###  3.3 Training with Meta-Sampling

Our M3S follows MAML training framework Finn et al. (2017) with augmentation
sampling. For each training iteration, we adopt the following steps.

First, we sample two independent batch of data, $\tilde{\mathcal{X}}_{1}$ and
$\tilde{\mathcal{X}}_{2}$, based on the augmented missing modality transforms,
$\mathcal{T}(\mathcal{X}_{1}\,;\mathcal{F})$ and
$\mathcal{T}(\mathcal{X}_{2}\,;\mathcal{F})$, where the missing rate for each
modality is determined by the sampling distribution $\mathcal{F}$.
$\tilde{\mathcal{X}}_{1}$ and $\tilde{\mathcal{X}}_{2}$ are used as tasks from
support set and query set, respectively, in the meta-learning.

Then, in the meta-train process, the model’s parameter $\bm{\theta}$ is
updated using gradient descent based on the loss function
${l}\left(f(\tilde{\mathcal{X}}_{1};\bm{\theta}),\mathcal{Y}_{1}\right)$ with
the inner learning rate $\alpha$ for each iteration $n$ as follows:

$\bm{\theta}_{n+1}\leftarrow\bm{\theta}_{n}-\alpha\nabla_{\theta_{n}}\,{l}\left(f(\tilde{\mathcal{X}}_{1};\bm{\theta}_{n}),\mathcal{Y}_{1}\right),$
(1)

where $\mathcal{Y}_{1}$ is the set of sentiment labels of
$\tilde{\mathcal{X}}_{1}$, and the loss function
${l}\left(f(\tilde{\mathcal{X}}_{1};\bm{\theta}),\mathcal{Y}_{1}\right)$ is
determined by loss used in each base model (i.e., MMIM, MISA, Self-MM, MMIN.
See Section 4.2 for more details). The meta-train process is conducted for $K$
iterations. We denote $\bm{\theta}_{K}$ as $\bm{\theta}^{*}$.

Figure 2: The Overall Architecture of M3S. We first use augmented transform to
generate two batches of data for features from each modality. Then the meta-
train and meta-update are conducted on the two batches of data to learn the
model parameters $\bm{\theta}$.

Finally, we use the query set $\tilde{\mathcal{X}}_{2}$ and its set of
sentiment labels $\mathcal{Y}_{2}$ in the outer loop meta-update step. The
model parameters are updated with the learning rate $\beta$ as follows:

$\bm{\theta}\leftarrow\bm{\theta}-\beta\nabla_{\theta^{*}}\,{l}\left(f(\tilde{\mathcal{X}}_{2};\bm{\theta}^{*}),\mathcal{Y}_{2}\right).$
(2)

The whole algorithm in general case is shown in Algorithm 1 and Figure 2
illustrates the meta-sampling training process.

##  4 Experiment Setup

In this section, we present the setup of our experiments, including the used
datasets, baseline methods, evaluation metrics, and implementation details of
the proposed method.

###  4.1 Datasets

We conduct our experiments on the following three datasets, i.e., IEMOCAP
Busso et al. (2008), SIMS Yu et al. (2020) and CMU-MOSI Zadeh et al. (2016).
The statistics of the datasets are reported in Table 1.

  * •

IEMOCAP comprises of several recorded videos in 5 conversation sessions, and
each session contains many scripted plays and dialogues. The actors performed
selected emotional scripts and also improvised hypothetical scenarios designed
to elicit specific types of emotions, which provided detailed information
about their facial expressions and hand movements.

  * •

SIMS dataset is a multimodal sentiment analysis benchmark containing 2281
video clips from various sources (i.e., movies, shows, TV serials, etc.). SIMS
contains fine-grained annotations of different modalities and includes
people’s natural expressions in video clips. And each sample in SIMS dataset
is labeled with a score from -1 to 1, standing for sentiment response (i.e.,
from strongly negative to strongly positive).

  * •

CMU-MOSI has 2199 video segments in total, which are sliced from 93 YouTube
videos. The videos address a large array of topics like books, products, and
movies. In these video segments, 89 narrators show their opinions on different
topics. Most of the speakers are around 20-30 years old. They all express
themselves in English, although they come from different countries.

Dataset | Train | Valid | Test | All
---|---|---|---|---
SIMS | 1368 | 456 | 457 | 2281
MOSI | 1284 | 229 | 686 | 2199
IEMOCAP | 4446 | 3342 | 3168 | 10956
Table 1: Statistics of the Used Datasets

Method | Self-MM (SIMS) | MMIN (IEMOCAP)
---|---|---
MAE | Corr | Acc-2 | F1-Score | Acc | Uar | F1-Score
ORIG | 0.5171 | 0.3918 | 0.7291 | 0.6980 | 0.6136 | 0.6403 | 0.6049
ORIG + SPL-TRN | 0.5049 | 0.4080 | 0.7392 | 0.7102 | 0.6357 | 0.6518 | 0.6235
ORIG + M3S | 0.5053 | 0.4091 | 0.7405 | 0.7119 | 0.6398 | 0.6536 | 0.6296
$\Delta_{ORIG}$ | $\downarrow 0.0118$ | $\uparrow 0.0173$ | $\uparrow 0.0114$ | $\uparrow 0.0139$ | $\uparrow 0.0262$ | $\uparrow 0.0133$ | $\uparrow 0.0247$
Method | MISA (MOSI) | - | MMIM (MOSI)
MAE | Corr | Acc-7 | - | MAE | Corr | Acc-7
ORIG | 0.8886 | 0.7349 | 0.3863 | - | 0.7175 | 0.7883 | 0.4592
ORIG + SPL-TRN | 0.8279 | 0.7355 | 0.4155 | - | 0.7126 | 0.7825 | 0.4650
ORIG + M3S | 0.8393 | 0.7346 | 0.4282 | - | 0.7014 | 0.7985 | 0.4852
$\Delta_{ORIG}$ | $\downarrow 0.0493$ | $\downarrow 0.0003$ | $\uparrow 0.0419$ | - | $\downarrow 0.0161$ | $\uparrow 0.0102$ | $\uparrow 0.0260$

Table 2: Results of four baseline models with different training methods
applied. Input and test data both have missing rates between 40% and 60%. ORIG
stands for original model; SPL-TRN stands for sampling-training.
$\Delta_{ORIG}$ presents the improved performance based on original model that
M3S has achieved.

###  4.2 Baseline Methods

We use four recent SOTA methods for comparison in the experiments. The methods
include MMIM Han et al. (2021b), MISA Hazarika et al. (2020), Self-MM Yu et
al. (2021) and MMIN Zhao et al. (2021), which are summarized as follows.

  * †

MMIM helps mutual information reach maximum and maintains information related
to tasks during the process of multimodal fusion, which shows significant
results in multimodal sentiment analysis tasks.

  * †

MISA is a novel model in emotion recognition that represents modality more
effectively and improves the fusion process significantly.

  * †

Self-MM has novel architecture containing several innovative modules (like a
module for label generation) and reaches brilliant results in multimodal
sentiment analysis tasks.

  * †

MMIN handles the problem that input data has uncertain modalities completely
missing and achieves superior results under various missing modality
conditions.

###  4.3 Evaluation Metrics

Following the four baseline methods mentioned above, we use the following
evaluation metrics, including mean absolute error (MAE), Pearson correlation
(Corr), binary classification accuracy (Acc-2), weighted F1 score (F1-Score),
accuracy score (Acc), unweighted average recall (Uar), and seven-class
classification accuracy (Acc-7). Acc-7 denotes the ratio of predictions that
are in the correct interval among the seven intervals ranging from -3 to 3.
For all metrics, higher values show better performance except for MAE.

###  4.4 Implementation Details

#### Hyperparameter Settings.

The settings of inner learning rate, outer learning rate and batch size
$\\{\alpha$, $\beta$, batch_size$\\}$ are as follows: MMIN $\\{$2e-4, 1e-4,
256$\\}$; MMIM $\\{$1e-3, 1e-3, 32$\\}$; MISA $\\{$1e-4, 1e-4, 128$\\}$; For
Self-MM, the learning rate for three modalities
$\\{\mathcal{A},\mathcal{V},\mathcal{L}\\}$ is $\\{$5e-3, 5e-3, 5e-5$\\}$, and
the batch size is 32.

#### Feature Extraction Details.

Following the baseline methods, we adopt the extracted features as the input
for each modality. The feature extraction methods on each modality
$\\{\mathcal{A},\mathcal{V},\mathcal{L}\\}$ are listed as follows: MMIN
$\\{$OpenSMILE-"IS13_ComParE" Eyben et al. (2010), DenseNet Huang et al.
(2017) trained on FER+ corpus Barsoum et al. (2016), BERT Devlin et al.
(2018)$\\}$; Self-MM, MMIM, MISA $\\{$sLSTM Hochreiter and Schmidhuber (1997),
sLSTM, BERT$\\}$.

#### Experimental Details.

We use Adam as the optimizer for all four baseline models. The training epoch
for $\\{$MMIN, MMIM, MISA$\\}$ is $\\{60,40,500\\}$. Self-MM adopts the "early
stop" strategy to obtain the best result. Therefore, its training epoch is
unfixed. In Section 5.1, We compare the performance of three different
training methods dealing with missing modalities in our experiment results: 1)
original model’s training method (ORIG), where the missing rate of each sample
is fixed along the training process during different epochs; 2) original model
with Sampling-Training strategy applied (ORIG + SPL-TRN), which adopts
augmented sampling without meta-learning process, as illustrated in Section
3.2; 3) original model with M3S added on (ORIG + M3S), which is the proposed
method.

##  5 Results and Analysis

###  5.1 Main Results

Input Missing Rate | Method | MMIN (IEMOCAP) | MMIM (MOSI)
---|---|---|---
Acc | Uar | F1-Score | MAE | Corr | Acc-7
60% $\sim$ 80% | ORIG | 0.5849 | 0.5915 | 0.5748 | 0.7132 | 0.7905 | 0.4577
ORIG + SPL-TRN | 0.5812 | 0.5901 | 0.5689 | 0.7268 | 0.7867 | 0.4549
2-8 | ORIG + M3S | 0.5900 | 0.6026 | 0.5764 | 0.7208 | 0.7890 | 0.4588
| $\Delta_{ORIG}$ | $\uparrow 0.0051$ | $\uparrow 0.0111$ | $\uparrow 0.0016$ | $\uparrow 0.0076$ | $\downarrow 0.0015$ | $\uparrow 0.0011$
40% $\sim$ 60% | ORIG | 0.6136 | 0.6403 | 0.6049 | 0.7175 | 0.7883 | 0.4592
ORIG + SPL-TRN | 0.6357 | 0.6518 | 0.6235 | 0.7126 | 0.7825 | 0.4650
2-8 | ORIG + M3S | 0.6398 | 0.6536 | 0.6296 | 0.7014 | 0.7985 | 0.4852
| $\Delta_{ORIG}$ | $\uparrow 0.0262$ | $\uparrow 0.0133$ | $\uparrow 0.0247$ | $\downarrow 0.0161$ | $\uparrow 0.0102$ | $\uparrow 0.0260$
20% $\sim$ 40% | ORIG | 0.6192 | 0.6453 | 0.6078 | 0.7129 | 0.7893 | 0.4694
ORIG + SPL-TRN | 0.6335 | 0.6513 | 0.6221 | 0.7218 | 0.7832 | 0.4665
2-8 | ORIG + M3S | 0.6367 | 0.6504 | 0.6266 | 0.7049 | 0.7923 | 0.4838
| $\Delta_{ORIG}$ | $\uparrow 0.0175$ | $\uparrow 0.0051$ | $\uparrow 0.0188$ | $\downarrow 0.0080$ | $\uparrow 0.0030$ | $\uparrow 0.0144$

Table 3: Results on MMIN and MMIM under three different missing rate levels.
Test data have the same range of missing rates as input data.

(a) Valid Loss

(b) Test Loss

Figure 3: Validation and testing losses of three methods along training built
on the MMIM Model.

Built on the baseline models, we conduct experiments with the proposed M3S
method and show its effectiveness in Table 4.1. The missing rate is set as the
medium rate, between 40% and 60%. Since M3S can be an add-on component to
existing methods with the capability of dealing with missing modality, we
compare M3S with Sampling-Training (SPL-TRN) and four original baseline
methods. For all the testing datasets, M3S achieves superior performance in
almost all evaluation metrics compared with the original baseline methods, as
expected. Since SPL-TRN only adopts augmented sampling without meta-learning
process, it achieves worse performance than our M3S method in most of the
experiments. This result demonstrates that the meta-sampling training process
can better learn the common knowledge from other modalities to deal with the
missing information. It also verifies that meta-training can better utilize
the information from random augmentations. As a matter of fact, with the help
of M3S, MMIN model achieves the highest Acc, highest Uar, and highest
F1-Score. Also, built upon the other three baselines (Self-MM, MISA, MMIM),
M3S helps in reaching the lowest MAE, highest Corr, and highest Acc in most
situations, which shows the efficiency and universality of M3S.

###  5.2 Studies of Various Missing Rates

To verify the effectiveness of methods on different missing rates, we conduct
experiments on two datasets by varying the input missing rate to three levels
(i.e., 20%-40%, 40%-60%, and 60%-80%). Results in Table 5.1 show that for
nearly all the cases, our method M3S outperforms ORIG and ORIG+SPL-TRN
methods. Specifically, when input missing rate falls within the range 40%-60%,
ORIG+M3S shows the greatest increment in all metrics, which shows that M3S
achieves the most significant effect on models with medium missing level.

###  5.3 Convergence Comparison

(a) Uar

(b) F1-Score

Figure 4: Uar and F1-Score of three methods along training built on the MMIN
Model.

As is shown in Figure 3(a) and 3(b), we plot the process of MMIM model’s loss
decline. It is clearly shown in plots that M3S helps original model converge
to the lowest loss after 10 to 15 epochs of training. As shown in Figure 4(a)
and Figure 4(b), we also select MMIN model and plot its convergence process
because the trend of its metrics changes more obviously. These two figures,
along with Figure 1 show the characteristic of our method: although M3S does
not show strong competitiveness in the first few epochs, with the progress of
training, M3S helps model achieve faster growth of various metrics and finally
converge to a higher result.

MMIN | ORIG | ORIG + | ORIG + | $\Delta_{ORIG}$
---|---|---|---|---
SPL-TRN | M3S
Acc | 0.6035 | 0.6152 | 0.6206 | $\uparrow 0.0171$
Uar | 0.6281 | 0.6166 | 0.6140 | $\downarrow 0.0141$
F1-Score | 0.5953 | 0.6023 | 0.6072 | $\uparrow 0.0119$

MMIM | ORIG | ORIG + | ORIG + | $\Delta_{ORIG}$
---|---|---|---|---
SPL-TRN | M3S
MAE | 0.7201 | 0.7412 | 0.7025 | $\downarrow 0.0176$
Corr | 0.7794 | 0.7695 | 0.7884 | $\uparrow 0.0090$
Acc-7 | 0.4534 | 0.4461 | 0.4825 | $\uparrow 0.0291$

Table 4: Results on MMIN (IEMOCAP) and MMIM (MOSI), where input data have
missing rates 40%-60% and test data have missing rates 60%-80%.

P-value of t-test | Self-MM (SIMS) | MMIN (IEMOCAP)
---|---|---
MAE | Corr | Acc-2 | F1-Score | Acc | Uar | F1-Score
$P(T\leq t)$ | 0.1959 | 0.0384 | 0.0018 | 0.0615 | 0.0007 | 7.95E-5 | 0.0005
P-value of t-test | MISA (MOSI) | - | MMIM (MOSI)
MAE | Corr | Acc-7 | - | MAE | Corr | Acc-7
$P(T\leq t)$ | 0.0473 | 0.1873 | 0.0405 | - | 0.0277 | 0.1971 | 0.0263

Table 5: Two-tailed significance test (t-test) of M3S.

###  5.4 Adaptation across Different Missing Rates

In order to further discover the efficiency of our method in helping models
adapt to different missing rates, we conduct experiments with testing rates
different from input rates. As shown in Table 4, compared to ORIG method, we
can see that M3S significantly improves nearly all metrics by at least 1%. It
is worth noticing that a large missing rate (60%-80%) is adopted in the
testing, and M3S achieves much better performance than the other two methods.
For example, the Acc-7 of M3S on MOSI dataset is over 3.6% higher than the one
of ORIG+SPL-TRN method, demonstrating the capability of M3S when different
modalities have large missing information.

###  5.5 Further Discussion and Limitations

The qualitative results and ablation study above show that M3S significantly
helps baseline models improve their performance on inputs with various missing
rates. However, when we apply M3S to Self-MM model and conduct experiments on
CMU-MOSI dataset, we find that the results show little difference from the
original model’s result. Besides, from Table 4.1 we know that M3S improves
Self-MM’s performance on SIMS dataset significantly. Hence we assume that this
is because Self-MM model has good adaptability to CMU-MOSI dataset but not
SIMS dataset when both datasets have a mixture of missing across modalities.
Therefore, some models may show adaptivity to certain datasets. And M3S may
not significantly improve the model’s performance on those datasets that model
is already quite adaptive to.

Also, as shown in Table 5.1, it’s revealed that when inputs have a large
missing rate (60%-80%), M3S becomes limited in improving evaluation metrics.
We attribute this to the change of sampling range. That is, when inputs have
missing rates no more than 60%, we can create sufficient augmented missing
data to perform M3S. However, when inputs have large missing rates, we can
only get augmented data with missing rates restricted to a smaller range. Thus
we get a smaller sampling range containing large missing rate data, which
makes M3S limited.

But in general, M3S method is recommended as it is easy to be added on
different models and efficient in improving models’ performance on multimodal
sentiment analysis tasks most of the time, especially when input data has a
medium missing rate. As shown in Table 5, nearly all evaluation metrics’
$P$-value is smaller than 0.05 in the significance test, indicating
significant improvement when M3S is applied.

##  6 Conclusion and Future Work

In this paper, we focus on a challenging problem, i.e., multimodal sentiment
analysis on a mixture of missing across modalities, which was seldom studied
in the past. We propose a simple yet effective method called M3S to handle the
problem. M3S is a meta-sampling training method that follows the MAML
framework and combines the sampling strategy for augmented transforms. M3S
maintains the advantages of meta-learning and helps SOTA models achieve
superior performance on various missing input modalities.

In the experiments, we show that our method M3S improves four baselines’
performance and helps them adapt to inputs with various missing rates.
Furthermore, M3S is easy to realize in different multimodal sentiment analysis
models. In future work, we plan to investigate how to better combine M3S with
other training methods and extend the method to other multimodal learning
tasks.

## Ethical Considerations

Our proposed method aims to help improve the performance of different SOTA
methods on data with various missing rates. All experiments we conduct are
based on the open public datasets (Section 4.1) and pretraining baseline
methods (Section 4.2). When applying our method in experiments, there is
minimal risk of privacy leakage. Furthermore, since our method is an add-on
component for different baselines, it is safe to apply it as long as the
baseline model provides adequate protection for privacy.

## Acknowledgements

This work is supported by National Natural Science Foundation of China
(62106219). We would like to thank the anonymous reviewers for their valuable
and detailed suggestions.

## References

  * Barsoum et al. (2016) Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016.  Training deep networks for facial expression recognition with crowd-sourced label distribution.  In _Proceedings of the 18th ACM International Conference on Multimodal Interaction_ , pages 279–283.
  * Burkert et al. (2015) Peter Burkert, Felix Trier, Muhammad Zeshan Afzal, Andreas Dengel, and Marcus Liwicki. 2015.  Dexpression: Deep convolutional neural network for expression recognition.  _arXiv preprint arXiv:1509.05371_.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_ , 42(4):335–359.
  * Cai et al. (2018) Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. 2018.  Deep adversarial learning for multi-modality missing data completion.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1158–1166.
  * Calefato et al. (2017) Fabio Calefato, Filippo Lanubile, and Nicole Novielli. 2017.  Emotxt: A toolkit for emotion recognition from text.  In _2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos_ , pages 79–80. IEEE.
  * Deng et al. (2020) Didan Deng, Zhaokang Chen, and Bertram E Shi. 2020.  Multitask emotion recognition with incomplete labels.  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition_ , pages 592–599. IEEE.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ding et al. (2014) Zhengming Ding, Shao Ming, and Yun Fu. 2014.  Latent low-rank transfer subspace learning for missing modality recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 28.
  * Eyben et al. (2010) Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In _Proceedings of the 18th ACM International Conference on Multimedia_ , pages 1459–1462.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pages 1126–1135. PMLR.
  * Han et al. (2021a) Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Poria. 2021a.  Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , pages 6–15.
  * Han et al. (2021b) Wei Han, Hui Chen, and Soujanya Poria. 2021b.  Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.  _arXiv preprint arXiv:2109.00412_.
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1122–1131.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Hori et al. (2017) Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. 2017.  Attention-based multimodal fusion for video description.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4193–4202.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017.  Densely connected convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4700–4708.
  * Koolagudi and Rao (2012) Shashidhar G Koolagudi and K Sreenivasa Rao. 2012.  Emotion recognition from speech: a review.  _International Journal of Speech Technology_ , 15(2):99–117.
  * Ma et al. (2021a) Fei Ma, Shao-Lun Huang, and Lin Zhang. 2021a.  An efficient approach for audio-visual emotion recognition with missing labels and missing modalities.  In _2021 IEEE International Conference on Multimedia and Expo_ , pages 1–6. IEEE.
  * Ma et al. (2021b) Fei Ma, Xiangxiang Xu, Shao-Lun Huang, and Lin Zhang. 2021b.  Maximum likelihood estimation for multimodal learning with missing modality.  _arXiv preprint arXiv:2108.10513_.
  * Ma et al. (2022) Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. 2022.  Are multimodal transformers robust to missing modality?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18177–18186.
  * Ma et al. (2021c) Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. 2021c.  Smil: Multimodal learning with severely missing modality.  _arXiv preprint arXiv:2103.05677_.
  * Parthasarathy and Sundaram (2020) Srinivas Parthasarathy and Shiva Sundaram. 2020.  Training strategies to handle missing modalities for audio-visual expression recognition.  In _Companion Publication of the 2020 International Conference on Multimodal Interaction_ , pages 400–404.
  * Shaheen et al. (2014) Shadi Shaheen, Wassim El-Hajj, Hazem Hajj, and Shady Elbassuoni. 2014.  Emotion recognition from text based on automatically generated rules.  In _2014 IEEE International Conference on Data Mining Workshop_ , pages 383–392. IEEE.
  * Tran et al. (2017) Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. 2017.  Missing modalities imputation via cascaded residual autoencoder.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1405–1414.
  * Wagner et al. (2011) Johannes Wagner, Elisabeth Andre, Florian Lingenfelser, and Jonghwa Kim. 2011.  Exploring fusion methods for multimodal emotion recognition with missing data.  _IEEE Transactions on Affective Computing_ , 2(4):206–218.
  * Yoon et al. (2019) Seunghyun Yoon, Seokhyun Byun, Subhadeep Dey, and Kyomin Jung. 2019.  Speech emotion recognition using multi-hop attention mechanism.  In _2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 2822–2826. IEEE.
  * Yu et al. (2020) Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020.  Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3718–3727.
  * Yu et al. (2021) Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis.  _arXiv preprint arXiv:2102.04830_.
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6):82–88.
  * Zhao et al. (2021) Jinming Zhao, Ruichen Li, and Qin Jin. 2021.  Missing modality imagination network for emotion recognition with uncertain missing modalities.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , volume 1, pages 2608–2618.


# Test-Time Training for Deformable
Multi-Scale Image Registration ††thanks: Corresponding author: Xiaohui Xie,
University of California, Irvine (xhx@ics.uci.edu).

Wentao Zhu1 Yufang Huang2 Daguang Xu3 Zhen Qian4 Wei Fan4 Xiaohui Xie5
1Kuaishou Technology 2Cornell University 3NVIDIA 4Tencent 5University of
California, Irvine

###### Abstract

Registration is a fundamental task in medical robotics and is often a crucial
step for many downstream tasks such as motion analysis, intra-operative
tracking and image segmentation. Popular registration methods such as ANTs and
NiftyReg optimize objective functions for each pair of images from scratch,
which are time-consuming for 3D and sequential images with complex
deformations. Recently, deep learning-based registration approaches such as
VoxelMorph have been emerging and achieve competitive performance. In this
work, we construct a test-time training for deep deformable image registration
to improve the generalization ability of conventional learning-based
registration model. We design multi-scale deep networks to consecutively model
the residual deformations, which is effective for high variational
deformations. Extensive experiments validate the effectiveness of multi-scale
deep registration with test-time training based on Dice coefficient for image
segmentation and mean square error (MSE), normalized local cross-correlation
(NLCC) for tissue dense tracking tasks.

###### Index Terms:

Computer Vision for Medical Robotics, Image Registration, Visual Tracking

##  I Introduction

Image registration tries to establish the correspondence between organs,
tissues, landmarks, edges, or surfaces in different images and it is critical
to many clinical tasks such as tumor growth monitoring and surgical robotics
[1]. Manual image registration is time-consuming, laborious and lacks
reproducibility which causes clinical disadvantage potentially. Thus,
automated registration is desired in many clinical practices. Generally,
registration can be necessary to analyze motion from videos, auto-segment
organs given atlases, and align a pair of images from different modalities,
acquired at different times, from different viewpoints or even from different
patients. Thus designing a robust image registration can be challenging due to
the high variability of scenarios.

Traditional registration methods estimate the registration fields by
optimizing certain objective functions. Such registration field can be modeled
in several ways, e.g. elastic-type models [2, 3], free-form deformation [4],
Demons [5], and statistical parametric mapping [6]. Beyond the deformation
model, diffeomorphic transformations preserve topology and many methods adopt
them such as large deformation diffeomorphic metric mapping [7], symmetric
image normalization (SyN) [8] and diffeomorphic anatomical registration using
exponential Lie algebra [9]. One limitation of these methods is that the
optimization can be computationally expensive especially for 3D images.

Deep learning-based registration methods have recently been emerging as a
practicable alternative to the conventional registrations [10, 11, 12, 13, 14,
15, 16, 17, 18, 19]. These methods employ sparse or weak label of registration
field, or conduct supervised learning [20, 10, 21, 11, 22, 23] purely based on
registration field ground truth. Facilitated by a spatial transformer network
[24], recent unsupervised deep learning-based registrations [25, 26, 27, 28,
29, 30, 31, 32, 33], such as VoxelMorph [34, 35], have been explored because
of annotation free in the training. VoxelMorph is further extended to
diffeomorphic transformation and Bayesian framework [36]. Adversarial
similarity network adds an extra discriminator to model the appearance loss
between warped image and fixed image, and uses adversarial training to improve
the unsupervised registration [37, 30, 38, 39]. However, most of these
registration methods have comparable accuracy as traditional registrations,
although with potential speed advantages.

In this work, we design a novel test-time training for deep deformable image
registration framework with multi-scale parameterization of complicated
deformations for accurate registration to handle large deformation and noise
widely existing in medical images. The framework, called self-supervision
optimized multi-scale registration as illustrated in Fig. 1, is motivated by
the fact that these purely learning-based registrations generally cannot
generalize well on noisy images with large and complicated deformations
because of domain shift between training data and test data [40]. More
specifically, we propose a registration optimized in both training and test to
tackle the generalization gap between training images and test images.
Different from unsupervised registration [34], registration with test-time
training [40] further tunes the deep network in each test image pair, which is
vital for the success of noisy image registration in Fig. 3 and 4. Inspired by
the improvement of multi-scale registration in conventional registration [41],
we further employ a multi-scale scheme to estimate accurate deformation where
we conduct test-time training with a U-Net [42] to model the residual
registration field in each scale.

Figure 1: Framework of multi-scale deep registration networks with test-time
training in scale $s$.

Our main contributions are as follows. 1) We design a novel test-time training
for deep deformable registration to improve generalization ability of
learning-based registration. The test-time training with self-supervised
optimization for deep registration network yields accurate registration field
estimation even for noisy images with large deformations by eliminating the
accuracy gap between the estimation on the training set and that on the test
set. 2) We design a deep multi-scale registration based on unsupervised
learning framework to model the large deformations. The multi-scale strategy
estimates the residual registration field consecutively, which enforces a
coarse to fine consistency of the morphing, and it provides a sequential
optimization pathway which yields much more accurate registration field.

##  II Method

###  II-A Notations and Framework

Let $\bm{M}$ and $\bm{I}$ be two images defined over an n-dimensional spatial
domain $\mathrm{\Omega}\in\mathbb{R}^{n}$. In this paper, we focus on either
$n=2$ for 2D images such as X-ray and ultrasound, or $n=3$ for volumetric
images such as computed tomography (CT) or magnetic resonance imaging (MRI).
For simplicity, we assume both $\bm{M}$ and $\bm{I}$ are gray-scale containing
a single channel. Our goal is to align $\bm{M}$ to $\bm{I}$ through a
deformable transformation so that the anatomical features in transformed
$\bm{M}$ are aligned to those in $\bm{I}$. To distinguish them, we call
$\bm{M}$ and $\bm{I}$ the moving and fixed images, respectively.

Let $\phi\colon\mathrm{\Omega}\to\mathrm{\Omega}$ be the registration field
that maps coordinates of $\bm{I}$ to coordinates of $\bm{M}$. The image
registration problem [43, 2, 44] can be generally formulated as minimizing the
anatomical differences between $\bm{M}\circ\phi$ ($\bm{M}$ warped by $\phi$)
and $\bm{I}$, regularized by a smoothness constraint on the registration
field,

$\hat{\phi}=\operatorname*{arg\,min}_{\phi}\
\mathcal{L}_{R}(\bm{I},\bm{M}\circ\phi)+\lambda\mathcal{L}_{S}(\phi)$ (1)

where $\mathcal{L}_{R}$ is a reconstruction loss measuring the dissimilarity
between two images, and $\mathcal{L}_{S}$ measures the smoothness of the
registration field. In this work, we assume $\phi=\mathrm{id}+\mathbf{u}$ with
$\mathrm{id}$ denoting the identify mapping, and model the displacement field
$\mathbf{u}$ instead. Let $u_{i}$ denote the $i$-th coordinate of the
displacement field. The smoothness term is taken to be

$\mathcal{L}_{S}(\phi)=\sum_{\mathbf{p}\in\mathrm{\Omega}}\sum_{i=1}^{n}\|\nabla
u_{i}(\mathbf{p})\|^{2}$

throughout this paper with the gradients approximated by displacement
differences between neighboring grids in the actual implementation.

###  II-B Deep Learning-Based Registration

We model the displacement field through a deep neural net,
$\mathbf{u}=\mathcal{N}(\bm{I},\bm{M};\bm{\theta})$, receiving $\bm{I}$ and
$\bm{M}$ as input and generating $\mathbf{u}$ through a neural net with weight
parameters $\bm{\theta}$. Finding the registration field is thus expressed as
a learning problem to identify the optimal parameters $\bm{\theta}$ to
minimize the loss function shown in Eq. (1). We employ a U-Net [42]
$\mathcal{N}(\cdot,\cdot;\bm{\theta})$ with skip connections to estimate the
displacement field $\mathbf{u}$. The input to the network is the concatenation
of $\bm{M}$ and $\bm{I}$ across the channel dimension. The U-Net configuration
is illustrated in Fig. 1.

Given a point $\mathbf{p}\in\mathrm{\Omega}$, the registration field aligns
its image feature $\bm{I}(\mathbf{p})$ in the fixed image to the image feature
at location $\phi(\mathbf{p})$ of the moving image. Because image values are
only defined over a grid of integer points, we linearly interpolate the values
of $\bm{M}\circ\phi(\mathbf{p})$ from the grid points neighboring
$\phi(\mathbf{p})$ through a spatial transformer network $\mathcal{T}$ [24],
which generates a warped moving image according to the registration field
$\bm{I}_{R}=\mathcal{T}(\bm{M},\phi)$. This formulation allows gradient
calculation and back-propagating errors during learning.

Many metrics have been proposed to measure the anatomical similarity between
fixed image $\bm{I}$ and warped moving image $\bm{I}_{R}$, including mean-
squared error, mutual information, cross-correlation. In this work, we use the
negative normalized local cross-correlation as the reconstruction loss, which
tends to be more robust for noisy images, although other metrics can be
equally applied

$\displaystyle\mathcal{L}_{R}(\bm{I},\bm{M}\circ\phi)=$ (2)
$\displaystyle-\frac{1}{|\mathrm{\Omega}|}\sum_{\bm{p}\in\mathrm{\Omega}}\frac{\big{(}\sum_{\bm{q}}{(\bm{I}(\bm{q})-\overline{\bm{I}(\bm{p})})(\bm{I}_{R}(\bm{q})-\overline{\bm{I}_{R}(\bm{p})})}\big{)}^{2}}{\sum_{\bm{q}}(\bm{I}(\bm{q})-\overline{\bm{I}(\bm{p})})^{2}\sum_{\bm{q}}(\bm{I}_{R}(\bm{q})-\overline{\bm{I}_{R}(\bm{p})})^{2}},$

where $\bm{q}$ is a location within a local neighborhood around $\bm{p}$, and
$\overline{\bm{I}(\bm{p})}$ and $\overline{\bm{I}_{R}(\bm{p})}$ are the local
means of the fixed and warped images, respectively.

###  II-C Test-Time Training for Registration

Given training and test data, previous neural based registration methods take
the standard paradigm of learning the parameters of the neural net by
minimizing the loss function Eq. (1) on the training set, and then derive the
displacement field on the test set based on the learned parameters. The
benefit of this approach is that the inference is fast but coming at the cost
of significant performance reduction. The main cause for this is that each
pair of medical images has its own unique characteristics and it is difficult
for learned models to generalize well on new test images. For medical images
with low signal-to-noise ratios, such as ultrasound images, the performance
gap between the training and test sets can be substantial.

We propose to use self-supervised optimization to improve the generalization
accuracy of learning-based registration. Under this paradigm, the network
parameters $\bm{\theta}$ is further optimized based on both training and the
test image pairs

$\displaystyle\bm{\theta}_{\star}=\mathop{\arg\min}_{\bm{\theta}}\mathbb{E}_{(\bm{M},\bm{I})\in\mathcal{D}}[{\mathcal{L}_{R}(\bm{I},\mathcal{T}(\bm{M},\phi);\bm{\theta})+\lambda\mathcal{L}_{S}(\phi;\bm{\theta})}],$
(3)

where $\mathcal{D}$ contains both the image pairs in the training set as well
as test image pairs, and $\lambda$ is a regularization parameter balancing the
trade-off between the reconstruction and smoothness losses.

Our approach aims to seek a middle ground between traditional optimization-
based approaches and pure learning-based approaches. The neural network
parameterized registration trained with stochastic optimization alleviates
some of key challenges in traditional optimization-based approaches such as
high cost in optimization, long running time, and poor performance due to
local optimality [45], while at the same time improves the performance of
purely learning-based approach through further fine tuning on test images.

###  II-D Multi-Scale Neural Registration

The registration field for medical images often have both large and small-
scale displacement vectors throughout the spatial domain. This is most
apparent in echocardiogram, where we use image registration to align
temporally nearby images to detect tissue or blood flow movements. In order to
capture the displacement field at various scales, we propose a multi-scale
scheme to parameterize the residual registration field. At each scale, a self-
supervision optimized registration network uses the concatenation of
reconstructed image from last scale and fixed image of current scale as input.
A U-Net is used to parameterize the residual registration field of each scale.
The final registration field is calculated by fusing the residual registration
fields across different scales.

We choose a sequence of spatial scales $\\{S,2S,\cdots,s,\cdots,1/2,1\\}$ from
coarse to fine to parameterize the neural registration field, with each
modeled by a U-Net. We employ resize with different image sizes for different
spatial scales. For instance, $\mathcal{N}(\cdot,\cdot;\bm{\theta}^{s})$
models the neural registration field $\tilde{\phi}^{s}$ at scale $s$. At each
scale, we use self-supervised optimization to obtain the parameter
$\bm{\theta}^{s}$ in the corresponding U-Net.

At each scale $s$, the input to the neural registration field
$\mathcal{N}(\cdot,\cdot;\bm{\theta}^{s})$ is the concatenation of
reconstructed image $\bm{M}^{s}$ and fixed image $\bm{I}^{s}$, where
$\bm{M}^{s}$ is the reconstructed moving image obtained from the previous
scale $s/2$ downsampled to scale $s$ and $\bm{I}^{s}$ is the fixed image
downsampled to scale $s$. At the coarsest scale $S$, $\bm{M}^{S}$ is taken to
be downsampled moving image to scale $S$. Let $\phi^{s}$ denote the
registration field obtained at each scale $s$. (Note that $\phi^{s}$ covers
the original spatial domain of the moving image, different from
$\tilde{\phi}^{s}$, which is a residual mapping between down-sampled domains,
i.e., the domain of $\bm{I}^{s}$.) Then the reconstructed image from scale
$s/2$ is $\mathcal{T}(\bm{M},\phi^{s/2})$ and $\bm{M}^{s}$ is the down-sampled
version of this image to scale $s$. By default, we assume $\phi^{S/2}$ is the
identify map because $S$ is the coarsest scale, and
$\mathcal{T}(\bm{M},\phi^{S/2})=\bm{M}$.

More specifically, at scale $s$, we first resize the moving image
$\mathcal{T}(\bm{M},\phi^{s/2})$ and the fixed image $\bm{I}$ to the current
scale $s$ for data preparation as

$\displaystyle\bm{M}^{s}=\mathcal{P}_{1}(\mathcal{T}({\bm{M}},\phi^{s/2}),s),\quad\bm{I}^{s}=\mathcal{P}_{1}(\bm{I},s)$
(4)

where $\bm{M}^{s}$ is the downsampled reconstructed image and
$\mathcal{P}_{1}$ is the down-sampling operator, ${\bm{M}}$ is the moving
image. We employ a U-Net $\mathcal{N}(\cdot,\cdot;\bm{\theta}^{s})$ to model
the residual registration field of scale $s$,
$\tilde{\phi^{s}}=\mathcal{N}(\bm{I}^{s},\bm{M}^{s};\bm{\theta}^{s})$, where
$\bm{\theta}^{s}$ is the parameters in the U-Net.

###  II-E Multi-Scale Registration Field Aggregation

After the test-time training, we obtain the optimal parameters
$\bm{\theta}^{s}_{\star}$ of the neural network and calculate the residual
registration field $\tilde{\bm{\phi}}^{s}$ for the last scale’s reconstructed
image $\mathcal{T}({\bm{M}},{\bm{\phi}}^{s/2})$ of scale $s$. We calculate the
registration field $\bm{\phi}^{s}$ for the moving image $\bm{M}$ by
aggregating registration field $\bm{\phi}^{s/2}$ of the last scale $s/2$ and
intermediate/residual registration field $\tilde{\bm{\phi}}^{s}$. For each
pixel position $\bm{p}$ in the moving image $\bm{M}$, we obtain the final
position $\hat{\bm{p}}$ and the combined registration field $\bm{\phi}^{s}$ by

$\displaystyle\bm{\phi}^{s}$ $\displaystyle=\phi^{s/2}\circ\tilde{\phi^{s}}$
(5) $\displaystyle\bm{\phi}^{s}(\bm{p})$ $\displaystyle=\hat{\bm{p}}-\bm{p}$
$\displaystyle=\bm{p}+\bm{\phi}^{s/2}(\bm{p})+\mathcal{P}_{2}(\frac{1}{s}\tilde{\phi}^{s},\frac{1}{s})(\bm{p}+\bm{\phi}^{s/2}(\bm{p}))-\bm{p}$
$\displaystyle=\phi^{s/2}(\bm{p})+\mathcal{P}_{2}(\frac{1}{s}\tilde{\phi}^{s},\frac{1}{s})(\bm{p}+\phi^{s/2}(\bm{p})),$

where $\mathcal{P}_{2}$ is the linear interpolation operator for up-sampling
and we use linear intepolation to calculate the field for point
$\bm{p}+\phi^{s/2}(\bm{p})$ in the current generated residual registration
field $\tilde{\phi}^{s}$. We do not need to calculate the combined
registration for the coarsest scale $S$ because $\phi^{S/2}$ is a zero field
as predefined.

##  III Results

We validate test-time training for multi-scale deformable registration,
including ablation study, on three datasets.

###  III-A Data

We employ a 3D hippocampus MRI dataset from the medical segmentation decathlon
[46] to validate the proposed method for registration-based segmentation. On
the hippocampus dataset, we randomly split the dataset into 208 training
images and 52 test images. There are two foreground categories, hippocampus
head and hippocampus body. We re-sample the MR images to $1\times 1\times 1$
$\mathrm{mm}^{3}$ spacing. To reduce the discrepancy of the intensity
distributions of the MR images, we calculate the mean and standard deviation
of each volume and clip each volume up to six standard deviations. Finally, we
linearly transform each 3D image into range $[0,1]$. The image size is within
$48\times 64\times 48$ voxels.

We collect echocardiograms from 19 patients with 3,052 frames in total for
myocardial tracking, and contrast echocardiograms from 71 patients with 11,462
frames in total for cardiac blood tracking. Contrast enhanced
echocardiography-based vortex imaging has been used in patients with
cardiomyopathy, LV hypertrophy, valvular diseases and heart failure. For
testing, we randomly choose three patients’ echocardiograms with 291 frames
for myocardial tracking, and three patients’ echocardiograms with 216 frames
for cardiac blood tracking from the two datasets. The rest of echocardiograms
are used for training. We use large training set to facilitate the learning
based-method, VoxelMorph [34], to perform well. The frame per second (FPS) of
ultrasound for myocardial tracking is 75 and the FPS for cardiac blood
tracking is from 72 to 93. Because of the consistency of FPS, it has little
impact on the results. All echocardiograms have no registration field ground
truth.

We only use the first channel of echocardiography images (e.g., treated as
gray-scale images) with the pixel values normalized to be in $[0,1]$ by being
divided by 255. The image size is $1024\times 768$. To remove the background
and improve the robustness of registration, we conduct registration on region
of pixel value within the range $[0.005,0.995]$ which is the default setting
in ANTs [8, 43]. To obtain a smooth boundary for stable optimization, we
further employ morphological dilation with a disk-shaped structure of radius
of 16 pixels to extract myocardial region. For cardiac blood flow tracking, we
extract cardiac blood region by 1) creating masks of the left ventricular
blood pool at the end of the systole and the end of the diastole, 2) using
active contour model to fit 100 uniformly sampled spline points along a circle
into the boundary of cardiac blood mask [47], 3) using linear interpolation to
get 100 interpolated spline points for each frame, 4) using radial basis
function in interpolation to obtain the final smooth cardiac blood boundary
from the 100 spline points. Removing myocardial region is crucial to cardiac
blood tracking.

###  III-B Performance Evaluation and Implementation

For unsupervised learning, one of the main challenges is the model evaluation.
Manually labeling the corresponding points for evaluation is time-consuming,
laborious and inaccurate, because the image size is typically large especially
for ultrasound images with low signal-to-noise ratio. We can use segmentation
accuracy to evaluate registration by conducting registration-based
segmentation, which employs the most similar training image based on NLCC as
the moving image and obtains segmentation prediction by transforming the
training label with the predicted registration field. For the segmentation, we
compare our method with NiftyReg [48], ANTs [8, 43], VoxelMorph [34] and
NeurReg [49] which uses simulated registration field to conduct supervised
learning. ANTs and NiftyReg are traditional optimization-based methods and
VoxelMorph is a deep learning-based unsupervised registration with the same
network structure and loss function as our method for a fair comparison. We
use Dice coefficient (DSC) as the evaluation metric, defined to be
$\frac{2\times TP}{2\times TP+FN+FP}$, where $TP$, $FN$, and $FP$ are true
positives, false negatives, false positives, respectively. Because the image
size is within $48\times 64\times 48$ voxels, we use a $5\times 5\times
5$-voxel window in the reconstruction loss $\mathcal{L}_{R}$ in Eq. (2). We
follow the same settings as NeurReg and the performances of the compared
methods are reported from [49].

For the motion analysis, we use the last frame as the moving image and the
current frame as the fixed image. Because there is no registration ground
truth for tissue tracking based on echocardiogram, we use reconstruction based
metrics, i.e., the mean square error (MSE) and the normalized local cross
correlation (NLCC) with radius of ten pixels to replace pixel position based
evaluation metric. We calculate the average MSE and NLCC over all frame pairs
with the linearly normalized pixel value of range $[0,1]$. For MSE and NLCC of
one pair of frames, we take the average of square error and normalized local
cross correlation over the masked region obtained from Section III-A. The
method with low MSE and high NLCC has good reconstruction and is the preferred
method. We compare our approach to ANTs [8, 43], and VoxelMorph [34]. We use a
$6\times 6$-voxel window for reconstruction loss $\mathcal{L}_{R}$. For a fair
comparison, we use the same network structure and number of channels in each
convolution as VoxelMorph.

For the purpose of ablation studies, we report the results of self-supervision
optimized registration (SSR), self-supervision optimized multi-scale
registration (SSMSR (1)) and different scales of self-supervision optimized
multi-scale registration (SSMSR ($s$)). We use two scales $1/2$ and $1$ on
hippocampus dataset and four different scales $1/8$, $1/4$, $1/2$ and $1$ on
echocardiogram datasets because echocardiogram is relative large ($1024\times
768$) and hippocampus image size is relative small ($48\times 64\times 48$).
We use learning rate of $1\times 10^{-3}$ and Adam optimizer to update the
weights in neural networks for both our method and VoxelMorph [45]. The
$\lambda$, which is the weight ratio of the smoothness loss and the
reconstruction loss, is set to be $10$ based on the performance on the
validation set. We set the number of optimization steps to $3,500$ per test
pair or ultrasound sequence for the self-supervision optimized registration
(of each scale), and set the number of iterations to $3,500\times$ the number
of training ultrasound sequences for VoxelMorph for fair comparison. To
improve the generalization performance of our method, we also firstly update
the weights on the training set the same as VoxelMorph. In the experiment, we
find, for self-supervision optimized multi-scale registration, the consecutive
optimization across scales, i.e., weights from scale $s$ using optimized
weights from scale $s/2$ as initialization, yields better registration.

We conduct deformable registration and use the recommended hyper-parameters
for ANTs and NiftyReg in [34], because the field of view in the hippocampus
and cardiac tissues is roughly aligned during image acquisition. For ANTs, we
use three scales with 200 iterations each, B-Spline SyN step size of 0.25,
updated field mesh size of five. For NiftyReg, we use three scales, the same
local negative cross correlation objective function with control point grid
spacing of five voxels and 500 iterations. We have tried our best to tune the
best hyper-parameters for ANTs and VoxelMorph.

###  III-C Registration-Based Segmentation

On the Hippocampus dataset, the image size is within $48\times 64\times 48$
voxels, which is small. We only use two scales for multi-scale registration.
For ablative study, we calculate DSCs of test-time training, self-supervision
optimized, registration without multi-scale scheme (SSR), self-supervision
optimized multi-scale registration of scale 1/2 (SSMSR (1/2)) and final self-
supervision optimized multi-scale registration (SSMSR (1)) in Table I. We
compare our method with previous registration approaches including ANTs,
NiftyReg and recently proposed NeurReg [49]. We follow the same experimental
settings as NeurReg. The results of ANTs, NiftyReg, VoxelMorph and NeurReg are
reported in [49]. We use only one atlas for each test image based on NLCC
similarity score from VoxelMorph. The segmentation comparisons are listed in
Table I. With the same set of atlases, self-supervision optimized registration
yields better segmentation than VoxelMorph, which means that the alleviation
of domain shift by self-supervised optimization improves the accuracy of
registration-based segmentation. Test-time training by self-supervision
optimized multi-scale registration further improves registration-based
segmentation accuracy based on segmentation dice score on both classes.

TABLE I: Segmentation (Dice scores, %) on hippocampus dataset. Method | hippocampus head/body | Average $\uparrow$
---|---|---
ANTs | 80.86$\pm$5.13/78.34$\pm$5.24 | 79.60
NiftyReg | 80.53$\pm$4.86/77.92$\pm$5.47 | 79.23
VoxelMorph |  | 78.92$\pm$6.79/75.85$\pm$8.13
---
77.39
NeurReg |  | 78.99$\pm$6.24/78.72$\pm$7.47
---
78.86
| SSR (w/o multi-scale)
---
| 80.61$\pm$5.76/78.73$\pm$8.14
---
79.67
| SSMSR (1/2) (coarsest scale)
---
| 77.66$\pm$4.90/75.08$\pm$8.01
---
76.37
SSMSR (1) |  | 81.18$\pm$5.06/79.13$\pm$7.04
---
80.16

###  III-D Registration-Based Tissue Dense Tracking

To validate the effectiveness of our method on noisy and large deformation
ultrasound images, we calculate the performances of registration fields from
ANTs, VoxelMorph, test-time training by self-supervision optimized
registration (SSR) and self-supervision optimized multi-scale registrations
(SSMSR ($s$)). Quantitative comparison results of these models on both
myocardial and cardiac blood flow dense tracking are shown in Table II.

TABLE II: Comparisons on myocardial (Upper) and cardiac blood flow (Lower) dense tracking among ANTs, VoxelMorph and Ours.  Methods | MSE (10-3) $\downarrow$ | NLCC (10-1) $\uparrow$
---|---|---
ANTs | 15.5179$\pm$9.4637 | 3.1597$\pm$1.3913
VoxelMorph | 1.2266$\pm$0.5457 | 4.7363$\pm$0.5457
SSR (w/o multi-scale) | 1.2158$\pm$0.5473 | 4.7809$\pm$0.4952
SSMSR (1/8) (coarest scale) | 1.6937$\pm$0.5616 | 4.2084$\pm$0.5086
SSMSR (1/4) | 1.3504$\pm$0.4885 | 4.4140$\pm$0.5005
SSMSR (1/2) | 1.0929$\pm$0.3775 | 4.7155$\pm$0.4569
SSMSR (1) | 0.9206$\pm$0.3236 | 5.0881$\pm$0.4107
ANTs | 3.9344$\pm$1.4660 | 4.2062$\pm$1.0576
VoxelMorph | 5.8654$\pm$1.6943 | 3.3462$\pm$0.5891
SSR (w/o multi-scale) | 5.7780$\pm$1.6222 | 3.3942$\pm$0.5868
SSMSR (1/8) (coarest scale) | 6.8108$\pm$1.8189 | 2.3948$\pm$0.5495
SSMSR (1/4) | 5.6106$\pm$1.2424 | 2.7331$\pm$0.6779
SSMSR (1/2) | 4.5089$\pm$1.0272 | 3.3879$\pm$0.7524
SSMSR (1) | 3.7944$\pm$0.9384 | 4.2519$\pm$0.7181

From Table II, we highlight the following observations. 1) SSMSR (1) achieves
the best performances, and outperforms ANTs in terms of both MSE and NLCC on
both myocardial and cardiac blood tracking, likely due to the representation
and optimization efficiency of deep neural nets; Deep learning has a good
capacity to model the registration field. 2) SSR yields consistently better
results than VoxelMorph, demonstrating the efficacy of self-supervised
optimization during the test phase for improving registration field estimation
and reducing the estimation gap between training and testing. 3) SSMSR (1)
obtains better performance than SSR on all experiments, demonstrating the
benefit of sequential multi-scale optimization in echocardiogram registration.
The multi-scale scheme alleviates the over-optimization of reconstruction loss
compared with ANTs, which can be visually noticed from Fig. 3 and 4 in Section
IV-B. The multi-scale registration with consecutive test-time training, SSMSR
(1), achieves the best performance on the two tasks and outperforms
traditional registration, ANTs, and current unsupervised registration,
VoxelMorph, on the two tasks based on the two metrics.

##  IV Discussion

###  IV-A Understand Test-Time Training

To further understand the importance of test-time training in each scale of
multi-scale registration, we visualize the reconstructions of cardiac blood
from test-time training based multi-scale registration (scale 1/8) with
(second column) and without (first column) test-time training respectively in
Fig. 2. From the visual comparison, multi-scale registration without test-time
training (first column) reconstructs worse blood details than multi-scale
registration with test-time training (second column). In the multi-scale
registration framework, the current registration relies on the previous
reconstructed image which makes the optimization more difficult if the
reconstruction from coarse level cannot do well.

Figure 2: Reconstruction comparison for cardiac blood from test-time training
based multi-scale registration (scale 1/8) w/o (first), i.e., VoxelMorph [34],
w/ (middle) test-time training and ground truth image (last column).

###  IV-B Understand Multi-Scale Registration

To further understand the self-supervision optimized multi-scale registration
(SSMSR) in the test-time, we randomly choose two neighbor frames from
ultrasound sequences and visualize the myocardial tracking results based on
ANTs, VoxelMorph and the intermediate registration fields from SSMSR with four
different scales in Fig. 3. More visualization results can be found in the
supplemental materials.

Figure 3: Visualization of myocardial tracking based on ANTs, VoxelMorph [34]
and Ours.

From Fig. 3, we note that the registration field from ANTs is noisy, and the
velocity direction from VoxelMorph for the right myocardial is incorrect
because of contradictory in the estimated direction of myocardial. By
contrast, SSMSR (1/8) produces the smoothest registration field, and SSMSR (1)
generates more detailed velocity estimation that preserves both large and low-
scale velocity variations. The coarse-to-fine results illustrate the multi-
scale optimization scheme coupled with deep neural nets can be very effective
in dealing with the highly challenging case of image registration in
echocardiograms.

To facilitate the understanding of proposed SSMSR in the test-time on dense
blood flow tracking based on echocardiograms, we also visualize the cardiac
blood flow tracking results from ANTs, VoxelMorph and the intermediate
registration fields from SSMSR with four different scales in Fig. 4. We
randomly choose two neighbor frames from these ultrasound images.

Figure 4: Visualization of cardiac blood flow tracking based on ANTs,
VoxelMorph [34] and Ours.

From the registration fields generated from ANTs and VoxelMorph in Fig. 4, we
cannot easily recognize the vortex in cardiac blood flow. By contrast, the
vortex flow pattern from SSMSR is readily recognizable. The general vortex
pattern is apparent from the coarsest level registration by SSMSR (1/8),
followed by finer-scale registrations to introduce details of local velocity
field variations. The final velocity field produced by SSMSR (1) includes both
easily recognizable vortex flow, as well as details of local field variations.

###  IV-C Computational Cost

We compare the computation cost on the echocardiogram dataset. NiftyReg takes
the same scale of time as ANTs as validated in [49]. For ANTs, we cannot find
implementation on GPU and the average computational time is 214.10$\pm$54.04
seconds for the registration of two consecutive frames on 12 processors of
Intel i7-6850K CPU @ 3.60GHz. For an ultrasound sequence of 50 frames, the
computational time is about three hours for ANTs. Because the inference of
VoxelMorph only relies on one feed forward pass of deep neural network, the
average computational time is 0.11$\pm$0.47 seconds for one pair frames on one
NVIDIA 1080 Ti GPU. The test-time training based multi-scale registration
takes 279.97, 101.65, 68.79, 66.09 seconds for self-supervision optimization
with the scale 1, 1/2, 1/4, 1/8 respectively on one ultrasound sequence of 49
frames by one NVIDIA 1080 Ti GPU. The SSMSR takes less than nine minutes in
test time in total for one ultrasound sequence, achieving 20 times speedup
over ANTs even using four scales.

##  V Conclusions

In this work, we propose a novel framework, test-time training based multi-
scale registration, as a general framework for image deformable registration.
To produce accurate registration field estimation from noisy medical images
and reduce the estimation gap between training and testing, we incorporate
test-time training in the registration framework. To handle large variations
of registration fields, a multi-scale scheme is integrated into the proposed
framework to reduce the over-optimization of similarity functions and provides
a sequential residual optimization pathway which alleviates the optimization
difficulties in registration. Our proposed method consistently outperforms
previous approaches on both registration-based segmentation task of 3D MR
images and myocardial and cardiac blood flow dense tracking task of
echocardiograms.

In future work, the current framework can be extended by: 1) forcing the
network to generate consistent displacement fields between moving and fixed
images [29], 2) adapting iterative module and improving the efficiency by
predicting the final registration field directly [23], 3) integrating
registration into segmentation facilitated by the smart strategy for large GPU
memory consumption [50, 51, 49].

## References

  * [1] G. Haskins, U. Kruger, and P. Yan, “Deep learning in medical image registration: A survey,” _arXiv preprint arXiv:1903.02026_ , 2019.
  * [2] R. Bajcsy and S. Kovačič, “Multiresolution elastic matching,” _Computer vision, graphics, and image processing_ , vol. 46, no. 1, pp. 1–21, 1989.
  * [3] D. Shen and C. Davatzikos, “Hammer: hierarchical attribute matching mechanism for elastic registration.” _IEEE transactions on medical imaging_ , vol. 21, no. 11, p. 1421, 2002.
  * [4] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and D. J. Hawkes, “Nonrigid registration using free-form deformations: application to breast mr images,” _IEEE transactions on medical imaging_ , vol. 18, no. 8, pp. 712–721, 1999.
  * [5] J.-P. Thirion, “Image matching as a diffusion process: an analogy with maxwell’s demons,” _Medical image analysis_ , vol. 2, no. 3, pp. 243–260, 1998.
  * [6] J. Ashburner _et al._ , “Voxel-based morphometry—the methods,” _Neuroimage_ , 2000.
  * [7] M. F. Beg, M. I. Miller, A. Trouvé, and L. Younes, “Computing large deformation metric mappings via geodesic flows of diffeomorphisms,” _International journal of computer vision_ , vol. 61, no. 2, pp. 139–157, 2005.
  * [8] B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain,” _Medical image analysis_ , vol. 12, no. 1, pp. 26–41, 2008.
  * [9] J. Ashburner, “A fast diffeomorphic image registration algorithm,” _Neuroimage_ , vol. 38, no. 1, pp. 95–113, 2007.
  * [10] M.-M. Rohé, M. Datar, T. Heimann, M. Sermesant, and X. Pennec, “Svf-net: Learning deformable image registration using shape matching,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 266–274.
  * [11] H. Sokooti, B. de Vos, F. Berendsen, B. P. Lelieveldt, I. Išgum, and M. Staring, “Nonrigid image registration using multi-scale 3d convolutional neural networks,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 232–239.
  * [12] X. Yang, R. Kwitt, M. Styner, and M. Niethammer, “Quicksilver: Fast predictive image registration–a deep learning approach,” _NeuroImage_ , vol. 158, pp. 378–396, 2017.
  * [13] G. Wu, M. Kim, Q. Wang, Y. Gao, S. Liao, and D. Shen, “Unsupervised deep feature learning for deformable registration of mr brain images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2013, pp. 649–656.
  * [14] M. Blendowski and M. P. Heinrich, “Combining mrf-based deformable registration and deep binary 3d-cnn descriptors for large lung motion estimation in copd patients,” _International journal of computer assisted radiology and surgery_ , vol. 14, no. 1, pp. 43–52, 2019.
  * [15] Y. Hu, R. Song, and Y. Li, “Efficient coarse-to-fine patchmatch for large displacement optical flow,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5704–5712.
  * [16] R. Liao, S. Miao, P. de Tournemire, S. Grbic, A. Kamen, T. Mansi, and D. Comaniciu, “An artificial agent for robust image registration,” in _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017.
  * [17] K. Ma, J. Wang, V. Singh, B. Tamersoy, Y.-J. Chang, A. Wimmer, and T. Chen, “Multimodal image registration with deep context reinforcement learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 240–248.
  * [18] S. Miao, S. Piat, P. Fischer, A. Tuysuzoglu, P. Mewes, T. Mansi, and R. Liao, “Dilated fcn for multi-agent 2d/3d medical image registration,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [19] J. Krebs, T. Mansi, H. Delingette, L. Zhang, F. C. Ghesu, S. Miao, A. K. Maier, N. Ayache, R. Liao, and A. Kamen, “Robust non-rigid registration through agent-based action learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 344–352.
  * [20] E. Chee and J. Wu, “Airnet: Self-supervised affine registration for 3d medical images using neural networks,” _arXiv preprint arXiv:1810.02583_ , 2018.
  * [21] X. Cao, J. Yang, J. Zhang, D. Nie, M. Kim, Q. Wang, and D. Shen, “Deformable image registration based on similarity-steered cnn regression,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 300–308.
  * [22] H. Uzunova, M. Wilms, H. Handels, and J. Ehrhardt, “Training cnns for image registration from few samples with model-based data augmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 223–231.
  * [23] J. Hur and S. Roth, “Iterative residual refinement for joint optical flow and occlusion estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5754–5763.
  * [24] M. Jaderberg, K. Simonyan, A. Zisserman _et al._ , “Spatial transformer networks,” in _Advances in neural information processing systems_ , 2015, pp. 2017–2025.
  * [25] T. C. Mok and A. C. Chung, “Large deformation diffeomorphic image registration with laplacian pyramid networks,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 211–221.
  * [26] J. Neylon, Y. Min, D. A. Low, and A. Santhanam, “A neural network approach for fast, automated quantification of dir performance,” _Medical physics_ , vol. 44, no. 8, pp. 4126–4138, 2017.
  * [27] H. Li and Y. Fan, “Non-rigid image registration using self-supervised fully convolutional networks without training data,” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 1075–1078.
  * [28] A. Sheikhjafari, M. Noga, K. Punithakumar, and N. Ray, “Unsupervised deformable image registration with fully connected generative neural network,” in _International conference on Medical Imaging with Deep Learning_ , 2018.
  * [29] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 270–279.
  * [30] J. Fan, X. Cao, Z. Xue, P.-T. Yap, and D. Shen, “Adversarial similarity network for evaluating image alignment in deep learning based registration,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 739–746.
  * [31] P. Jiang and J. A. Shackleford, “Cnn driven sparse multi-level b-spline image registration,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9281–9289.
  * [32] B. D. de Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, and I. Išgum, “A deep learning framework for unsupervised affine and deformable image registration,” _Medical image analysis_ , vol. 52, pp. 128–143, 2019.
  * [33] S. Zhao, Y. Dong, E. I. Chang, Y. Xu _et al._ , “Recursive cascaded networks for unsupervised medical image registration,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 10 600–10 610.
  * [34] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca, “An unsupervised learning model for deformable medical image registration,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9252–9260.
  * [35] B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. Išgum, “End-to-end unsupervised deformable image registration with a convolutional neural network,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 2017, pp. 204–212.
  * [36] A. V. Dalca, G. Balakrishnan, J. Guttag, and M. R. Sabuncu, “Unsupervised learning for fast probabilistic diffeomorphic registration,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 729–738.
  * [37] W. Zhu, X. Xiang, T. D. Tran, G. D. Hager, and X. Xie, “Adversarial deep structured nets for mass segmentation from mammograms,” in _2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)_. IEEE, 2018, pp. 847–850.
  * [38] L. Shen, W. Zhu, X. Wang, L. Xing, J. M. Pauly, B. Turkbey, S. A. Harmon, T. H. Sanford, S. Mehralivand, P. Choyke _et al._ , “Multi-domain image completion for random missing input data,” _IEEE Transactions on Medical Imaging_ , 2020.
  * [39] Y. Huang, W. Zhu, D. Xiong, Y. Zhang, C. Hu, and F. Xu, “Cycle-consistent adversarial autoencoders for unsupervised text style transfer,” in _Proceedings of the 28th International Conference on Computational Linguistics_ , 2020, pp. 2213–2223.
  * [40] Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt, “Test-time training for out-of-distribution generalization,” _Proceedings of the International Conference on Machine Learning_ , 2020.
  * [41] A. H. Curiale, G. Vegas-Sánchez-Ferrero, and S. Aja-Fernández, “Influence of ultrasound speckle tracking strategies for motion and strain estimation,” _Medical image analysis_ , vol. 32, pp. 184–200, 2016.
  * [42] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [43] B. B. Avants, N. Tustison, and G. Song, “Advanced normalization tools (ants),” _Insight j_ , vol. 2, pp. 1–35, 2009.
  * [44] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca, “Voxelmorph: a learning framework for deformable medical image registration,” _IEEE transactions on medical imaging_ , 2019.
  * [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [46] A. L. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. van Ginneken, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze _et al._ , “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” _arXiv preprint arXiv:1902.09063_ , 2019.
  * [47] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,” _International journal of computer vision_ , vol. 1, no. 4, pp. 321–331, 1988\.
  * [48] M. Modat, G. R. Ridgway, Z. A. Taylor, M. Lehmann, J. Barnes, D. J. Hawkes, N. C. Fox, and S. Ourselin, “Fast free-form deformation using graphics processing units,” _Computer methods and programs in biomedicine_ , vol. 98, no. 3, pp. 278–284, 2010.
  * [49] W. Zhu, A. Myronenko, Z. Xu, W. Li, H. Roth, Y. Huang, F. Milletari, and D. Xu, “Neurreg: Neural registration and its application to image segmentation,” in _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2020.
  * [50] W. Zhu, Y. Huang, L. Zeng, X. Chen, Y. Liu, Z. Qian, N. Du, W. Fan, and X. Xie, “Anatomynet: Deep learning for fast and fully automated whole-volume segmentation of head and neck anatomy,” _Medical physics_ , vol. 46, no. 2, pp. 576–589, 2019.
  * [51] W. Zhu, C. Zhao, W. Li, H. Roth, Z. Xu, and D. Xu, “Lamp: Large deep nets with automated model parallelism for image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 374–384.