# GMC - Geometric Multimodal Contrastive Representation Learning

Petra Poklukar  Miguel Vasco  Hang Yin  Francisco S. Melo  Ana Paiva  Danica
Kragic

###### Abstract

Learning representations of multimodal data that are both informative and
robust to missing modalities at test time remains a challenging problem due to
the inherent heterogeneity of data obtained from different channels. To
address it, we present a novel Geometric Multimodal Contrastive (GMC)
representation learning method comprised of two main components: i) a two-
level architecture consisting of _modality-specific_ base encoder, allowing to
process an arbitrary number of modalities to an intermediate representation of
fixed dimensionality, and a _shared_ projection head, mapping the intermediate
representations to a latent representation space; ii) a multimodal contrastive
loss function that encourages the geometric alignment of the learned
representations. We experimentally demonstrate that GMC representations are
semantically rich and achieve state-of-the-art performance with missing
modality information on three different learning problems including prediction
and reinforcement learning tasks.

Multimodal, Representation Learning, Contrastive Learning

##  1 Introduction

Information regarding objects or environments in the world can be recorded in
the form of signals of different nature. These different _modality_ signals
can be for instance images, videos, sounds or text, and represent the same
underlying phenomena. Naturally, the performance of machine learning models
can be enhanced by leveraging the redundant and complementary information
provided by multiple modalities (Baltrušaitis et al., 2018). In particular,
exploiting such multimodal information has been shown to be successful in
tasks such as classification (Tsai et al., 2018, 2019), generation (Wu &
Goodman, 2018; Shi et al., 2019) and control (Silva et al., 2019; Vasco et
al., 2021).

Figure 1: Overview of the _Geometric Multimodal Contrastive_ (GMC) method to
learn aligned modality-specific and complete representations of multimodal
data: the _modality-specific_ base encoders $f(\cdot)$ map observations to
intermediate representations $h$ that are further projected with a _shared_
projection head $g(\cdot)$ to a latent representation space where we apply a
novel multimodal contrastive loss.

The advances of many of these methods can be attributed to the efficient
learning of multimodal data representations, which reduces the inherent
complexity of raw multimodal data and enables the extraction of the underlying
semantic correlations among the different modalities (Baltrušaitis et al.,
2018; Guo et al., 2019). Generally, good representations of multimodal data
_i) capture the semantics_ from individual modalities necessary for performing
a given downstream task. Additionally, in scenarios such as real-world
classification and control, it is essential that the obtained representations
are _ii) robust to missing modality_ information during execution (Meo &
Lanillos, 2021; Tremblay et al., 2021; Zambelli et al., 2020). In order to
fulfill _i)_ and _ii)_ , the unique characteristics of each modality need to
be processed accordingly and efficiently combined, which remains a challenging
problem known as the _heterogeneity gap_ in multimodal representation learning
(Guo et al., 2019).

(a) MVAE (Wu & Goodman, 2018)

(b) MMVAE (Shi et al., 2019)

(c) Nexus (Vasco et al., 2022)

(d) MUSE (Vasco et al., 2021)

(e) MFM (Tsai et al., 2018)

(f) GMC (Ours)

Figure 2: UMAP visualization of complete representations $z_{1:4}$ (blue) and
image representations $z_{1}$ (orange) in a latent space $z\in\mathbb{R}^{64}$
obtained from several state-of-the-art multimodal representation learning
models on the MHD dataset considered in Section 5.1. Only GMC is able to learn
_modality-specific_ and _complete_ representations that are geometrically
aligned. More visualizations in Appendix F.

An intuitive idea to mitigate the heterogeneity gap is to project
heterogeneous data into a shared representation space such that the
representations of complete observations capture the semantic content shared
across all modalities. In this regard, two directions have shown promise,
namely, generation-based methods commonly extending the Variational
Autoencoder (VAE) framework (Kingma & Welling, 2014) to multimodal data such
as MVAE (Wu & Goodman, 2018) and MMVAE (Shi et al., 2019), as well as methods
relying on the fusion of modality-specific representations such as MFM (Tsai
et al., 2018) and the Multimodal Transformer (Tsai et al., 2019). Fusion based
methods by construction fulfill objective i) but typically do not provide a
mechanism to cope with missing modalities. While this is better accounted for
in the generation based methods, these approaches often struggle to align
complete and modality-specific representations due to the demanding
reconstruction objective. We thoroughly discuss the geometric misalignment of
these methods in Section 2.

In this work, we learn _geometrically aligned_ multimodal data representations
that provide robust performance in downstream tasks under missing modalities
at test time. To this end, we present the Geometric Multimodal Contrastive
(GMC) representation learning framework, shown in Figure 1. Inspired by the
recently proposed Normalized Temperature-scaled Cross Entropy (NT-XEnt) loss
in visual contrastive representation learning (Chen et al., 2020), we
contribute a novel multimodal contrastive loss that explicitly aligns
modality-specific representations with the representations obtained from the
corresponding complete observation. GMC assumes a two-level neural-network
model architecture consisting of a collection of _modality-specific_ base
encoders, processing modality data into an intermediate representation of a
fixed dimensionality, and a _shared_ projection head, mapping the intermediate
representations into a latent representation space where the contrastive
learning objective is applied. It can be scaled to an arbitrary number of
modalities, and provides semantically rich representations that are robust to
missing modality information. Furthermore, as shown in our experiments, GMC is
general as it can be integrated into existing models and applied to a variety
of challenging problems, such as learning representations in an unsupervised
manner (Section 5.1), for prediction tasks using a weak supervision signal
(Section 5.2) or downstream reinforcement learning tasks (Section 5.3). We
show that GMC is able to achieve state-of-the-art performance with missing
modality information compared to existing models.

##  2 The Problem of Geometric Misalignment in Multimodal Representation
Learning

We consider scenarios where information is provided in the form of a dataset
$X$ of $N$ tuples, i.e.,
$X=\\{x_{1:M}^{i}=(x_{1}^{i},\dots,x_{M}^{i})\\}_{i=1}^{N}$, where each tuple
$x_{1:M}=(x_{1},\dots,x_{M})$ represents observations provided by $M$
different modalities. We refer to the tuples $x_{1:M}$ consisting of all $M$
modalities as complete observations and to the single observations $x_{m}$ as
modality-specific. The goal is to learn complete representations $z_{1:M}$ of
$x_{1:M}$ and modality-specific representations $\\{z_{1},\dots,z_{M}\\}$ of
$\\{x_{1},\dots,x_{M}\\}$ that are:

  * i)

informative, i.e., both $z_{1:M}$ and any of $\\{z_{1},\dots,z_{M}\\}$
contains relevant semantic information for some downstream task, and thus,

  * ii)

robust to missing modalities during test time, i.e., the success of a
subsequent downstream task is independent of whether the provided input is the
complete representation $z_{1:M}$ or any of the modality-specific
representations $\\{z_{1},\dots,z_{M}\\}$.

Prior work has demonstrated success in using complete representations
$z_{1:M}$ in a diverse set of applications, such as image generation (Wu &
Goodman, 2018; Shi et al., 2019) and control of Atari games (Silva et al.,
2019; Vasco et al., 2021). Intuitively, if complete representations $z_{1:M}$
are sufficient to perform a downstream task then learning modality-specific
representations $z_{m}$ that are geometrically aligned with $z_{1:M}$ in the
same representation space should ensure that $\\{z_{m}\\}$ contain necessary
information to perform the task even when $z_{1:M}$ cannot be provided.
Therefore, in Section 5 we study the geometric alignment of $z_{1:M}$ and each
$z_{m}$ on several multimodal datasets and state-of-the-art multimodal
representation models. In Figure 2, we visualize an example of encodings of
$z_{1:M}$ (in blue) and $z_{m}$ corresponding to the image modality (in
orange) where we see that the existing approaches produce geometrically
misaligned representations. As we empirically show in Section 5.1, this
misalignment is consistent across different learning scenarios and datasets,
and can lead to a poor performance on downstream tasks.

To fulfill i) and ii), we propose a novel approach that builds upon the simple
idea of geometrically aligning modality-specific representations $z_{m}$ with
the corresponding complete representations $z_{1:M}$ in a latent
representation space, framing it as a contrastive learning problem.

##  3 Geometric Multimodal Contrastive Learning

We present the _Geometric Multimodal Contrastive_ (GMC) framework, visualized
in Figure 1, consisting of three main components:

  * •

A collection of neural network _modality-specific_ base encoders
$f(\cdot)=\\{f_{1:M}(\cdot)\\}\cup\\{f_{1}(\cdot),\dots,f_{M}(\cdot)\\}$,
where $f_{1:M}(\cdot)$ and $f_{m}(\cdot)$ take as input the complete $x_{1:M}$
and modality-specific observations $x_{m}$, respectively, and output
intermediate $d$-dimensional representations
$\\{h_{1:M},h_{1},\dots,h_{M}\\}\in\mathbb{R}^{d}$;

  * •

A neural network _shared_ projection head $g(\cdot)$ that maps the
intermediate representations given by the base encoders $f(\cdot)$ to the
latent representations $\\{z_{1:M},z_{1},\dots,z_{M}\\}\in\mathbb{R}^{s}$ over
which we apply the contrastive term. The projection head $g(\cdot)$ enables to
encode the intermediate representations in a shared representation space while
preserving modality-specific semantics;

  * •

A multimodal contrastive NT-Xent loss function (_mNT-Xent_), defined below,
that is inspired by the recently proposed SimCLR framework (Chen et al., 2020)
and encourages the geometric alignment of $z_{m}$ and $z_{1:M}$.

We phrase the problem of geometrically aligning $z_{m}$ with $z_{1:M}$ as a
contrastive prediction task where the goal is to identify $z_{m}$ and its
corresponding complete representation $z_{1:M}$ in a given mini-batch. Let
$\mathcal{B}=\\{z_{1:M}^{i}\\}_{i=1}^{B}\subset g(f(X))$ be a mini-batch of
$B$ complete representations. Let $\operatorname{\text{sim}}(u,v)$ denote the
cosine similarity among vectors $u$ and $v$ and let $\tau\in(0,\infty)$ be the
temperature hyperparameter. We denote by

$\displaystyle
s_{m,n}(i,j)=\exp(\operatorname{\text{sim}}(z_{m}^{i},z_{n}^{j})/\tau),$ (1)

the similarity between representations $z_{m}^{i}$ and $z_{n}^{j}$ (modality-
specific or complete) corresponding to the $i$th and $j$th samples from the
mini-batch $\mathcal{B}$. For a given modality $m$, we define positive pairs
as $(z_{m}^{i},z_{1:M}^{i})$ for $i=1,\dots,B$ and treat the remaining pairs
as negative ones. In particular, we denote by

$\displaystyle\Omega_{m}(i)=\sum_{i\neq
j}(s_{m,1:M}(i,j)+s_{m,m}(i,j)+s_{1:M,1:M}(i,j)),$

the sum of similarities among negative pairs that correspond to the positive
pair $(z_{m}^{i},z_{1:M}^{i})$, and define the contrastive loss for the same
pair of samples as

$\displaystyle l_{m}(i)=-\log\frac{s_{m,1:M}(i,i)}{\Omega_{m}(i)}.$

Lastly, we combine the loss terms for each modality $m=1,\dots,M$ and obtain
the final mNT-Xent training loss

$\displaystyle\mathcal{L}_{\text{mNT-
Xent}}(\mathcal{B})=\sum_{m=1}^{M}\sum_{i=1}^{B}l_{m}(i).$ (2)

As we only contrast single modality-specific representations to the complete
ones, $\mathcal{L}_{\text{mNT-Xent}}$ scales linearly to an arbitrary number
of modalities. In Section 5, we show that $\mathcal{L}_{\text{mNT-Xent}}$ can
be added as an additional term to existing frameworks to improve their
robustness to missing modalities. Moreover, we experimentally demonstrate that
the architectures of the base encoders and shared projection head can be
flexibly adjusted depending on the task.

##  4 Related Work

Learning multimodal representations suitable for downstream tasks has been
extensively addressed in literature (Baltrušaitis et al., 2018; Guo et al.,
2019). In this work, we focus on the problem of aligning modality-specific
representations in a (shared) latent space emerging from the heterogeneity gap
between different data sources. Prior work promoting such alignment can be
separated into two groups: generation-based methods adjusting Variational
Autoencoder (VAE) (Kingma & Welling, 2013) frameworks that considers a prior
distribution over the shared latent space, and fusion-based methods that merge
modality-specific representations into a shared representation.

Generation-based methods Associative VAE (AVAE) (Yin et al., 2017) and Joint
Multimodal VAE (JMVAE) (Suzuki et al., 2016) explicitly enforce the alignment
of modality-specific representations by minimizing the Kullback–Leibler
divergence between their distributions. However, these models are not easily
scalable to large number of modalities due to the combinatorial increase of
inference networks required to account for all subsets of modalities. In
contrast, GMC scales linearly with the number of modalities as it separately
contrasts individual modality-specific representations to complete ones.

Other multimodal VAE models promote the approximation of modality-specific
representations through dedicated training schemes. MVAE (Wu & Goodman, 2018)
uses sub-sampling to learn a joint-modality representation obtained from a
Product-of-Experts (PoE) inference network. This solution is prone to learning
overconfident experts, hindering both the alignment of the modality-specific
representations and the performance of downstream tasks under incomplete
information (Shi et al., 2019). Mixture-of-Experts MVAE (MMVAE) (Shi et al.,
2019) instead employs a doubly reparameterized gradient estimator which is
computationally expensive compared to the lower-bound objective of traditional
multimodal VAEs because of its Monte-Carlo-based training scheme. GMC, on the
other hand, presents an efficient training scheme without suffering from
modality-specific biases.

Recently, hierarchical multimodal VAEs have been proposed to facilitate the
learning of aligned multimodal representations such as Nexus (Vasco et al.,
2022) and Multimodal Sensing (MUSE) (Vasco et al., 2021). Nexus considers a
two-level hierarchy of modality-specific and multimodal representation spaces
employing a dropout-based training scheme. The average aggregator solution
employed to merge multimodal information lacks expressiveness which hinders
the performance of the model on downstream tasks. To address this issue, MUSE
introduces a PoE solution that merges lower-level modality-specific
information to encode a high-level multimodal representation, and a dedicated
training scheme to counter the overconfident expert issue. In contrast to both
solutions, GMC is computationally efficient without requiring hierarchy.

Fusion-based methods Other class of methods approach the alignment of
modality-specific representations through complex fusion mechanisms (Liang et
al., 2021). The Multimodal Factorized model (MFM) (Tsai et al., 2018) proposes
the factorization of a multimodal representation into distinct multimodal
discriminative factors and modality-specific generative factors, which are
subsequently fused for downstream tasks. More recently, the Multimodal
Transformer model (Tsai et al., 2019) has shown remarkable classification
performance in multimodal time-series datasets, employing a directional
pairwise cross-modal attention mechanism to learn a rich representation of
heterogeneous data streams without requiring their explicit time-alignment. In
contrast to both models, GMC is able to learn multimodal representations of
modalities of arbitrary nature without explicitly requiring a supervision
signal (e.g. labels).

##  5 Experiments

Table 1: Performance of different multimodal representation methods in the MHD dataset, in a downstream classification task under complete and partial observations. Accuracy (%) results averaged over 5 independent runs. Higher is better. Input |  1MVAE1 | MMVAE |  1Nexus |  1MUSE |  1MFM |  1GMC (Ours)
---|---|---|---|---|---|---
Complete $(x_{1:4})$ | $100.0\pm 0.00$ | $99.81\pm\phantom{0}0.21$ | $99.98\pm 0.05$ | $99.99\pm 4\mathrm{e}{-5}$ | $100.0\pm 0.00$ | $100.0\pm 0.00$
Image $(x_{1})$ | $77.94\pm 3.16$ | $94.63\pm\phantom{0}2.61$ | $95.89\pm 0.34$ | $79.37\pm\phantom{0}2.75$ | $34.66\pm 6.48$ | $\bf 99.75\pm 0.03$
Sound $(x_{2})$ | $61.75\pm 4.59$ | $69.43\pm 26.43$ | $39.07\pm 5.82$ | $41.39\pm\phantom{0}0.18$ | $10.07\pm 0.20$ | $\bf 93.04\pm 0.45$
Trajectory $(x_{3})$ | $10.03\pm 0.06$ | $95.33\pm\phantom{0}2.56$ | $98.55\pm 0.34$ | $89.49\pm\phantom{0}2.44$ | $25.61\pm 5.41$ | $\bf 99.96\pm 0.02$
Label $(x_{4})$ | $100.0\pm 0.00$ | $87.99\pm\phantom{0}7.49$ | $100.0\pm 0.00$ | $100.0\pm\phantom{0}0.00$ | $100.0\pm 0.00$ | $100.0\pm 0.00$
Table 2: DCA score of the models in the MHD dataset, evaluating the geometric
alignment of complete representations $z_{1:4}$ and modality-specific ones
$\\{z_{1},\dots z_{4}\\}$ used as $R$ and $E$ inputs in DCA, respectively. The
score is averaged over 5 independent runs. Higher is better.

$R$ $E$ 1MVAE1 MMVAE 1Nexus 1MUSE 1MFM 1GMC (Ours) Complete $(z_{1:4})$ Image
$(z_{1})$ $0.01\pm 0.01$ $0.21\pm 0.29$ $0.00\pm 0.00$ $0.54\pm 0.44$ $0.00\pm
0.00$ $\bf 0.96\pm 0.02$ Complete $(z_{1:4})$ Sound $(z_{2})$ $0.00\pm 0.00$
$0.00\pm 0.00$ $0.00\pm 0.00$ $0.00\pm 0.00$ $0.00\pm 0.00$ $\bf 0.87\pm 0.16$
Complete $(z_{1:4})$ Trajectory $(z_{3})$ $0.00\pm 0.00$ $0.01\pm 0.01$
$0.08\pm 0.02$ $0.00\pm 0.00$ $0.00\pm 0.00$ $\bf 0.86\pm 0.05$ Complete
$(z_{1:4})$ Label $(z_{4})$ $0.99\pm 0.01$ $0.74\pm 0.22$ $0.43\pm 0.05$
$0.93\pm 0.05$ $0.85\pm 0.06$ $\bf 1.00\pm 0.00$

We evaluate the quality of the representations learned by GMC on three
different scenarios:

  * •

An _unsupervised learning_ problem, where we learn multimodal representations
on the Multimodal Handwritten Digits (MHD) dataset (Vasco et al., 2022). We
showcase the geometric alignment of representations and demonstrate the
superior performance of GMC compared to the baselines on a downstream
classification task with missing modalities (Section 5.1);

  * •

A supervised learning problem, where we demonstrate the flexibility of GMC by
integrating it into state-of-the-art approaches to provide robustness to
missing modalities in challenging classification scenarios (Section 5.2);

  * •

A reinforcement learning (RL) task, where we show that GMC produces general
representations that can be applied to solve downstream control tasks and
demonstrate state-of-the-art performance in actuation with missing modality
information (Section 5.3).

In each corresponding section, we describe the dataset, baselines, evaluation
and training setup used. We report all hyperparameters and architectures in
Appendix E and D. All results are averaged over $5$ different randomly-seeded
runs except for the RL experiments where we consider $10$ runs.

Evaluation of geometric alignment To evaluate the geometric alignment of
representations, we use a recently proposed _Delaunay Component Analysis_
(DCA) (Poklukar et al., 2022) method designed for general evaluation of
representations. DCA is based on the idea of comparing geometric and
topological properties of an evaluation set of representations $E$ with the
reference set $R$, acting as an approximation of the true underlying manifold.
The set $E$ is considered to be well aligned with $R$ if its global and local
structure resembles well the one captured by $R$, i.e., the manifolds
described by the two sets have similar number, structure and size of connected
components.

DCA approximates the manifolds described by $R$ and $E$ with a Delaunay
neighbourhood graph and derives several scores reflecting their alignment. We
consider three of them: _network quality_ $q\in[0,1]$ which measures the
overall geometric alignment of $R$ and $E$ in the connected components, as
well as _precision_ $\mathcal{P}\in[0,1]$ and _recall_ $\mathcal{R}\in[0,1]$
which measure the proportion of points from $E$ and $R$, respectively, that
are contained in geometrically well-aligned components. To account for all
three normalized scores, we report the harmonic mean defined as
$3/(1/\mathcal{P}+1/\mathcal{R}+1/q)$ when all $\mathcal{P},\mathcal{R},q>0$
and $0$ otherwise. In all experiments, we compute DCA using complete
representations $z_{1:M}$ as the reference set $R$ and modality-specific
$z_{m}$ as the evaluation set $E$, both obtained from testing observations. A
detailed description of the method and definition of the scores is found in
Appendix A.

###  5.1 Experiment 1: Unsupervised Learning

Datasets The MHD dataset is comprised of images ($x_{1}$), sounds ($x_{2}$),
motion trajectories ($x_{3}$) and label information ($x_{4}$) related to
handwriting digits. The authors collected $60,000$ $28\times 28$ greyscale
images per class as well as normalized $200$-dimensional representations of
trajectories and $128\times 32$-dimensional representations of audio. The
dataset is split into $50,000$ training and $10,000$ testing samples.

Table 3: Number of parameters (in millions) of the representation models
employed in the Multimodal Handwritten Digits dataset.

MVAE MMVAE Nexus MUSE MFM GMC (Ours) 9.3 9.0 12.9 9.9 9.4 2.9

Models We consider several generation-based and fusion-based state-of-the-art
multimodal representation methods: MVAE, MMVAE, Nexus, MUSE and MFM (see
Section 4 for a detailed description). For a fair comparison, when possible,
we employ the same encoder architectures and latent space dimensionality
across all baseline models, described in Appendix D. For GMC, we employ the
same modality-specific base encoders $f_{m}(\cdot)$ as the baselines with an
additional base encoder $f_{1:4}(\cdot)$ taking complete observations as
input. The shared projection head $g(\cdot)$ comprises of $3$ fully-connected
layers. We set the temperature $\tau=0.1$ and consider $64$-dimensional
intermediate and shared representation spaces, i.e.,
$h\in\mathbb{R}^{64},z\in\mathbb{R}^{64}$. We train all the models for 100
epochs using a learning rate of $10^{-3}$, employing the training schemes and
hyperparameters suggested by the authors (when available).

Table 4: Performance of different multimodal representation methods in the
CMU-MOSEI dataset, in a classification task under complete and partial
observations. Results averaged over 5 independent runs. Arrows indicate the
direction of improvement.

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.643\pm 0.019$ | $\bf 0.634\pm 0.008$
Cor ($\uparrow$) | $\bf 0.664\pm 0.004$ | $0.653\pm 0.004$
F1 ($\uparrow$) | $\bf 0.809\pm 0.003$ | $0.798\pm 0.008$
Acc ($\%,\uparrow$) | $\bf 80.75\pm 00.28$ | $79.73\pm 00.69$
(a) Complete Observations $(x_{1:3})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.805\pm 0.028$ | $\bf 0.712\pm 0.015$
Cor ($\uparrow$) | $0.427\pm 0.061$ | $\bf 0.590\pm 0.013$
F1 ($\uparrow$) | $0.713\pm 0.086$ | $\bf 0.779\pm 0.005$
Acc ($\%,\uparrow$) | $66.53\pm 09.86$ | $\bf 77.85\pm 00.36$
(b) Text Observations $(x_{1})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.873\pm 0.065$ | $\bf 0.837\pm 0.008$
Cor ($\uparrow$) | $0.090\pm 0.062$ | $\bf 0.256\pm 0.007$
F1 ($\uparrow$) | $0.622\pm 0.122$ | $\bf 0.676\pm 0.015$
Acc ($\%,\uparrow$) | $53.17\pm 09.47$ | $\bf 65.59\pm 00.62$
(c) Audio Observations $(x_{2})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $1.025\pm 0.164$ | $\bf 0.845\pm 0.010$
Cor ($\uparrow$) | $0.110\pm 0.060$ | $\bf 0.278\pm 0.011$
F1 ($\uparrow$) | $0.574\pm 0.095$ | $\bf 0.655\pm 0.003$
Acc ($\%,\uparrow$) | $44.33\pm 09.40$ | $\bf 65.02\pm 00.28$
(d) Video Observations $(x_{3})$

Evaluation We follow the established evaluation in the literature using
classification as a downstream task (Shi et al., 2019) and train a $10$-class
classifier neural network on complete representations
$z_{1:M}=g\left(h_{1:M}(x_{1:M})\right)$ from the training split (see Appendix
D for the exact architecture). The classifier is trained for 50 epochs using a
learning rate of $1\mathrm{e}{-3}$. We report the testing accuracy obtained
when the classifier is provided with both complete $z_{1:4}$ and modality-
specific representations $z_{m}$ as inputs.

Classification results The classification results are shown in Table
1111Results for the MVAE model averaged over 3 randomly-seeded runs due to
divergence during training in the remaining seeds.. While all the models
attain perfect accuracy on $x_{1:4}$ and $x_{4}$, we observe that GMC is the
only model that successfully performs the task when given only $x_{1},x_{2}$
or $x_{3}$ as input, significantly outperforming the baselines.

Table 5: Performance of different multimodal representation methods in the
CMU-MOSEI dataset, in a classification task under complete and partial
observations. Results averaged over 5 independent runs. Arrows indicate the
direction of improvement.

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.643\pm 0.019$ | $\bf 0.634\pm 0.008$
Cor ($\uparrow$) | $\bf 0.664\pm 0.004$ | $0.653\pm 0.004$
F1 ($\uparrow$) | $\bf 0.809\pm 0.003$ | $0.798\pm 0.008$
Acc ($\%,\uparrow$) | $\bf 80.75\pm 00.28$ | $79.73\pm 00.69$
(a) Complete Observations $(x_{1:3})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.805\pm 0.028$ | $\bf 0.712\pm 0.015$
Cor ($\uparrow$) | $0.427\pm 0.061$ | $\bf 0.590\pm 0.013$
F1 ($\uparrow$) | $0.713\pm 0.086$ | $\bf 0.779\pm 0.005$
Acc ($\%,\uparrow$) | $66.53\pm 09.86$ | $\bf 77.85\pm 00.36$
(b) Text Observations $(x_{1})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $0.873\pm 0.065$ | $\bf 0.837\pm 0.008$
Cor ($\uparrow$) | $0.090\pm 0.062$ | $\bf 0.256\pm 0.007$
F1 ($\uparrow$) | $0.622\pm 0.122$ | $\bf 0.676\pm 0.015$
Acc ($\%,\uparrow$) | $53.17\pm 09.47$ | $\bf 65.59\pm 00.62$
(c) Audio Observations $(x_{2})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $1.025\pm 0.164$ | $\bf 0.845\pm 0.010$
Cor ($\uparrow$) | $0.110\pm 0.060$ | $\bf 0.278\pm 0.011$
F1 ($\uparrow$) | $0.574\pm 0.095$ | $\bf 0.655\pm 0.003$
Acc ($\%,\uparrow$) | $44.33\pm 09.40$ | $\bf 65.02\pm 00.28$
(d) Video Observations $(x_{3})$

Geometric alignment To validate that the superior performance of GMC
originates from a better geometric alignment of representations, we evaluate
the testing representations obtained from all the models using DCA. For each
modality $m$, we compared the alignment of the evaluation set $E=\\{z_{m}\\}$
and the reference set $R=\\{z_{1:4}\\}$. The obtained DCA scores are shown in
Table 2 where we see that GMC outperforms all the considered baselines. For
some cases, we observe the obtained representations are completely misaligned
yielding $\mathcal{P}=\mathcal{R}=q=0$. While some of the baselines are to
some extend able to align $z_{1}$ and/or $z_{4}$ to $z_{1:4}$, GMC is the only
method that is able to align even the sound and trajectory representations,
$z_{2}$ and $z_{3}$, resulting in a superior classification performance.

We additionally validate the geometric alignment by visualizing 2-dimensional
UMAP projections (McInnes et al., 2018) of the representations $z$. In Figure
2 we show projections of $z_{1:4}$ and image representations $z_{1}$ obtained
using the considered models. We clearly see that GMC not only correctly aligns
$z_{1:4}$ and $z_{1}$ but also separates the representations in $10$ clusters.
Moreover, we can see that among the baselines only MMVAE and MUSE somewhat
align the representations which is on par with the quantitative results
reported in Table 2. For MVAE, Nexus and MFM, Figure 2 visually supports the
obtained DCA score $0$. Note that points marked as outliers by DCA are omited
from the visualization. We provide similar visualizations of other modalities
in Appendix F.

Model Complexity In Table 6 we present the number of parameters required by
the multimodal representation models employed in this task. The results show
that GMC requires significantly fewer parameters than the smallest baseline
model – 74% fewer parameters than MMVAE.

Table 6: Number of parameters (in millions) of the representation models
employed in the Multimodal Handwritten Digits dataset.

MVAE MMVAE Nexus MUSE MFM GMC (Ours) 9.3 9.0 12.9 9.9 9.4 2.9

Table 7: DCA score of the models in the CMU-MOSEI dataset evaluating the
geometric alignment of complete representations $z_{1:4}$ and modality-
specific ones $\\{z_{1},\dots z_{4}\\}$ used as $R$ and $E$ inputs in DCA,
respectively. The score is averaged over 5 independent runs. Higher is better.

R E Baseline GMC (Ours) Complete $(z_{1:3})$ Text $(z_{1})$ $0.50\pm 0.05$
$\bf 0.95\pm 0.01$ Complete $(z_{1:3})$ Audio $(z_{2})$ $0.41\pm 0.14$ $\bf
0.86\pm 0.04$ Complete $(z_{1:3})$ Vision $(z_{3})$ $0.50\pm 0.14$ $\bf
0.92\pm 0.02$

###  5.2 Experiment 2: Supervised Learning

In this section, we evaluate the flexibility of GMC by adjusting both the
architecture of the model and training procedure to receive an additional
supervision signal during training to guide learning of complete
representations. We demonstrate how GMC can be integrated into existing
approaches to provide additional robustness to missing modalities with minimal
computational cost.

Datasets We employ the CMU-MOSI (Zadeh et al., 2016) and CMU-MOSEI (Bagher
Zadeh et al., 2018), two popular datasets for sentiment analysis and emotion
recognition with challenging temporal dynamics. Both datasets consist of
textual ($x_{1}$), sound ($x_{2}$) and visual ($x_{3}$) modalities extracted
from videos. CMU-MOSI consists of 2199 short monologue videos clips of
subjects expressing opinions about various topics. CMU-MOSEI is an extension
of CMU-MOSI dataset containing 23453 YouTube video clips of subjects
expressing movie reviews. In both datasets, each video clip is annotated with
labels in $[-3,3]$, where ${-3}$ and $3$ indicate strong negative and strongly
positive sentiment scores, respectively. We employ the temporally-aligned
version of these datasets: CMU-MOSEI consists of $18134$ and $4643$ training
and testing samples, respectively, and CMU-MOSI consists of $1513$ and $686$
training and testing samples, respectively.

Models We consider the Multimodal Transformer (Tsai et al., 2019) which is the
state-of-the-art model for classification on the CMU-MOSI and CMU-MOSEI
datasets (see Appendix D for a detailed description of the architecture). For
GMC, we employ the same architecture for the joint-modality encoder
$f_{1:3}(\cdot)$ as the Multimodal Transformer but remove the last
classification layers. For the modality-specific base encoders
$\\{f_{1}(\cdot),f_{2}(\cdot),f_{3}(\cdot)\\}$, we employ a simple GRU layer
with 30 hidden units and a fully-connected layer. The shared projection head
$g(\cdot)$ is comprised of a single fully connected layer. We set $\tau=0.3$
and consider $60$-dimensional intermediate and shared representations
$h,z\in\mathbb{R}^{60}$.

In addition, we employ a simple classifier consisting of 2 linear layers over
the complete representations $z_{1:M}$ to provide the supervision signal to
the model during training. We follow the training scheme proposed by (Tsai et
al., 2019) and train all models for 40 epochs with a decaying learning rate of
$10^{-3}$.

Evaluation We evaluate the performance of representation models in sentiment
analysis classification with missing modality information. We evaluate same
metrics as in (Tsai et al., 2018, 2019) and report binary accuracy (Acc), mean
absolute error (MAE), correlation (Cor) and F1 score (F1) of the predictions
obtain on the test dataset. In Appendix C we present similar results on the
CMU-MOSI dataset.

Results The results obtained on CMU-MOSEI are reported in Table 5(d). When
using the complete observations $x_{1:3}$ as inputs, GMC achieves competitive
performance with the baseline model indicating that the additional contrastive
loss does not deteriorate the model’s capabilities (Table 5(a)). However, GMC
significantly improves the robustness of the model to the missing modalities
as seen in Tables 5(b), 5(c) and 5(d) where we use only individual modalities
as inputs. While GMC consistently outperforms the baseline in all metrics, we
observe the largest improvement on the F1 score and binary accuracy (Acc)
where the baseline often performs worse than random. As before, we
additionally evaluate the geometric

Table 8: Number of parameters (in millions) of the representation models employed in the CMU-MOSEI dataset. Baseline | GMC (Ours)
---|---
1.1 | 1.4

alignment of the modality-specific representations $z_{m}$ (comprising the set
$E$) and complete representations $z_{1:3}$ (comprising the set $R$). The
resulting DCA score, reported in Table 7, supports the results shown in Table
5(d) and verifies that GMC significantly improves the geometric alignment
compared to the baseline.

Model Complexity In Table 8 we report the complexity of the models and show
that employing GMC to provide robustness to the original framework incurs in a
small computational cost requiring only 300K extra parameters.

###  5.3 Experiment 3: Reinforcement Learning

Table 9: Performance after zero-shot policy transfer in the multimodal Pendulum task. At test time, the agent is provided with either image ($x_{1}$), sound ($x_{2}$), or complete ($x_{1:2}$) observations. Total reward averaged over 100 episodes and 10 randomly seeded runs. Higher is better. Observation | MVAE + DDPG | MUSE + DDPG2 | GMC + DDPG (Ours)
---|---|---|---
Complete $(x_{1:2})$ | $-1.114\pm 0.110$ | $-1.005\pm 0.117$ | $\bf-0.935\pm 0.057$
Image $(x_{1})$ | $-1.116\pm 0.121$ | $-4.752\pm 0.994$ | $\bf-0.940\pm 0.056$
Sound $(x_{2})$ | $-6.642\pm 0.106$ | $-3.459\pm 0.519$ | $\bf-0.956\pm 0.075$
Table 10: DCA score of the models in the multimodal Pendulum task evaluating the geometric alignment of complete representations $z_{1:2}$ and modality-specific ones $\\{z_{1},z_{2}\\}$ used as $R$ and $E$ inputs in DCA, respectively. Results averaged over 10 independent runs. Higher is better. $R$ | $E$ | MVAE + DDPG | MUSE + DDPG2 | GMC + DDPG (Ours)
---|---|---|---|---
Complete $(z_{1:2})$ | Image $(z_{1})$ | $0.79\pm 0.01$ | $0.20\pm 0.09$ | $\bf 0.87\pm 0.01$
Complete $(z_{1:2})$ | Sound $(z_{2})$ | $0.00\pm 0.00$ | $0.01\pm 0.01$ | $\bf 0.88\pm 0.02$

In this section, we demonstrate how GMC can be employed as a representation
model in the design of RL agents yielding state-of-the-art performance using
missing modality information during task execution.

Scenario We consider the recently proposed multimodal inverted Pendulum task
(Silva et al., 2019) which is an extension of the classical control scenario
to a multimodal setting. In this task, the goal is to swing the pendulum up so
it remains balanced upright. The observations of the environment include both
an image ($x_{1}$) and a sound ($x_{2}$) component. The sound component is
generated by the tip of the pendulum emitting a constant frequency $f_{0}$.
This frequency is received by a set of $S$ sound receivers
$\left\\{\rho_{1},\dots,\rho_{S}\right\\}$. At each timestep, the frequency
$f^{\prime}_{i}$ heard by each sound receiver $\rho_{i}$ is modified by the
Doppler effect, modifying the frequency heard by an observer as a function of
the velocity of the sound emitter. The amplitude is modified as function of
the relative position of the emitter in relation to the observer following an
inverse square law. To train the representation models, we employ a random
policy to collect a dataset composed of 20,000 training samples and 2,000 test
samples following the procedure in (Silva et al., 2019).

Models We consider the MVAE (Wu & Goodman, 2018) and the MUSE (Vasco et al.,
2021) models which are two commonly used approaches for the perception of
multimodal RL agents. For GMC, we employ the same modality-specific encoders
$f_{1}(\cdot),f_{2}(\cdot)$ as the baselines in addition to a joint-modality
encoder $f_{1:2}(\cdot)$. The shared projection head $g(\cdot)$ is comprised
of $2$ fully-connected layers. We use $\tau=0.3$ and set the dimensions of
intermediate and latent representations spaces to $d=64$ and $s=10$. We follow
the two-stage agent pipeline proposed in (Higgins et al., 2017) and initially
train all representation models on the dataset of collected observations for
500 epochs using a learning rate of $10^{-3}$. We subsequently train a Deep
Deterministic Policy Gradient (DDPG) controller (Lillicrap et al., 2015) that
takes as input the representations $z_{1:2}$ encoded from complete
observations $x_{1:2}$ following the network architecture and training
hyperparameters used by Silva et al. (2019).

Evaluation We evaluate the performance of RL agents acting under incomplete
perceptions that employ the representation models to encode raw observations
of the environment. During execution, the environment may provide any of the
modalities {$x_{1:2},x_{1},x_{2}$}. As such, we compare the performance of the
RL agents when directly using the policy learned from complete observations in
scenarios with possible missing modalities without any additional training.

Results Table 9 summarizes the total reward collected per episode for the
Pendulum scenario averaged over 100 episodes and 10 randomly seeded runs.
222Results for the MUSE model averaged over 9 randomly-seeded runs due to
divergence during training in the remaining seed.

Table 11: Number of parameters (in millions) of the representation models employed in the multimodal Pendulum scenario. MVAE | MUSE | GMC (Ours)
---|---|---
3.8 | 4.3 | 1.9

The results show that only GMC is able to provide the agent with a
representation model robust to partial observations allowing the agent to act
under incomplete perceptual conditions with no performance loss. This is on
par with the DCA scores reported in Table 10 indicating that GMC geometrically
better aligns the representations compared to the baselines. Once again, as
shown in Table 11, GMC can achieve such performance with $50\%$ fewer
parameters than the smallest baseline, evidence of its efficiency.

###  5.4 Ablation studies

We perform an ablation study on the hyperparameters of GMC using the setup
from Section 5.1 on MHD dataset. In particular, we investigate: i) the
robustness of the GMC framework when varying the temperature parameter $\tau$;
ii) the performance of GMC when varying dimensionalities $d$ and $s$ of the
intermediate and latent representation spaces, respectively; and iii) the
performance of GMC trained with a modified loss function that uses only
complete observations as negative pairs. We report both classification results
and DCA scores in Appendix B and observe that GMC is robust to different
experimental conditions both in terms of performance and geometric alignment
of representations.

##  6 Conclusion

We addressed the problem of learning multimodal representations that are both
semantically rich and robust to missing modality information. We contributed
with a novel Geometric Multimodal Contrastive (GMC) learning framework that is
inspired by the visual contrastive learning methods and geometrically aligns
complete and modality-specific representations in a shared latent space. We
have shown that GMC is able to achieve state-of-the-art performance with
missing modality information across a wide range of different learning
problems while being computationally efficient (often requiring 90% fewer
parameters than similar models) and straightforward to integrate with existing
state-of-the-art approaches. We believe that GMC broadens the range of
possible applications of contrastive learning methods to multimodal scenarios
and opens many future work directions, such as investigating the effect of
modality-specific augmentations or usage of inherent intermediate
representations for modality-specific downstream tasks.

## Acknowledgements

This work has been supported by the Knut and Alice Wallenberg Foundation,
Swedish Research Council and European Research Council. This work was also
partially supported by Portuguese national funds through the Portuguese
Fundação para a Ciência e a Tecnologia under project UIDB/50021/2020 (INESC-ID
multi annual funding) and project PTDC/CCI-COM/5060/2021. In addition, this
research was partially supported by TAILOR, a project funded by EU Horizon
2020 research and innovation programme under GA No. 952215. This work was also
supported by funds from Europe Research Council under project BIRD 884887.
Miguel Vasco acknowledges the Fundação para a Ciência e a Tecnologia PhD grant
SFRH/BD/139362/2018.

## References

  * Bagher Zadeh et al. (2018) Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2236–2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1208.  URL https://aclanthology.org/P18-1208.
  * Baltrušaitis et al. (2018) Baltrušaitis, T., Ahuja, C., and Morency, L.-P.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443, 2018.
  * Cao & Wu (2021) Cao, Y.-H. and Wu, J.  Rethinking self-supervised learning: Small is beautiful.  _arXiv preprint arXiv:2103.13559_ , 2021.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In III, H. D. and Singh, A. (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 1597–1607. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/chen20j.html.
  * Guo et al. (2019) Guo, W., Wang, J., and Wang, S.  Deep multimodal representation learning: A survey.  _IEEE Access_ , 7:63373–63394, 2019.  doi: 10.1109/ACCESS.2019.2916887.
  * Higgins et al. (2017) Higgins, I., Pal, A., Rusu, A., Matthey, L., Burgess, C., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A.  Darla: Improving zero-shot transfer in reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1480–1490. PMLR, 2017.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In Bengio, Y. and LeCun, Y. (eds.), _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.  URL http://arxiv.org/abs/1312.6114.
  * Liang et al. (2021) Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L. Y., Wu, P., Lee, M. A., Zhu, Y., et al.  Multibench: Multiscale benchmarks for multimodal representation learning.  2021\.
  * Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * McInnes et al. (2018) McInnes, L., Healy, J., Saul, N., and Großberger, L.  Umap: Uniform manifold approximation and projection.  _Journal of Open Source Software_ , 3(29):861, 2018.  doi: 10.21105/joss.00861.  URL https://doi.org/10.21105/joss.00861.
  * Meo & Lanillos (2021) Meo, C. and Lanillos, P.  Multimodal vae active inference controller.  _arXiv preprint arXiv:2103.04412_ , 2021.
  * Poklukar et al. (2021) Poklukar, P., Varava, A., and Kragic, D.  Geomca: Geometric evaluation of data representations.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8588–8598. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/poklukar21a.html.
  * Poklukar et al. (2022) Poklukar, P., Polianskii, V., Varava, A., Pokorny, F. T., and Jensfelt, D. K.  Delaunay component analysis for evaluation of data representations.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=HTVch9AMPa.
  * Shi et al. (2019) Shi, Y., Paige, B., Torr, P., et al.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  _Advances in Neural Information Processing Systems_ , 32:15718–15729, 2019.
  * Silva et al. (2019) Silva, R., Vasco, M., Melo, F. S., Paiva, A., and Veloso, M.  Playing games in the dark: An approach for cross-modality transfer in reinforcement learning.  _arXiv preprint arXiv:1911.12851_ , 2019.
  * Suzuki et al. (2016) Suzuki, M., Nakayama, K., and Matsuo, Y.  Joint multimodal learning with deep generative models.  _arXiv preprint arXiv:1611.01891_ , 2016.
  * Tremblay et al. (2021) Tremblay, J.-F., Manderson, T., Noca, A., Dudek, G., and Meger, D.  Multimodal dynamics modeling for off-road autonomous vehicles.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1796–1802. IEEE, 2021.
  * Tsai et al. (2018) Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R.  Learning factorized multimodal representations.  _arXiv preprint arXiv:1806.06176_ , 2018.
  * Tsai et al. (2019) Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Florence, Italy, 7 2019\. Association for Computational Linguistics.
  * Vasco et al. (2021) Vasco, M., Yin, H., Melo, F. S., and Paiva, A.  How to sense the world: Leveraging hierarchy in multimodal perception for robust reinforcement learning agents.  _arXiv preprint arXiv:2110.03608_ , 2021.
  * Vasco et al. (2022) Vasco, M., Yin, H., Melo, F. S., and Paiva, A.  Leveraging hierarchy in multimodal generative models for effective cross-modality inference.  _Neural Networks_ , 146:238–255, 2022.
  * Wu & Goodman (2018) Wu, M. and Goodman, N.  Multimodal generative models for scalable weakly-supervised learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 5580–5590, 2018.
  * Yin et al. (2017) Yin, H., Melo, F. S., Billard, A., and Paiva, A.  Associate latent encodings in learning from demonstrations.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Zadeh et al. (2016) Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6):82–88, 2016\.  doi: 10.1109/MIS.2016.94.
  * Zambelli et al. (2020) Zambelli, M., Cully, A., and Demiris, Y.  Multimodal representation models for prediction and control from partial information.  _Robotics and Autonomous Systems_ , 123:103312, 2020.

##  Appendix A Delaunay Component Analysis

Delaunay Component Analysis (DCA) (Poklukar et al., 2022) is a recently
proposed method for general evaluation of data representations. The basic idea
of DCA is to compare geometric and topological properties of two sets of
representations – a reference set $R$ representing the true underlying data
manifold and an evaluation set $E$. If the sets $R$ and $E$ represent data
from the same underlying manifold, then the geometric and topological
properties extracted from manifolds described by $R$ and $E$ should be
similar. DCA approximates these manifolds using a type of a neighbourhood
graph called Delaunay graph $\mathcal{G}$ build on the union $R\cup E$. The
alignment of $R$ and $E$ is then determined by analysing the connected
components of $\mathcal{G}$ from which several global and local scores are
derived.

DCA first evaluates each connected component $\mathcal{G}_{i}$ of
$\mathcal{G}$ by analyzing the number of points from $R$ and $E$ contained in
$\mathcal{G}_{i}$ as well as number of edges among these points. In
particular, each component $\mathcal{G}_{i}$ is evaluated by two scores:
consistency and quality. Intuitively, $\mathcal{G}_{i}$ has a high consistency
if it is equality represented by points from $R$ and $E$, and high quality if
$R$ and $E$ points are geometrically well aligned. The latter holds true if
the number of homogeneous edges among points in each of the sets is small
compared to the number of heterogeneous edges connecting representations from
$R$ and $E$.

To formally define the scores, we introduce the notation as in (Poklukar et
al., 2022): for a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ we denote by
$|\mathcal{G}|_{\mathcal{V}}$ the size of its vertex set and by
$|\mathcal{G}|_{\mathcal{E}}$ the size of its edge set. Moreover,
$\mathcal{G}^{Q}=(\mathcal{V}|_{Q},\mathcal{E}|_{Q\times
Q})\subset\mathcal{G}$ denotes its restriction to a set $Q\subset\mathcal{V}$.

######  Definition A.1.

Consistency $c$ and quality $q$ of a connected component
$\mathcal{G}_{i}\subset\mathcal{G}$ are defined as the ratios

$\displaystyle c(\mathcal{G}_{i})$
$\displaystyle=1-\frac{|\,|\mathcal{G}_{i}^{R}|_{\mathcal{V}}-|\mathcal{G}_{i}^{E}|_{\mathcal{V}}\,|}{|\mathcal{G}_{i}|_{\mathcal{V}}},$
$\displaystyle q(\mathcal{G}_{i})$
$\displaystyle=\begin{cases}1-\frac{(|\mathcal{G}_{i}^{R}|_{\mathcal{E}}+|\mathcal{G}_{i}^{E}|_{\mathcal{E}})}{|\mathcal{G}_{i}|_{\mathcal{E}}}&\text{if
}|\mathcal{G}_{i}|_{\mathcal{E}}\geq 1\\\ 0&\text{otherwise},\end{cases}$

respectively. Moreover, the scores computed on the entire Delaunay graph
$\mathcal{G}$ are called network consistency $c(\mathcal{G})$ and network
quality $q(\mathcal{G})$.

Besides the two global scores, network consistency and network quality defined
above, two more global similarity scores are derived from the local ones by
extracting the so-called fundamental components of high consistency and high
quality. In this work, we define a component $\mathcal{G}_{i}$ to be
fundamental if $c(\mathcal{G}_{i})>0$ and $q(\mathcal{G}_{i})>0$ and denote by
$\mathcal{F}$ the union of all fundamental components of the Delaunay graph
$\mathcal{G}$. By examining the proportion of points from $E$ and $R$ that are
contained in $\mathcal{F}$, DCA derives two global scores precision and recall
defined below.

######  Definition A.2.

Precision $\mathcal{P}$ and recall $\mathcal{R}$ associated to a Delaunay
graph $\mathcal{G}$ built on $R\cup E$ are defined as

$\displaystyle\mathcal{P}=\frac{|\mathcal{F}^{E}|_{\mathcal{V}}}{|\mathcal{G}^{E}|_{\mathcal{V}}}\quad\text{and}\quad\mathcal{R}=\frac{|\mathcal{F}^{R}|_{\mathcal{V}}}{|\mathcal{G}^{R}|_{\mathcal{V}}},$

respectively, where $\mathcal{F}^{R},\mathcal{F}^{E}$ are the restrictions of
$\mathcal{F}$ to the sets $R$ and $E$.

We refer the reader to (Poklukar et al., 2022) and (Poklukar et al., 2021) for
further details.

Table 12: Performance of GMC with different temperature values $\tau$ (Equation (1)) in the MHD dataset, in a downstream classification task under complete and partial observations. Accuracy results averaged over 5 independent runs. Higher is better. Observations | $\tau=0.05$ |  $\tau=0.1$ (Default) | $\tau=0.2$ | $\tau=0.3$ | $\tau=0.5$
---|---|---|---|---|---
Complete Observations | $\phantom{1}99.99\pm 0.01$ | $100.00\pm 0.00$ | $\phantom{1}99.99\pm 0.01$ | $\phantom{1}99.97\pm 3\mathrm{e}{-5}$ | $\phantom{1}99.96\pm 0.01$
Image Observations | $\phantom{1}99.78\pm 0.02$ | $\phantom{1}99.75\pm 0.03$ | $\phantom{1}99.84\pm 0.03$ | $\phantom{1}99.80\pm 0.04$ | $\phantom{1}99.89\pm 0.03$
Sound Observations | $\phantom{1}93.55\pm 0.22$ | $\phantom{1}93.04\pm 0.45$ | $\phantom{1}91.98\pm 0.29$ | $\phantom{1}91.87\pm 0.58$ | $\phantom{1}95.01\pm 0.38$
Trajectory Observations | $\phantom{1}99.94\pm 0.01$ | $\phantom{1}99.96\pm 0.02$ | $\phantom{1}99.97\pm 0.02$ | $\phantom{1}99.96\pm 0.01$ | $\phantom{1}99.80\pm 0.20$
Label Observations | $100.00\pm 0.00$ | $100.00\pm 0.00$ | $100.00\pm 0.00$ | $100.00\pm 0.00$ | $100.00\pm 0.00$
Table 13: DCA score obtained on GMC representations when trained with
different temperature values $\tau$ (Equation (1)) in the MHD dataset,
evaluating the geometric alignment of complete representations $z_{1:4}$ and
modality-specific ones $z_{1},\dots z_{4}$ used as $R$ and $E$ inputs in DCA,
respectively. The score is averaged over 5 independent runs. Higher is better.

$R$ $E$ $\tau=0.05$ $\tau=0.1$ (Default) $\tau=0.2$ $\tau=0.3$ $\tau=0.5$
Complete $(z_{1:4})$ Image $(z_{1})$ $0.96\pm 0.02$ $0.96\pm 0.02$ $0.93\pm
0.01$ $0.92\pm 0.00$ $0.89\pm 0.02$ Complete $(z_{1:4})$ Sound $(z_{2})$
$0.95\pm 0.02$ $0.87\pm 0.16$ $0.96\pm 0.02$ $0.99\pm 0.00$ $0.87\pm 0.04$
Complete $(z_{1:4})$ Trajectory $(z_{3})$ $0.96\pm 0.02$ $0.86\pm 0.05$
$0.90\pm 0.03$ $0.92\pm 0.00$ $0.64\pm 0.11$ Complete $(z_{1:4})$ Label
$(z_{4})$ $1.00\pm 0.00$ $1.00\pm 0.00$ $1.00\pm 0.00$ $1.00\pm 0.00$ $0.94\pm
0.02$

##  Appendix B Ablation Study on GMC

We perform a thorough ablation study on the hyperparameters of GMC using the
setup from Section 5.1 on the MHD dataset. In particular, we investigate:

  1. 1.

the robustness of the GMC framework when varying the temperature parameter
$\tau$;

  2. 2.

the performance of GMC with different dimensionalities of the intermediate
representations $h\in\mathbb{R}^{d}$;

  3. 3.

the performance of GMC with different dimensionalities of the shared latent
representations $z\in\mathbb{R}^{s}$;

  4. 4.

the performance of GMC with a modified loss $\mathcal{L}_{\text{mNT-
Xent}}^{*}$ that only uses complete observations as negative pairs.

In all experiments we report both classification results and DCA scores.

Table 14: Performance of GMC with different values of intermediate
representation dimensionality $h\in\mathbb{R}^{d}$ in the MHD dataset, in a
downstream classification task under complete and partial observations.
Accuracy results averaged over 5 independent runs. Higher is better.

Observations $d=32$ $d=64$ (Default) $d=128$ Complete Observations
$\phantom{1}99.99\pm 0.01$ $100.00\pm 0.00$ $\phantom{1}99.99\pm 0.01$ Image
Observations $\phantom{1}99.75\pm 0.04$ $\phantom{1}99.75\pm 0.03$
$\phantom{1}99.72\pm 0.07$ Sound Observations $\phantom{1}93.31\pm 0.41$
$\phantom{1}93.04\pm 0.45$ $\phantom{1}93.34\pm 0.51$ Trajectory Observations
$\phantom{1}99.96\pm 0.01$ $\phantom{1}99.96\pm 0.02$ $\phantom{1}99.96\pm
0.01$ Label Observations $100.00\pm 0.00$ $100.00\pm 0.00$ $100.00\pm 0.00$

Table 15: DCA score obtained on GMC representations when varying the dimension
of intermediate representations $h\in\mathbb{R}^{d}$ in the MHD dataset,
evaluating the geometric alignment of complete representations $z_{1:4}$ and
modality-specific ones $z_{1},\dots z_{4}$ used as $R$ and $E$ inputs in DCA,
respectively. The score is averaged over 5 independent runs. Higher is better.

$R$ $E$ $d=32$ $d=64$ (Default) $d=128$ Complete $(z_{1:4})$ Image $(z_{1})$
$0.91\pm 0.04$ $0.96\pm 0.02$ $0.92\pm 0.04$ Complete $(z_{1:4})$ Sound
$(z_{2})$ $0.77\pm 0.17$ $0.87\pm 0.16$ $0.96\pm 0.04$ Complete $(z_{1:4})$
Trajectory $(z_{3})$ $0.86\pm 0.04$ $0.86\pm 0.05$ $0.86\pm 0.07$ Complete
$(z_{1:4})$ Label $(z_{4})$ $1.00\pm 0.00$ $1.00\pm 0.00$ $1.00\pm 0.00$

Table 16: Performance of GMC with different values of latent representation
dimensionality $z\in\mathbb{R}^{s}$ in the MHD dataset, in a downstream
classification task under complete and partial observations. Accuracy results
averaged over 5 independent runs. Higher is better.

Observations $d=32$ $d=64$ (Default) $d=128$ Complete Observations
$\phantom{1}99.99\pm 0.01$ $100.00\pm 0.00$ $\phantom{1}99.99\pm 0.01$ Image
Observations $\phantom{1}99.75\pm 0.04$ $\phantom{1}99.75\pm 0.03$
$\phantom{1}99.72\pm 0.07$ Sound Observations $\phantom{1}93.31\pm 0.41$
$\phantom{1}93.04\pm 0.45$ $\phantom{1}93.34\pm 0.51$ Trajectory Observations
$\phantom{1}99.96\pm 0.01$ $\phantom{1}99.96\pm 0.02$ $\phantom{1}99.96\pm
0.01$ Label Observations $100.00\pm 0.00$ $100.00\pm 0.00$ $100.00\pm 0.00$

Table 17: DCA score obtained on GMC representations when varying the dimension
of latent representations $z\in\mathbb{R}^{d}$ in the MHD dataset, evaluating
the geometric alignment of complete representations $z_{1:4}$ and modality-
specific ones $z_{1},\dots z_{4}$ used as $R$ and $E$ inputs in DCA,
respectively. The score is averaged over 5 independent runs. Higher is better.

$R$ $E$ $d=32$ $d=64$ (Default) $d=128$ Complete $(z_{1:4})$ Image $(z_{1})$
$0.93\pm 0.03$ $0.96\pm 0.02$ $0.91\pm 0.03$ Complete $(z_{1:4})$ Sound
$(z_{2})$ $0.89\pm 0.01$ $0.87\pm 0.16$ $0.86\pm 0.19$ Complete $(z_{1:4})$
Trajectory $(z_{3})$ $0.81\pm 0.03$ $0.86\pm 0.05$ $0.88\pm 0.06$ Complete
$(z_{1:4})$ Label $(z_{4})$ $1.00\pm 0.00$ $1.00\pm 0.00$ $1.00\pm 0.00$

Table 18: Performance of GMC with different loss functions in the MHD dataset,
in a downstream classification task under complete and partial observations.
Accuracy results averaged over 5 independent runs. Higher is better.

Observations $\mathcal{L}_{\text{mNT-Xent}}$ (Default)
$\mathcal{L}_{\text{mNT-Xent}}^{*}$ Complete Observations $100.00\pm 0.00$
$\phantom{1}99.97\pm 0.02$ Image Observations $\phantom{1}99.75\pm 0.03$
$\phantom{1}99.87\pm 0.01$ Sound Observations $\phantom{1}93.04\pm 0.45$
$\phantom{1}92.79\pm 0.24$ Trajectory Observations $\phantom{1}99.96\pm 0.02$
$\phantom{1}99.98\pm 0.01$ Label Observations $100.00\pm 0.00$ $100.00\pm
0.00$

Table 19: DCA score obtained on GMC representations when trained different
loss functions in the MHD dataset, evaluating the geometric alignment of
complete representations $z_{1:4}$ and modality-specific ones $z_{1},\dots
z_{4}$ used as $R$ and $E$ inputs in DCA, respectively. The score is averaged
over 5 independent runs. Higher is better.

$R$ $E$ $\mathcal{L}_{\text{mNT-Xent}}$ (Default) $\mathcal{L}_{\text{mNT-
Xent}}^{*}$ Complete $(z_{1:4})$ Image $(z_{1})$ $0.96\pm 0.02$ $0.80\pm 0.02$
Complete $(z_{1:4})$ Sound $(z_{2})$ $0.87\pm 0.16$ $0.27\pm 0.14$ Complete
$(z_{1:4})$ Trajectory $(z_{3})$ $0.86\pm 0.05$ $0.86\pm 0.03$ Complete
$(z_{1:4})$ Label $(z_{4})$ $1.00\pm 0.00$ $0.24\pm 0.10$

Temperature parameter We study the performance of GMC when varying
$\tau\in\\{0.05,0.1,0.2,0.3,0.5\\}$ (see Equation (1)). We present the
classification results and DCA scores in Table 12 and Table 13, respectively.
We observe that classification results are rather robust to different values
of temperature, while increasing the temperature seems to have slightly
negative effect on the geometry of the representations. For example, in Table
13, we observe that for $\tau=0.5$ the trajectory representations $z_{3}$ are
worse aligned with $z_{1:4}$.

Dimensionality of intermediate representations We vary the dimension of the
intermediate representations space $d=\\{32,64,128\\}$ and present the
resulting classification results and DCA scores in Table 14 and Table 15,
respectively. The differences in classification results across different
dimensions are covered by the margin of error, indicating the robustness of
GMC to different sizes of the intermediate representations. We observe similar
stability of the DCA scores in Table 13 with minor variations in the geometric
alignment for the sound modality $z_{2}$ which benefits from the larger
intermediate representation space.

Dimensionality of latent representations We repeat a similar evaluation for
the dimension of the latent space $s=\\{32,64,128\\}$ and present the
classification and DCA scores in Table 16 and Table 17, respectively. We
observe that GMC is robust to changes in $s$ both in terms of performance and
geometric alignment.

Loss function We consider an ablated version of the loss function,
$\mathcal{L}_{\text{mNT-Xent}}^{*}$, that considers only complete-observations
as negative pair, i.e. $\Omega^{*}(i)=s_{1:M,1:M}(i,j)$ for $j=1,\dots,B$
where $B$ is the size of the mini-batch. We present the classification results
and DCA scores in Table 18 and Table 19, respectively. The results in Table 18
highlight the importance of the contrasting the complete representations to
learn a robust representation suitable for downstream tasks as we observe
minimal variation in classification accuracy when considering different loss.
However, we observe worse geometric alignment when using
$\mathcal{L}_{\text{mNT-Xent}}^{*}$ loss during training of GMC. This suggests
that contrasting among individual modalities is beneficial for geometrical
alignment of the representations.

Table 20: DCA score of the models in the CMU-MOSI dataset, evaluating the
geometric alignment of complete representations $z_{1:4}$ and modality-
specific ones $\\{z_{1},\dots z_{4}\\}$ used as $R$ and $E$ inputs in DCA,
respectively. The score is averaged over 5 independent runs. Higher is better.

R E Baseline GMC (Ours) Complete $(z_{1:3})$ Text $(z_{1})$ $0.54\pm 0.07$
$\bf 0.93\pm 0.02$ Complete $(z_{1:3})$ Audio $(z_{2})$ $0.14\pm 0.06$ $\bf
0.75\pm 0.05$ Complete $(z_{1:3})$ Vision $(z_{3})$ $0.36\pm 0.09$ $\bf
0.85\pm 0.04$

##  Appendix C Experiment 2: Supervised Learning with the CMU-MOSI dataset

Table 21: Performance of different multimodal representation methods in the
CMU-MOSI dataset, in a classification task under complete and partial
observations. Results averaged over 5 independent runs. Arrows indicate the
direction of improvement.

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $1.033\pm 0.037$ | $\bf 1.010\pm 0.070$
Cor ($\uparrow$) | $0.642\pm 0.008$ | $\bf 0.649\pm 0.019$
F1 ($\uparrow$) | $0.770\pm 0.017$ | $\bf 0.776\pm 0.023$
Acc ($\%,\uparrow$) | $77.07\pm 01.67$ | $\bf 77.59\pm 02.20$
(a) Complete Observations $(x_{1:3})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $1.244\pm 0.100$ | $\bf 1.119\pm 0.033$
Cor ($\uparrow$) | $0.431\pm 0.208$ | $\bf 0.573\pm 0.016$
F1 ($\uparrow$) | $0.698\pm 0.053$ | $\bf 0.727\pm 0.013$
Acc ($\%,\uparrow$) | $66.28\pm 07.74$ | $\bf 72.32\pm 0.013$
(b) Text Observations $(x_{1})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $1\bf.431\pm 0.025$ | $1.434\pm 0.017$
Cor ($\uparrow$) | $0.056\pm 0.071$ | $\bf 0.211\pm 0.010$
F1 ($\uparrow$) | $\bf 0.588\pm 0.076$ | $0.570\pm 0.006$
Acc ($\%,\uparrow$) | $47.20\pm 05.67$ | $\bf 55.91\pm 01.11$
(c) Audio Observations $(x_{2})$

Metric | Baseline | GMC (Ours)
---|---|---
MAE ($\downarrow$) | $\bf 1.406\pm 0.041$ | $1.452\pm 0.035$
Cor ($\uparrow$) | $0.021\pm 0.028$ | $\bf 0.176\pm 0.028$
F1 ($\uparrow$) | $\bf 0.659\pm 0.049$ | $0.550\pm 0.015$
Acc ($\%,\uparrow$) | $53.87\pm 05.77$ | $\bf 54.30\pm 01.96$
(d) Video Observations $(x_{3})$

In this section, we repeat the experimental evaluation of Section 5.2 with the
CMU-MOSI dataset. We employ the same baseline and GMC architectures as in the
CMU-MOSEI evaluation and consider the same evaluation setup.

Results The results obtained on CMU-MOSI are reported in Table 21(d). We
observe that GMC improves the robustness of the model to the missing
modalities as seen from Tables 21(b), 21(c) and 21(d) where we use only
individual modalities as inputs. However, the increase in performance is not
as signification as in the case of the CMU-MOSEI dataset for audio ($x_{2}$)
and video ($x_{3}$) modalities where the baseline outperforms GMC on MAE and
F1 scores. We hypothesise that this behaviour is due to the intrinsic
difficulty of forming good contrastive pairs in small-sizes datasets (Cao &
Wu, 2021): the CMU-MOSI dataset has only 1513 training samples which hinders
the learning of a quality latent representations. However, we observe that GMC
still significantly improves the geometric alignment (Table 20) of the
modality-specific representations $z_{m}$ (comprising the set $E$) and
complete representations $z_{1:3}$ (comprising the set $R$) compared to the
baseline, even in this regime of small data.

##  Appendix D Model Architecture

We report the model architectures employed in our experiments. In Table 22(f)
and Table 23(f) we present the model architectures of GMC and baselines,
respectively, employed in the unsupervised problem studied in Section 5.1; in
Table 24(e) and Table 25 we present the model architectures of GMC and
baselines, respectively, used in the supervised problem considered in Section
5.2; and lastly, in Table 26(c) and Table 27(c) we present the model
architectures of GMC and baselines, respectively, employed in the RL task from
Section 5.3. For training the controller in the RL task, we employ the same
architecture as in (Silva et al., 2019). We use the following notation: FC for
fully connected layer, BN for batch normalization, Conv for convolutional
layer, TConv for transponse convolutions,.

Table 22: Network architectures for GMC in the unsupervised problem of Section
5.1.

(a) $f_{1}(x_{1})$ (Image) Base encoder
---
Input $\mathbb{R}^{1+28+28}$
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish
FC, $d$

(b) $f_{2}(x_{2})$ (Sound)

Base encoder Input $\mathbb{R}^{4+32+32}$ Conv2D, 1x128 kernel, 1 stride, 0
padding + BN2D + ReLU Conv2D, 4x1 kernel, (2,1) stride, (1,0) padding + BN2D +
ReLU Conv2D, 4x1 kernel, (2,1) stride, (1,0) padding + BN2D + ReLU FC, $d$

(c) $f_{3}(x_{3})$ (Trajectory) Base encoder
---
Input $\mathbb{R}^{200}$
FC, 512 + Swish
FC, 512 + Swish
FC, $d$

(d) $f_{4}(x_{4})$ (Label) Base encoder
---
Input $\mathbb{R}^{10}$
FC, 512 + Swish
FC, 512 + Swish
FC, $d$

(e) Shared projection head $g(h)$ Shared encoder
---
Input $\mathbb{R}^{d}$
FC, 512 + Swish
FC, 512 + Swish
FC, $s$

(f) Classifier Classifier
---
Input $\mathbb{R}^{s}$
FC, 256 + ReLU
FC, 128 + ReLU
FC, 10

Table 23: Network architectures employed for the baselines in the unsupervised
problem of Section 5.1.

(a) $\mathbf{x}_{1}$ (Image) Encoder | Decoder
---|---
Input $\mathbb{R}^{1+28+28}$ | Input $\mathbb{R}^{s}$
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish | FC, 512 + Swish
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish | FC, 6272 + Swish
FC, 512 + Swish | TConv2D, 4x4 kernel, 2 stride, 1 padding + Swish
FC, $s$, FC, $s$ | TConv2D, 4x4 kernel, 2 stride, 1 padding + Sigmoid

(b) $\mathbf{x}_{2}$ (Sound)

Encoder Decoder Input $\mathbb{R}^{4+32+32}$ Input $\mathbb{R}^{s}$ Conv2D,
1x128 kernel, 1 stride, 0 padding + BN2D + ReLU FC, 2048 + BN1D + ReLU Conv2D,
4x1 kernel, (2,1) stride, (1,0) padding + BN2D + ReLU TConv2D, 4x1 kernel,
(2,1) stride, (1,0) padding + BN2D + ReLU Conv2D, 4x1 kernel, (2,1) stride,
(1,0) padding + BN2D + ReLU TConv2D, 4x1 kernel, (2,1) stride, (1,0) padding +
BN2D + ReLU FC, $s$, FC, $s$ TConv2D, 1x128 kernel, 1 stride, 0 padding +
Sigmoid

(c) $\mathbf{x}_{3}$ (Trajectory) Encoder | Decoder
---|---
Input $\mathbb{R}^{200}$ | Input $\mathbb{R}^{s}$
FC, 512 + Swish | FC, 512 + Swish
FC, 512 + Swish | FC, 512 + Swish
FC, $s$, FC, $s$ | FC, 200 + Sigmoid

(d) $\mathbf{x}_{4}$ (Label) Encoder | Decoder
---|---
Input $\mathbb{R}^{10}$ | Input $\mathbb{R}^{s}$
FC, 512 + Swish | FC, 512 + Swish
FC, 512 + Swish | FC, 512 + Swish
FC, $s$, FC, $s$ | FC, 10

(e) Nexus top-level networks

Encoder Decoder Input $\mathbb{R}^{d}$ Input $\mathbb{R}^{s}$ FC, 512 + BN1D +
LeakyReLU FC, 512 + BN1D + LeakyReLU FC, 512 + BN1D + LeakyReLU FC, 512 + BN1D
+ LeakyReLU FC, 512 + BN1D + LeakyReLU FC, 512 + BN1D + LeakyReLU FC, $s$, FC,
$s$ FC, $d$

(f) MUSE top-level networks Encoder | Decoder
---|---
Input $\mathbb{R}^{d}$ | Input $\mathbb{R}^{s}$
FC, 128 + ReLU | FC, 128 + ReLU
FC, 128 + ReLU | FC, 128 + ReLU
FC, $s$, FC, $s$ | FC, $d$

Table 24: Network architectures for GMC in the supervised problem of Section
5.2.

(a) $f_{1}(x_{1})$ (Text) Base encoder
---
Input $\mathbb{R}^{1+50+300}$
GRU, 30 hidden, batch first=True
FC, $d$

(b) $f_{2}(x_{2})$ (Audio) Base encoder
---
Input $\mathbb{R}^{1+50+74}$
GRU, 30 hidden, batch first=True
FC, $d$

(c) $f_{3}(x_{3})$ (Sound) Base encoder
---
Input $\mathbb{R}^{1+50+35}$
GRU, 30 hidden, batch first=True
FC, $d$

(d) Shared projection head $g(h)$ Shared encoder
---
Input $\mathbb{R}^{d}$
FC, $s$

(e) Classifier Classifier
---
Input $\mathbb{R}^{s}$
Residual block (FC, $s$ \+ ReLU + Dropout (0.1))
FC, 1

Table 25: Network architectures for the baseline in the supervised problem of
Section 5.2. For a detailed description of the cross-modality attention
blocks, please refer to (Tsai et al., 2019). Transformer Encoder
---
Text $x_{1}\in\mathbb{R}^{1+50+300}$ | Audio $x_{2}\in\mathbb{R}^{1+50+74}$ | Video $x_{3}\in\mathbb{R}^{1+50+35}$
Conv1D, kernel 1, padding 0 | Conv1D, kernel 1, padding 0 | Conv1D, kernel 1, padding 0
| Attention block (text, audio)
---
Attention block (text, video)
| Attention block (audio, text)
---
Attention block (audio, video)
| Attention block (video, text)
---
Attention block (video, audio)
Fusion (concatenation)
Residual block (FC, 180 + ReLU + Dropout (0.1))
FC, 1
Table 26: Network architectures for GMC in the RL task of Section 5.3.

(a) $f_{1}(x_{1})$ (Image) Base encoder
---
Input $\mathbb{R}^{1+28+28}$
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish
FC, $d$

(b) $f_{2}(x_{2})$ (Sound) Base encoder
---
Input $\mathbb{R}^{12}$
FC, 50 + Swish
FC, 50 + Swish
FC, $d$

(c) Shared projection head $g(h)$ Shared encoder
---
Input $\mathbb{R}^{d}$
FC, 128 + Swish
FC, $s$

Table 27: Network architectures employed for the baselines in the RL task of
Section 5.3 (best viewed with zoom).

(a) $\mathbf{x}_{1}$ (Image) Encoder | Decoder
---|---
Input $\mathbb{R}^{1+28+28}$ | Input $\mathbb{R}^{s}$
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish | FC, 128 + Swish
Conv2D, 4x4 kernel, 2 stride, 1 padding + Swish | FC, 14400 + Swish
FC, 128 + Swish | TConv2D, 4x4 kernel, 2 stride, 1 padding + Swish
FC, $s$, FC, $s$ | TConv2D, 4x4 kernel, 2 stride, 1 padding + Sigmoid

(b) $\mathbf{x}_{2}$ (Sound) Encoder | Decoder
---|---
Input $\mathbb{R}^{12}$ | Input $\mathbb{R}^{s}$
FC, 50 + Swish | FC, 50 + Swish
FC, 50 + Swish | FC, 50 + Swish
FC, $s$, FC, $s$ | FC, 12 + Sigmoid

(c) MUSE top-level networks Encoder | Decoder
---|---
Input $\mathbb{R}^{d}$ | Input $\mathbb{R}^{s}$
FC, 256 + ReLU | FC, 256 + ReLU
FC, 256 + ReLU | FC, 256 + ReLU
FC, 256 + ReLU | FC, 256 + ReLU
FC, $s$, FC, $s$ | FC, $d$

##  Appendix E Training Hyperparameters

In this section, we report the training hyperparameters used in our
experiments. In Tables 28, 29 and 30 we present the hyperparameters employed
in the unsupervised problem (Section 5.1), supervised problem (Section 5.2)
and RL task (Section 5.3), respectively. For training the controller in the RL
task, we employ the same training hyperparameters as in (Silva et al., 2019).

Table 28: Training hyperparameters employed in the unsupervised problem of Section 5.1. Parameter | Value
---|---
Intermediate representation size $d$ | 64
Latent representation size $s$ | 64
Model training epochs | 100
Classifier training epochs | 50
Learning rate | 1$\mathrm{e}{-3}$
Batch size $B$ | 64
Temperature $\tau$ | 0.1
Table 29: Training hyperparameters employed in the supervised problem of Section 5.2. Parameter | Value
---|---
Intermediate representation size $d$ | 60
Latent representation size $s$ | 60
Model training epochs | 40
Learning rate | 1$\mathrm{e}{-3}$ (Decaying)
Batch size $B$ | 40
Temperature $\tau$ | 0.3
Table 30: Training hyperparameters employed in the reinforcement learning task of Section 5.3. Parameter | Value
---|---
Intermediate representation size $d$ | 64
Latent representation size $s$ | 10
Model training epochs | 500
Learning rate | 1$\mathrm{e}{-3}$ (Decaying)
Batch size $B$ | 128
Temperature $\tau$ | 0.3

##  Appendix F Additional Visualizations of the Alignment of Complete and
Modality-Specific Representations

We present additional visualizations of encodings of complete and modality-
specific representations in the MHD dataset for multiple multimodal
representation models. In Figures 3, 4 and 5, we show visualizations of sound
representations $z_{2}$, trajectory $z_{3}$ and label $z_{4}$ (in orange),
respectively, and complete representations $z_{1:4}$ (in blue). Note that
points detected as outliers by DCA are _not_ included in the visualization.
For example, we observe that certain labels representations for baseline
models are marked as outliers in Figure 5.

(a) MVAE

(b) MMVAE

(c) Nexus

(d) MUSE

(e) MFM

(f) GMC (Ours)

Figure 3: UMAP visualization of complete representations $z_{1:4}$ (blue) and
sound representations $z_{2}$ (orange) obtained from several state-of-the-art
multimodal representation learning models on the MHD dataset considered in
Section 5.1. Best viewed in color.

(a) MVAE

(b) MMVAE

(c) Nexus

(d) MUSE

(e) MFM

(f) GMC (Ours)

Figure 4: UMAP visualization of complete representations $z_{1:4}$ (blue) and
trajectory representations $z_{3}$ (orange) obtained from several state-of-
the-art multimodal representation learning models on the MHD dataset
considered in Section 5.1. Best viewed in color.

(a) MVAE

(b) MMVAE

(c) Nexus

(d) MUSE

(e) MFM

(f) GMC (Ours)

Figure 5: UMAP visualization of complete representations $z_{1:4}$ (blue) and
label representations $z_{4}$ (orange) obtained from several state-of-the-art
multimodal representation learning models on the MHD dataset considered in
Section 5.1. Best viewed in color.