# Unity by Diversity: Improved Representation Learning for Multimodal VAEs

Thomas M. Sutter
Department of Computer Science
ETH Zurich
&Yang Meng
Department of Statistics
UC Irvine
&Andrea Agostini
Department of Computer Science
ETH Zurich
&Daphné Chopard
Department of Computer Science and Department of Intensive Care and
Neonatology
ETH Zurich and University Children’s Hospital Zurich, University of Zurich
&Norbert Fortin
Department of Neuroscience
UC Irvine
&Julia E. Vogt
Department of Computer Science
ETH Zurich
&Bahbak Shahbaba
Department of Statistics
UC Irvine
&Stephan Mandt
Department of Computer Science
UC Irvine

###### Abstract

Variational Autoencoders for multimodal data hold promise for many tasks in
data analysis, such as representation learning, conditional generation, and
imputation. Current architectures either share the encoder output, decoder
input, or both across modalities to learn a shared representation. Such
architectures impose hard constraints on the model. In this work, we show that
a better latent representation can be obtained by replacing these hard
constraints with a soft constraint. We propose a new mixture-of-experts prior,
softly guiding each modality’s latent representation towards a shared
aggregate posterior. This approach results in a superior latent representation
and allows each encoding to preserve information better from its uncompressed
original features. In extensive experiments on multiple benchmark datasets and
two challenging real-world datasets, we show improved learned latent
representations and imputation of missing data modalities compared to existing
methods.

##  1 Introduction

The fusion of diverse modalities and data types is transforming our
understanding of complex phenomena, enabling more nuanced and comprehensive
insights through the integration of varied information sources. Consider, for
instance, the role of a medical practitioner who synthesizes multiple tests
and measurements during diagnosis and treatment. This process involves merging
shared information across different tests and identifying test-specific
details, both of which are critical for optimal patient care and medical
decision-making.

Among the existing methods, multimodal Variational Autoencoders (VAEs) have
emerged as a promising approach for jointly modeling and learning from weakly-
supervised heterogeneous data sources. While scalable multimodal VAEs
utilizing a single shared latent space efficiently handle multiple modalities
(Wu and Goodman, 2018; Shi et al., 2019; Sutter et al., 2021), finding an
optimal method to aggregate these modalities remains challenging. The
aggregation methods and resulting joint representations are often suboptimal
and overly restrictive (Daunhawer et al., 2022; Sutter, 2023), leading to
inferior latent representations and generative quality. This trade-off between
shared and modality-specific information in the latent representations of
multimodal VAEs results in limited quality or coherence in generated samples,
even in relatively simple scenarios.

In this work, we propose a novel multimodal VAE, termed the multimodal
variational mixture-of-experts prior (MMVM) VAE, to overcome the
aforementioned limitations. Instead of modeling the dependencies between
different modalities through a joint posterior approximation, we introduce a
multimodal and data-dependent prior distribution (see Figure 1). Our proposed
multimodal objective is inspired by the VAMP-prior formulation introduced by
Tomczak and Welling (2017), which is traditionally used in the feature space
but adapted here for the modality space. The resulting regularization term in
the VAE objective can be interpreted as minimizing the distance between
positive pairs, similar to contrastive learning methods (Oord et al., 2019;
Tian et al., 2020); see Section 4 for details.

We demonstrate the superior performance of the MMVM VAE on three multimodal
benchmark datasets, comparing it to unimodal VAEs and multimodal VAEs with
joint posterior approximations. Our evaluation focuses on the generative
coherence and the quality of the learned latent representations. While
independent unimodal VAEs fail to leverage additional modalities during
training, they avoid multimodal aggregation disturbances in data
reconstruction. On the other hand, multimodal VAEs with a joint posterior
approximation must combine both shared and modality-specific information.
Previous work by Daunhawer et al. (2022) has shown that this approach results
in a trade-off between reconstruction quality and learned latent
representation. In contrast, the MMVM VAE accurately reconstructs all
modalities and learns meaningful latent representations.

In more practical settings, we address two challenging tasks from the
neuroscience and medical domain. First, we analyze hippocampal neural
activities from multiple subjects in a memory experiment (Allen et al., 2016).
By treating each subject as a modality, our MMVM VAE enables the description
of underlying neural patterns shared across subjects while quantifying
individual differences in brain activity and behavior, thereby providing
potential insights into the neural mechanisms underlying memory impairment.
Second, we tackle the identification of cardiopulmonary diseases from chest
radiographs using the MIMIC-CXR dataset (Johnson et al., 2019b), which
reflects real-world conditions with images of varying quality. By leveraging
both frontal and lateral X-ray views as distinct modalities, our MMVM method
learns representations that consistently improve disease classification
compared to existing VAEs.

This paper advances multimodal machine learning by providing a robust
framework for integrating diverse data types and improving the quality of
learned representations and generative models.

(a) Independent VAEs

(b) Aggregated VAE

(c) MMVM VAE

Figure 1:  Independent VAEs (Figure 1(a)) provide reconstructions for
individual modalities but lack information sharing across modalities.
Multimodal VAEs with joint posterior approximation (Figure 1(b)) aggregate
unimodal posteriors into a joint posterior but may incur poor reconstruction
quality. Our proposed MMVM VAE (Figure 1(c)) enhances independent VAEs with a
data-dependent prior, $h(\bm{z}\mid\bm{X})$, allowing soft-sharing of
information between modalities while preserving modality-specific
reconstructions.

##  2 Related Work

Multimodal Learning. While there is a long line of research on multimodal
machine learning (ML) (Baltrušaitis et al., 2018; Liang et al., 2022),
multimodal generative ML has gained additional attraction in recent years
(Manduchi et al., 2024), driven by impressive results in text-to-image
generation (Ramesh et al., 2021, 2022; Saharia et al., 2022). Unlike these
methods, we focus on methods that are designed for a large number of
modalities to generate any modality from any other modality without having to
train a prohibitive number of different models111There are $2^{M}-1$ different
subsets for a dataset of $M$ modalities and, hence, paths for any-to-any
mappings..

Multimodal VAEs. Multimodal VAEs that learn a joint posterior approximation of
all modalities (e.g., Wu and Goodman, 2018; Shi et al., 2019; Sutter et al.,
2021) require restrictive assumptions, which lead to inferior performance as
shown in Daunhawer et al. (2022). Multimodal VAEs using a joint posterior
approximation are based on aggregation in the latent space222Sutter et al.
(2021) describe how different implementations of joint posterior multimodal
VAEs relate to different abstract mean definitions.. Extensions (Sutter et
al., 2020; Daunhawer et al., 2020; Palumbo et al., 2023) have introduced
modality-specific latent subspaces that lead to improved generative quality
but cannot completely overcome these limitations and require strong
assumptions. In contrast, the proposed MMVM method uses neither an aggregated
latent space nor modality-specific latent subspaces as in previous works. It
only leverages a data-dependent prior distribution to regularize the learned
posterior approximations. A related line of work with different constraints
are multiview VAEs (Bouchacourt et al., 2018; Hosoya, 2018). In contrast to
multimodal VAEs, multiview VAEs often use a single encoder and decoder for all
views (thereby sharing the parameter weights between views). While initial
attempts also assume knowledge about the number of shared and independent
generative factors, extensions (Locatello et al., 2020; Sutter et al., 2023a,
b) infer these properties during training.

Role of Prior in VAE Formulations. Tomczak and Welling (2017) first
incorporated data-dependent priors into VAEs by introducing the VAMP-prior. In
contrast to Tomczak and Welling (2017), who are primarily interested in better
ELBO approximations, our focus is on learning better multimodal
representations and overcoming the limitations faced in previous multimodal
VAE works. Sutter et al. (2020) used a data-dependent prior combined with a
joint posterior approximation defining a Jensen-Shannon divergence
regularization based on the geometric mean. However, their work lacks a
rigorous derivation and relies on the suboptimal conditional generation during
training (Daunhawer et al., 2022). Joy et al. (2021) also presented a
multimodal VAE inspired by the VAMP-prior VAE. They leverage the VAMP prior to
model missing modalities, rather than using it as a regularization objective
between multimodal samples, as we do in this work. An additional line of work
(e.g., Bhattacharyya et al., 2019; Mahajan et al., 2020) leverages normalizing
flows to increase the expressivity of the multimodal prior distribution, but
this sacrifices the method’s scalability.

##  3 Background on Multimodal VAEs

Problem Specification. We consider a dataset
$\mathbb{X}=\\{\bm{X}^{(i)}\\}_{i=1}^{n}$ where each
$\bm{X}^{(i)}=\\{\bm{x}_{1}^{(i)},\ldots,\bm{x}_{M}^{(i)}\\}$ is a set of $M$
modalities $\bm{x}_{m}$ with latent variables
$\bm{z}=\\{\bm{z}_{1}^{(i)},\ldots,\bm{z}_{M}^{(i)}\\}$. The modalities
$\bm{x}_{m}^{(i)}$ could represent images of the same object taken from
different camera angles, text-image pairs, or—as in this paper—neuroscience
data from different animal subjects with shared experimental conditions and
multiple medical measurements of a patient, . When contextually clear, we
remove the superscript $(i)$ to improve readability.

Inspired by variational autoencoders (VAEs, Kingma and Welling, 2014), we aim
to learn an objective for representation learning while sharing information
from different data modalities. For example, we would like to embed
neuroscience data into a shared latent space to make brain activations
comparable across subjects. At the same time, we want to avoid imposing
assumptions on information sharing that are too strong to be able to take
individual traits of the data modalities into account. As is typical in VAEs,
this procedure involves a decoder (or likelihood)
$p_{\theta}(\bm{X}\mid\bm{z})$, an encoder (or variational distribution)
${q_{\phi}(\bm{z}\mid\bm{X})}$, and a prior ${h(\bm{z}|\bm{X})}$ that we allow
to depend on the input.

Data-Dependent Prior and Objective. The VAE framework allows us to derive an
ELBO-like learning objective $\mathcal{E}$ as follows

$\displaystyle\mathcal{E}(\bm{X})=$
$\displaystyle~{}\mathbb{E}_{q_{\phi}(\bm{z}\mid\bm{X})}\left[\log
p_{\theta}(\bm{X}\mid\bm{z})-\log\frac{q_{\phi}(\bm{z}\mid\bm{X})}{h(\bm{z}|\bm{X})}\right].$

Above, $\theta$ and $\phi$ denote the learnable model variational parameters.
Importantly, our approach allows for an input-dependent prior
$h(\bm{z}\mid\bm{X})$. Data-dependent priors can be justified from an
empirical Bayes standpoint (Efron, 2012) and enable information sharing across
data points with an intrinsic multimodal structure, as in our framework. They
effectively amortize computation over many interrelated inference tasks. We
stress that by making the prior data dependent, our model no longer allows for
unconditional generation; however, this property can be restored by
incorporating pseudo inputs (Tomczak and Welling, 2017), hyper-priors
(Sønderby et al., 2016), or ex-post density estimation techniques (Ghosh et
al., 2019). We discuss the objective in more detail in Appendix A, where we
prove that the resulting objective is upper bounded by the mean squared
reconstruction error, ensuring the existence of (local) optima and thus
tractable optimization.

Encoder and Decoder. We now specify our encoder and decoder assumptions. A
simple encoder choice relies on a single neural network encoder that expects
multi-modal inputs, but this approach fails if one or more modalities are
missing (Suzuki and Matsuo, 2022). This shortcoming has motivated multiple
approaches (Wu and Goodman, 2018; Shi et al., 2019; Sutter et al., 2021) with
separate encoders $q_{\phi}^{m}(\bm{z}_{m}|\bm{x}_{m})$—one for each modality
$m$—that are then _aggregated_ in the latent space, e.g., by using a product
or mixture distribution. These representations are then fed into a single,
shared decoder. As argued and discussed in this paper, such aggregation can be
overly restrictive.

Instead, this paper explores a different aggregation mechanism that preserves
the individual encoders _and_ decoders for each modality. Hence, we assume
independent decoders $p_{\theta}^{m}(\bm{x}_{m}|\bm{z}_{m})$ for every
modality $m$, assuming conditional independence of each modality given their
latent representation (see also Wu and Goodman, 2018; Shi et al., 2019; Sutter
et al., 2021).

Following this assumption, we rewrite the objective $\mathcal{E}$ as

$\displaystyle\mathcal{E}(\bm{X})=$
$\displaystyle~{}\mathbb{E}_{q_{\phi}(\bm{z}\mid\bm{X})}\left[\log
h(\bm{z}\mid\bm{X})\right]+\sum_{m=1}^{M}\mathbb{E}_{q_{\phi}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log\frac{p_{\theta}^{m}(\bm{x}_{m}\mid\bm{z}_{m})}{q^{m}_{\phi}(\bm{z}_{m}\mid\bm{x}_{m})}\right].$
(1)

Our assumptions imply that the likelihood and posterior entropy terms (the
second term in Equation 1) decouple across modalities, i.e.
$q_{\phi}(\bm{z}\mid\bm{X})=\prod_{m=1}^{M}q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$
and
$p_{\theta}(\bm{X}\mid\bm{z})=\prod_{m=1}^{M}p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})$.
In contrast, the cross-entropy between the encoder and prior (the first term
in Equation 1) does not decouple and may result in information sharing across
modalities. We specify further design choices in the next section.

##  4 Multimodal Variational Mixture VAE

We propose the multimodal variational mixture-of-experts prior (MMVM) VAE, a
novel multimodal VAE. The main idea is to design a mixture-of-experts prior
across modalities that induces a soft-sharing of information between modality-
specific latent representations rather than hard-coding this through an
aggregation approach.

VAEs are an appealing model class that allows us to infer meaningful
representations and preserve modality-specific information due to the
reconstruction loss. Contrastive learning approaches, on the other hand, have
shown impressive results on representation learning tasks related to
extracting shared information between modalities by maximizing the similarity
of their representations (Radford et al., 2021). Contrastive approaches focus
on the shared information between modalities, neglecting potentially useful
modality-specific information. We are interested in preserving modality-
specific information, which is necessary to generate missing modalities
conditionally.

Therefore, we leverage the idea of maximizing the similarity of
representations for _generative models_. We propose a prior distribution that
models the dependency between the different views and a new multimodal
objective that encourages similarity between the unimodal posterior
approximations $q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$ using the
regularization term in the objective as a "soft-alignment" without the need
for an over-restrictive and aggregation-based joint posterior approximation.
In more detail, we discuss objectives based on data-dependent priors in
Appendix A.

To this end, we define a data-dependent MMVM prior distribution in the form of
a mixture-of-experts distribution of all unimodal posterior approximations

$\displaystyle h(\bm{z}\mid\bm{X})$
$\displaystyle=\prod_{m=1}^{M}h(\bm{z}_{m}\mid\bm{X})\hskip
14.22636pt\text{where}\hskip
14.22636pth(\bm{z}_{m}\mid\bm{X})=\frac{1}{M}\sum_{\tilde{m}=1}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}\mid\bm{x}_{\tilde{m}}),\hskip
7.11317pt\forall~{}m\leq M.$ (2)

This notation implies that we use the variational distributions of all
modalities $\tilde{m}$ to construct a mixture distribution and then use the
same mixture distribution as a prior for any modality $m$. Finally, we build
the product distribution over the $M$ components.

Our construction of a variational mixture of posteriors is similar to the
VAMP-prior of Tomczak and Welling (2017) that proposes the aggregate posterior
$q(\bm{z})\equiv\frac{1}{N}\sum_{i=1}^{n}q_{\phi}(\bm{z}\mid\bm{x}^{(i)})$ of
a unimodal VAE as a prior. Note, however, that our approach considers mixtures
in _modality_ space and not data space. In contrast to Tomczak and Welling
(2017), our variational mixture is conditioned on a specific instance $\bm{X}$
and, therefore, does not share information across different instances
$\bm{X}^{(i)}\in\mathbb{X}$. Rather, we share information across the different
modalities $\bm{x}_{m}^{(i)}\in\bm{X}^{(i)}$ _within_ a given instance.
Intuitively, we build the _aggregate posterior_ in modality space and
replicate this aggregate posterior over all modalities. We stress that this
aggregate posterior differs from the standard definition as an average of
variational posteriors over the empirical data distribution. Even though the
prior appears factorized over the modality space, each factor still shares
information across all data modalities by conditioning on the multimodal
feature vector $\bm{X}$ (Equation 2).

Figure 1 graphically illustrates the behavior of the proposed MMVM VAE
compared to a set of independent VAEs and an aggregation-based multimodal VAE.
A set of independent VAEs (Figure 1(a)) cannot share information among
modalities. Aggregation-based VAEs (Figure 1(b)), in contrast, enforce a
shared representation between the modalities. The MMVM VAE (Figure 1(c))
enables the soft-sharing of information between modalities through its input
data-dependent prior $h(\bm{z}\mid\bm{X})$.

Minimizing Jenson-Shannon Divergence. The "rate" term $R$ in the objective,
i.e., the combination of variational entropy and cross-entropy, reveals a
better understanding of the effect of the mixture prior. Defining
$R=KL(q_{\phi}(\bm{z}\mid\bm{X})||h(\bm{z}|\bm{X}))$ where $KL$ denotes the KL
divergence, the factorization in Equation 2 implies that

$\displaystyle
R=\sum_{m=1}^{M}KL\left(q_{\phi}^{m}(\bm{z}_{m}|\bm{x}_{m})||\frac{1}{M}\sum_{\tilde{m}}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}|\bm{x}_{\tilde{m}})\right)=M\cdot
JS(q_{\phi}^{1}(\bm{z}_{1}|\bm{x}_{1}),\ldots,q_{\phi}^{M}(\bm{z}_{M}|\bm{x}_{M})),$

where $JS(\cdot)$ is the Jensen-Shannon divergence between $M$ distributions
(Lin, 1991). Hence, maximizing the objective $\mathcal{E}(\bm{X})$ of the
proposed MMVM VAE is equal to minimizing $M$ times the JS divergence between
all the unimodal posterior approximations
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$. Minimizing the Jensen-Shannon
divergence between the posterior approximations is directly related to
pairwise similarities between posterior approximation distributions of
positive pairs, similar to contrastive learning approaches but in a generative
approach.

###  4.1 Optimality of the MMVM Prior

Lemma 4.1 shows that Equation 2 is _optimal_ in the sense that it is the
unique minimizer of the cross entropy between our chosen variational
distribution and an arbitrary prior.

######  Lemma 4.1.

The expectation on the right-hand side of Equation 1 is maximized when for
each $m\in\\{1,\cdots,M\\}$, the prior $h(\bm{z}_{m}|\bm{X})$ is equal to the
aggregated posterior of a multimodal sample given on the first line of
Equation 2.

###### Proof.

Since the cross-entropy term in Equation 1 involves an expectation over the
data $\bm{X}$ and both $q_{\phi}(\bm{z}\mid\bm{X})$ and $h(\bm{z}\mid\bm{X})$
depend on $\bm{X}$, we can prove the identity for a given value of $\bm{X}$.

We exploit the factorization of both the variational posterior and the prior
over the modalities. Interpreting the cross-entropy between the variational
distribution and prior as a functional $F$ of the prior $h$, we have

$\displaystyle F[h(\bm{z}|\bm{X}))]$
$\displaystyle\equiv\mathbb{E}_{q_{\phi}(\bm{z}|\bm{X})}\left[\log
h(\bm{z}|\bm{X}))\right]=\mathbb{E}_{\prod_{m=1}^{M}q_{\phi}^{m}(\bm{z}_{m}|\bm{x}_{m})}\left[\log\prod_{m=1}^{M}h(\bm{z}_{m}|\bm{X}))\right]$
$\displaystyle=\sum_{m=1}^{M}\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}|\bm{x}_{m})}\left[\log
h(\bm{z}_{m}|\bm{X}))\right]=M\cdot\mathbb{E}_{\frac{1}{M}\sum_{\tilde{m}=1}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}|\bm{x}_{\tilde{m}})}\left[\log
h(\bm{z}_{m}|\bm{X}))\right].$

As a result, we see that $F[h(\cdot)]$ is an expectation over a mixture
distribution. We can solve for the optimal distribution $h(\cdot)$ by adding a
Lagrange multiplier that enforces $h(\cdot)$ normalizes to one:

$\displaystyle\max_{h(\bm{z}_{m}\mid\bm{X})}F[h(\bm{z}_{m}\mid\bm{X})]+\gamma\left(\int
h(\bm{z}_{m}\mid\bm{X})d\bm{z}_{m}-1\right)=\max_{h(\bm{z}_{m}\mid\bm{X})}\mathcal{L}[h,\gamma]$

To maximize the Lagrange functional $\mathcal{L}[h,\gamma]$, we compute its
(functional) derivatives with respect to $h(\bm{z}_{m}|\bm{X})$ and $\gamma$.

$\displaystyle\frac{\partial\mathcal{L}[h(\bm{z}_{m}|\bm{X}),\gamma]}{\partial
h(\bm{z}_{m}|\bm{X})}=$
$\displaystyle~{}\frac{\frac{1}{M}\sum_{\tilde{m}=1}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}|\bm{X})}{h(\bm{z}_{m}|\bm{X})}+\gamma\overset{!}{=}0$
$\displaystyle\frac{\partial\mathcal{L}[h(\bm{z}_{m}|\bm{X}),\gamma]}{\partial\gamma}=$
$\displaystyle~{}\int_{\bm{z}_{m}}h(\bm{z}_{m}|\bm{X})d\bm{z}_{m}-1\overset{!}{=}0$

The first condition implies that for _any_ value of $\bm{z}_{m}$, the ratio
between the mixture distribution and the prior is constant, while the second
condition demands that the prior be normalized. These conditions can only be
met if the prior _equals_ the mixture distribution, which proves the claim. ∎

(a) LR: translated PoyMNIST

(b) LR: Bimodal CelebA

(c) LR: CUB

(d) Coh: translated PoyMNIST

(e) Coh: Bimodal CelebA

(f) Coh: CUB

Figure 2:  Results on the benchmark datasets translated PolyMNIST, bimodal
CelebA, and CUB. An optimal model would be in the top right corner with low
reconstruction error and high classification performance. The proposed MMVM
method either achieves a higher classification performance, latent
representation (LR, Figures 2(a), 2(b) and 2(c)) or coherence of generated
samples (Coh, Figures 2(d), 2(e) and 2(f)), with the same reconstruction loss
or the same classification performance with lower reconstruction loss. Every
point averages runs over multiple seeds and a specific $\beta$ value (see
Section 5.1).

##  5 Experiments

We evaluate the proposed MMVM VAE on three benchmark datasets and two
challenging real-world applications.

###  5.1 Benchmark Datasets

We first compare the proposed method against five strong VAE-based learning
approaches on three frequently used multimodal benchmark datasets.

Datasets. We perform benchmark experiments on the translated PolyMNIST
(Daunhawer et al., 2022; Sutter et al., 2021), the bimodal CelebA (Sutter et
al., 2020), and the CUB image-captions (Shi et al., 2019) dataset. The
translated PolyMNIST dataset uses multiple instances of the MNIST dataset
(LeCun et al., 1998) with different backgrounds but shared digits. The digits
of the different modalities are randomly translated such that we cannot
predict their location across modalities. Bimodal CelebA extends the CelebA
dataset (Liu et al., 2015) with an additional text modality based on the
attributes describing the faces. Similarly, the CUB image-captions dataset
extends the Caltech bird dataset (Wah et al., 2011) with human-generated
captions describing the images. Please see Appendix B for more details
regarding the datasets.

Baselines. We evaluate our proposed method against a set of jointly-trained
independent VAEs (independent, Kingma and Welling, 2014) and different
aggregation-based multimodal VAEs. For the set of independent VAEs, there is
no interaction or regularization between the different modalities during
training. For the aggregation-based multimodal VAEs, we use a multimodal VAE
with a joint posterior approximation function. We evaluate four different
aggregation functions: a simple averaging (AVG, Hosoya, 2018), a product-of-
experts (PoE, Wu and Goodman, 2018), a mixture-of-experts (MoE, Shi et al.,
2019), and a mixture-of-products-of-experts (MoPoE, Sutter et al., 2021). We
train all VAE methods as $\beta$-VAEs (Higgins et al., 2016), where $\beta$ is
an additional hyperparameter weighting the rate term $R$ of the VAE (see
Section 4). Section B.4 provides the implementation details of the proposed
method and the baseline alternatives.

Evaluation. We evaluate all methods in terms of their data imputation
performance, where we withhold a subset of modalities at test time and
conditionally generate them from the shared latent representations. In this
imputation task, we assess whether the generated modalities are both of high
quality and display the expected shared information, which we refer to as
_coherence_.

We test the methods’ ability to learn meaningful representation when only a
subset of modalities is available. We assess the quality of the learned latent
representations using linear classifiers trained on representations of the
training set and the coherence using nonlinear classifiers trained on original
samples of the training set. We use the reconstruction error as a proxy for
how well each method learns the underlying data distribution. We assess each
method by relating their achieved reconstruction error to either the learned
latent representation classification or the coherence (Figure 2). We evaluate
all methods for multiple values of $\beta$, where the average performance over
multiple seeds with a single $\beta$ value leads to a single point in Figure
2. We chose $\beta\in\\{2^{-8},\ldots,2^{3}\\}$ on the PolyMNIST dataset,
$\beta\in\\{2^{-5},\ldots,2^{4}\\}$ on the CelebA dataset, and
$\beta\in\\{2^{-2},\ldots,2^{2}\\}$ on the CUB dataset. This way, we can
assess the trade-off between accurately reconstructing data samples and
inferring shared information. In all figures, the arrows go from small to
large values of $\beta$. See Section B.2 for more details on the evaluation
metrics.

Results. Figure 2 shows that the proposed MMVM VAE consistently outperforms
the other VAE-based methods on all datasets and both tasks. We can show that
our method overcomes the limitations of aggregation-based multimodal VAEs on
translated PolyMNIST described in Daunhawer et al. (2020). Also, MMVM VAE can
learn meaningful representations and generate coherent samples across
different modalities without a trade-off in reconstruction quality for both
text-image datasets bimodal CelebA and CUB image-captions. The coherent
conditional generation is especially surprising as the proposed MMVM VAE
decoder is never confronted with a sample from another modality. For all
benchmark datasets, the proposed MMVM either achieves better latent
representation classification and coherence performance with a similar
reconstruction loss or lower reconstruction loss with a similar classification
performance than other multimodal VAE approaches. In summary, we can show that
the newly proposed MMVM VAE overcomes the limitations of previous aggregation-
based approaches to multimodal learning (see Section 2) and outperforms
previous works on all three benchmark datasets. We provide more results on the
benchmark datasets in Appendix B.

(a) Latent Representation Classification

(b) Conditional Generation Coherence

Figure 3:  Results based on a memory experiment conducted on five rats, each
regarded as a separate modality. We report the performance of the latent
representation classification and the conditional generation coherence against
the reconstruction loss for different $\beta$ values for the different VAE
methods. Every point in the figures represents a specific $\beta$ value, where
$\beta=(10^{-5},10^{-4},10^{-3},2.5\times 10^{-3},5\times 10^{-3},10^{-2})$.
An optimal model would be in the top right corner.

###  5.2 Hippocampal Neural Activities

Dataset. Temporal organization is crucial to memory, affecting various
perceptual, cognitive, and motor processes. While we have made progress in
understanding the brain’s processing of the spatial context of memories, our
knowledge of their temporal structure is still very limited. To this end,
neuroscientists have recorded neural activity in the hippocampus of rats
performing a complex sequence memory task (Allen et al., 2016; Shahbaba et
al., 2022). More specifically, this study investigates the temporal
organization of memory and behavior by recording neural activity from the
dorsal CA1 region of the hippocampus. Briefly, the task involves presenting
rats with a repeated sequence of non-spatial events (four stimuli: odors A, B,
C, D) at a single port (Shahbaba et al., 2022). Since the same experimental
setup was conducted across all rats, we consider the rats as different
”modalities” and apply our proposed MMVM method to extract meaningful latent
representations. While the existence (and importance) of subject-specific
effects is well-known in neuroscience, such effects tend to be treated as
unexplained variance because of the lack of the required analytical tools to
extract and utilize this information properly.

(a) independent

(b) AVG

(c) MoPoE

(d) MMVM

Figure 4: Latent neural representation during a memory experiment. Each
model’s performance is evaluated based on its own optimal $\beta$ value
(0.00001, 0.01, 0.00001, 0.001 for independent, AVG, MoPoE, and MMVM
respectively) in terms of the unimodal latent representation classification
accuracy according to Figure 3(a). Our method can distinguish the odor stimuli
in the latent space with a clear separation of odors similar to MoPoE VAE (4
different colors). Conversely, unimodal and AVG models failed to combine
multi-views as the odor separation only occurred within single views.

Results. Our proposed MMVM method outperforms most previous works regarding
learned latent representations and conditional generation coherence. Only the
MoPoE VAE achieves a classification performance comparable to the MMVM method
but with a higher reconstruction loss. Figure 3(a) shows the separation of the
latent representation (measured by the accuracy of a multinomial logistic
regression classifier) against the reconstruction loss. Similar to the results
on the benchmark datasets, the proposed MMVM VAE outperforms previous works by
providing a clear separation of odors in the latent space while maintaining a
low reconstruction loss. Figure 3(b) compares the coherence of conditional
generation accuracy against the reconstruction loss. As before, our proposed
approach outperforms the alternatives. The proposed MMVM method allows
learning an aligned latent representation across different modalities. We show
the 2-dimensional latent representations for every rat and four VAE encoders
in Figure 4. Each dot is the two-dimensional latent representation of a 100 ms
sub-window of one odor trial for one rat and is colored according to its
ground truth odor value. Figure 4 shows the odor stimuli separation on the
latent space and how good MMVM VAE is in separating the odors. At the same
time, two baseline models fail to extract the shared information between rats.
Although it shows separation in some views, the independent model does not
provide a connection between views. The five tiny clusters in Figure 4(b) show
that, instead of showing a clear odor separation on the latent space, the AVG
VAE separated the data by rats. In other words, the five rats’ latent
representations were far from each other, so the AVG VAE failed to connect the
five views. See also Section B.8 for more results.

###  5.3 MIMIC-CXR

Dataset. To assess the performance of our approach in a real-world setting, we
evaluate the proposed MMVM method on the automated analysis of chest X-rays, a
common and critical medical task. For this purpose, we use the MIMIC-CXR
dataset (Johnson et al., 2019b), a well-established and extensive collection
of chest X-rays. The dataset reflects real clinical challenges with varying
image quality due to technical issues, patient positioning, and obstructions.
The dataset includes different views, which provide complementary information
valuable for improving diagnostic (Raoof et al., 2012). In this work, we
consider frontal and lateral images as two modalities (see Section B.7 for
further details). Each set of X-rays is labeled with different cardiopulmonary
conditions, which have been automatically extracted from the associated
radiology reports (Irvin et al., 2019). This results in instances with
incomplete label sets (Haque et al., 2023), which presents a challenge for
fully supervised approaches and motivates the need for self-supervised methods
instead.

Table 1: VAE latent representation quality evaluation. Average AUROC [in %] over three seeds of the two unimodal latent representations ($z_{F}$ and $z_{L}$) on a subset of MIMIC-CXR labels. The latent representations learned by the MMVM VAE lead to better classification performance compared to the other VAEs and are competitive with the fully-supervised method. Full results in Appendix B.7.3. |  All labels |  No Finding |  Cardiomegaly | Edema | Lung Lesion | Pneumonia
---|---|---|---|---|---|---
supervised |  70.5 $\pm$11.8 |  73.0 $\pm$1.4 |  80.3 $\pm$1.4 | 87.1 $\pm$0.9 | 54.8 $\pm$2.5 | 61.3 $\pm$0.4
independent |  68.0 $\pm$8.2 |  75.3 $\pm$1.4 |  73.5 $\pm$2.8 | 79.2 $\pm$3.9 | 60.1 $\pm$1.2 | 55.8 $\pm$0.8
PoE |  70.4 $\pm$8.2 |  75.9 $\pm$1.3 |  76.7 $\pm$1.9 | 81.8 $\pm$2.7 | 61.3 $\pm$2.1 | 57.8 $\pm$0.4
MMVM |  73.1 $\pm$8.6 |  78.7 $\pm$0.4 |  79.6 $\pm$0.9 | 85.3 $\pm$1.0 | 63.6 $\pm$0.7 | 59.5 $\pm$0.7

Results. We assess the quality of the unimodal latent representations of the
MMVM VAE by comparing them to those learned by independent VAEs and the PoE
VAE (see Section 5.1). We do this by training binary random forest classifiers
independently for each method and label on the inferred representations of a
subset of the training set. Table 1 shows the AUROC for these classifiers,
averaged over three seeds and both unimodal representations for a subset of
labels. In addition, we also report the performance of a deep nonlinear
network trained in a fully supervised manner on the same train/test split for
reference purposes. Detailed experiment information can be found in Section
B.7.3, with extensive results for each modality and label available in Table 2
and Table 3. Overall, our approach shows performance improvements across all
labels compared to the other VAEs and is highly competitive with the fully-
supervised method, surpassing it in average performance over all labels.
Examining each unimodal representation separately provides further insights
into the VAEs’ ability to leverage information from other modalities during
training. For example, in the Cardiomegaly prediction task, the MMVM VAE’s
lateral representations $z_{L}$ slightly outperform the PoE VAE’s frontal
representations $z_{F}$ (MMVM $z_{L}$: 78.7%, PoE $z_{F}$: 78.5%), even though
the lateral modality seems generally less informative (supervised $x_{L}$:
79.0%, $x_{F}$: 81.7%) for this task. The same observation can be made for
other labels (see detailed results and discussion in Section B.7.3). This
illustrates the MMVM VAE’s ability to soft-share information between modality-
specific latent representations during training, thereby enhancing the
representation quality of the weaker modality.

##  6 Broader Impact & Limitations

This paper aims to advance the field of Machine Learning by providing a
natural and fundamental solution for integrating data across modalities. The
proposed approach can be applied to various scientific and engineering
problems with a potentially significant societal impact. In the field of
neuroscience specifically, our method could allow neuroscientists to leverage
individual differences in brain activity and behavior to understand basic
information processing in the brain, as well as to capture distinct
longitudinal changes to understand how it is affected in disease. In
translational research, it could help identify subjects more susceptible to
disease or potentially more responsive to treatment. While we can show that
the proposed method learns better representations and generates more coherent
samples, we cannot directly generate random samples anymore (see also Section
4). Although we show results on two real-world datasets, Sections 5.3 and 5.2,
additional experiments on even larger scale multimodal datasets would help
further evaluate the proposed method, e.g. (Damen et al., 2018; Wu et al.,
2023). However, training and evaluating our methods on such datasets requires
immense computing resources.

##  7 Conclusion

In this work, we have presented a new multimodal VAE, called MMVM VAE, which
is based on a data-dependent multimodal variational mixture-of-experts prior.
By focusing on a multimodal prior, the proposed MMVM VAE overcomes the
limitations of previous methods with over-restrictive definitions of joint
posterior approximations. In extensive experiments, we show that MMVM VAE
outperforms previous works on three different benchmark datasets in terms of
learned latent representations as well as generative quality and coherence of
missing modalities. We also demonstrate its efficacy on two challenging real-
world applications and show improved performance compared to previous VAEs and
even a fully-supervised approach. Future research could involve applying
similar ideas to more powerful multimodal generative models, and studying the
representation-distortion tradeoff from an information-theoretical perspective
(Yang and Mandt, 2022; Yang et al., 2023).

## References

  * Allen et al. [2016] T. A. Allen, D. M. Salz, S. McKenzie, and N. J. Fortin.  Nonspatial sequence coding in CA1 neurons.  _Journal of Neuroscience_ , 36(5):1547–1563, 2016\.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.2874-15.2016.  URL http://www.jneurosci.org/content/36/5/1547.
  * Baltrušaitis et al. [2018] T. Baltrušaitis, C. Ahuja, and L.-P. Morency.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443, 2018.
  * Bhattacharyya et al. [2019] A. Bhattacharyya, M. Hanselmann, M. Fritz, B. Schiele, and C. Straehle.  Conditional flow variational autoencoders for structured sequence prediction.  _CoRR_ , abs/1908.09008, 2019.  URL http://arxiv.org/abs/1908.09008.
  * Bouchacourt et al. [2018] D. Bouchacourt, R. Tomioka, and S. Nowozin.  Multi-level variational autoencoder: Learning disentangled representations from grouped observations.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * Damen et al. [2018] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * Daunhawer et al. [2020] I. Daunhawer, T. M. Sutter, R. Marcinkevics, and J. E. Vogt.  Self-supervised Disentanglement of Modality-specific and Shared Factors Improves Multimodal Generative Models.  _German Conference on Pattern Recognition_ , 2020.
  * Daunhawer et al. [2022] I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, and J. E. Vogt.  On the Limitations of Multimodal VAEs.  _International Conference on Learning Representations_ , 2022.
  * Efron [2012] B. Efron.  _Large-scale inference: empirical Bayes methods for estimation, testing, and prediction_ , volume 1.  Cambridge University Press, 2012.
  * Falcon and The PyTorch Lightning team [2019] W. Falcon and The PyTorch Lightning team.  PyTorch Lightning, Mar. 2019.  URL https://github.com/Lightning-AI/lightning.
  * Ghosh et al. [2019] P. Ghosh, M. S. Sajjadi, A. Vergari, M. Black, and B. Schölkopf.  From variational to deterministic autoencoders.  _arXiv preprint arXiv:1903.12436_ , 2019.
  * Haque et al. [2023] M. I. U. Haque, A. K. Dubey, I. Danciu, A. C. Justice, O. S. Ovchinnikova, and J. D. Hinkle.  Effect of image resolution on automated classification of chest X-rays.  _Journal of Medical Imaging_ , 10(4):044503–044503, 2023.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Higgins et al. [2016] I. Higgins, L. Matthey, A. Pal, C. Burgess, and X. Glorot.  beta-vae: Learning basic visual concepts with a constrained variational framework.  2016\.  URL https://openreview.net/forum?id=Sy2fzU9gl.
  * Hosoya [2018] H. Hosoya.  A simple probabilistic deep generative model for learning generalizable disentangled representations from grouped data.  _CoRR_ , abs/1809.0, 2018.
  * Irvin et al. [2019] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al.  CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 590–597, 2019.
  * Johnson et al. [2019a] A. Johnson, M. Lungren, Y. Peng, Z. Lu, R. Mark, S. Berkowitz, and S. Horng.  MIMIC-CXR-JPG-chest radiographs with structured labels.  _PhysioNet_ , 2019a.
  * Johnson et al. [2019b] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1):317, 2019b.
  * Joy et al. [2021] T. Joy, Y. Shi, P. H. S. Torr, T. Rainforth, S. M. Schmon, and N. Siddharth.  Learning multimodal vaes through mutual supervision.  _CoRR_ , abs/2106.12570, 2021.  URL https://arxiv.org/abs/2106.12570.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Welling [2014] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  In _International Conference on Learning Representations_ , 2014\.
  * LeCun et al. [1998] Y. LeCun, C. Cortes, and C. J. Burges.  Gradient-based learning applied to document recognition.  In _Proceedings of the IEEE_ , volume 86, pages 2278–2324, 1998\.
  * Liang et al. [2022] P. P. Liang, A. Zadeh, and L.-P. Morency.  Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions.  _arXiv preprint arXiv:2209.03430_ , 2022.
  * Lin [1991] J. Lin.  Divergence measures based on the Shannon entropy.  _IEEE Transactions on Information Theory_ , 37(1):145–151, Jan. 1991.  ISSN 1557-9654.  doi: 10.1109/18.61115.  URL https://ieeexplore.ieee.org/document/61115.
  * Liu et al. [2015] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep Learning Face Attributes in the Wild.  In _The IEEE International Conference on Computer Vision (ICCV)_ , 2015.
  * Locatello et al. [2020] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen.  Weakly-supervised disentanglement without compromises.  In _International Conference on Machine Learning_ , pages 6348–6359. PMLR, 2020.
  * Mahajan et al. [2020] S. Mahajan, I. Gurevych, and S. Roth.  Latent normalizing flows for many-to-many cross-domain mappings.  _CoRR_ , abs/2002.06661, 2020.  URL https://arxiv.org/abs/2002.06661.
  * Manduchi et al. [2024] L. Manduchi, K. Pandey, R. Bamler, R. Cotterell, S. Däubener, S. Fellenz, A. Fischer, T. Gärtner, M. Kirchler, M. Kloft, et al.  On the challenges and opportunities in generative ai.  _arXiv preprint arXiv:2403.00025_ , 2024.
  * Oord et al. [2019] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation Learning with Contrastive Predictive Coding, Jan. 2019.  URL http://arxiv.org/abs/1807.03748.
  * Palumbo et al. [2023] E. Palumbo, I. Daunhawer, and J. E. Vogt.  MMVAE+: Enhancing the generative quality of multimodal vaes without compromises.  In _The Eleventh International Conference on Learning Representations_. OpenReview, 2023.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever.  Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021.  URL http://arxiv.org/abs/2103.00020.
  * Ramesh et al. [2021] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831, 2021.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Raoof et al. [2012] S. Raoof, D. Feigin, A. Sung, S. Raoof, L. Irugulpati, and E. C. Rosenow III.  Interpretation of plain chest roentgenogram.  _Chest_ , 141(2):545–558, 2012.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  _CoRR_ , abs/2205.1, 2022.  doi: 10.48550/arXiv.2205.11487.  URL https://doi.org/10.48550/arXiv.2205.11487.
  * Shahbaba et al. [2022] B. Shahbaba, L. Li, F. Agostinelli, M. Saraf, K. W. Cooper, D. Haghverdian, G. A. Elias, P. Baldi, and N. J. Fortin.  Hippocampal ensembles represent sequential relationships among an extended sequence of nonspatial events.  _Nature Communications_ , 13(1):787, Feb. 2022\.  ISSN 2041-1723.  doi: 10.1038/s41467-022-28057-6.  URL https://www.nature.com/articles/s41467-022-28057-6.
  * Shi et al. [2019] Y. Shi, N. Siddharth, B. Paige, and P. Torr.  Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models.  In _Advances in Neural Information Processing Systems_ , pages 15692–15703, 2019.
  * Sønderby et al. [2016] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther.  Ladder variational autoencoders.  _Advances in neural information processing systems_ , 29, 2016.
  * Sutter [2023] T. M. Sutter.  _Imposing and Uncovering Group Structure in Weakly-Supervised Learning_.  Doctoral Thesis, ETH Zurich, 2023.  URL https://www.research-collection.ethz.ch/handle/20.500.11850/634822.
  * Sutter et al. [2020] T. M. Sutter, I. Daunhawer, and J. E. Vogt.  Multimodal Generative Learning Utilizing Jensen-Shannon Divergence.  _Advances in Neural Information Processing Systems_ , 2020.  URL https://arxiv.org/abs/2006.08242.
  * Sutter et al. [2021] T. M. Sutter, I. Daunhawer, and J. E. Vogt.  Generalized Multimodal ELBO.  _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=5Y21V0RDBV.
  * Sutter et al. [2023a] T. M. Sutter, L. Manduchi, A. Ryser, and J. E. Vogt.  Learning Group Importance using the Differentiable Hypergeometric Distribution.  In _International Conference on Learning Representations_ , 2023a.
  * Sutter et al. [2023b] T. M. Sutter, A. Ryser, J. Liebeskind, and J. E. Vogt.  Differentiable Random Partition Models.  In _Advances in Neural Information Processing Systems_ , 2023b.
  * Suzuki and Matsuo [2022] M. Suzuki and Y. Matsuo.  A survey of multimodal deep generative models.  _Advanced Robotics_ , 36(5-6):261–278, 2022.
  * Tian et al. [2020] Y. Tian, D. Krishnan, and P. Isola.  Contrastive Multiview Coding, Dec. 2020.  URL http://arxiv.org/abs/1906.05849.
  * Tomczak and Welling [2017] J. M. Tomczak and M. Welling.  VAE with a VampPrior.  _arXiv preprint arXiv:1705.07120_ , 2017.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  Caltech-UCSD Birds-200-2011 (CUB-200-2011).  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Wu and Goodman [2018] M. Wu and N. Goodman.  Multimodal Generative Models for Scalable Weakly-Supervised Learning.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_ , pages 5580–5590, Feb. 2018.  URL http://arxiv.org/abs/1802.05335.
  * Wu et al. [2023] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua.  NExT-GPT: Any-to-Any Multimodal LLM, 2023.
  * Yang and Mandt [2022] Y. Yang and S. Mandt.  Towards empirical sandwich bounds on the rate-distortion function.  In _International Conference on Learning Representations_ , 2022.
  * Yang et al. [2023] Y. Yang, S. Eckstein, M. Nutz, and S. Mandt.  Estimating the rate-distortion function by wasserstein gradient descent.  _Advances in Neural Information Processing Systems_ , 36, 2023.

##  Appendix A MMVM VAE

###  A.1 Bound on the proposed objective

The objective $\mathcal{E}(\bm{x}_{m})$ for a single modality $m$ is given by

$\displaystyle\mathcal{E}(\bm{x}_{m})=$
$\displaystyle~{}\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log
p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})\right]-KL\left[q_{\phi}(\bm{z}_{m}\mid\bm{x}_{m})\mid\mid
p_{\theta}(\bm{z}_{m})\right]$ (3) $\displaystyle\leq$
$\displaystyle~{}\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log
p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})\right]$ (4) $\displaystyle\leq$
$\displaystyle~{}\log p_{\theta}(\bm{x}_{m}\mid\bm{\mu}_{m}(\bm{x}_{m})),$ (5)

where $\bm{\mu}_{m}(\bm{x}_{m})=f_{m}(\bm{x}_{m})$ is the output of the
optimized (mean) encoder $f_{m}(\cdot)$ of modality $m$. Equation 4 follows
from the non-negativity of the KL divergence. Without regularization term, the
maximizing distribution is a delta distribution with zero variance (Equation
5). Equation 5 equals the maximum-likelihood version of the proposed MMVM VAE
(for a single modality). Put differently, the MSE of a "vanilla" autoencoder
is an upper bound on the objective $\mathcal{E}(\bm{x}_{m})$ for any prior
distribution $p_{\theta}(\bm{z}_{m})$, hence, also for the newly introduced
MMVM prior distribution $h(\bm{z}_{m}\mid\bm{X})$.

Let us have a look at when the KL term actually vanishes given a MMVM prior
distribution $h(\bm{z}_{m}\mid\bm{X})$. The KL term can only vanish if all
posterior approximation
$q_{\phi}^{\tilde{m}}(\bm{z}_{m}\mid\bm{x}_{\tilde{m}})$ map to a single mode
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$. In that case

$\displaystyle
h(\bm{z}_{m}\mid\bm{X})=\frac{1}{M}\sum_{\tilde{m}=1}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}\mid\bm{x}_{\tilde{m}})=q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$
(6)

and $KL\left[q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})\mid\mid
h(\bm{z}_{m}\mid\bm{X})\right]=0$.

Another scenario is when $q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$ and
$h(\bm{z}_{m}\mid\bm{x}_{m})$ have disjoint modes. Hence,
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$ is only a match to itself. In this
case, we have

$\displaystyle\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log\left(\frac{1}{M}\sum_{\tilde{m}=1}^{M}q_{\phi}^{\tilde{m}}(\bm{z}_{m}\mid\bm{x}_{\tilde{m}})\right)\right]\approx$
$\displaystyle~{}\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log\left(\frac{1}{M}q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})\right)\right]$
(7) $\displaystyle=$ $\displaystyle~{}-\log
M+\mathbb{E}_{q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})}\left[\log
q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})\right]$ (8)

So, our objective will still be (up to a constant) the maximum likelihood
objective, leading the variances to shrink to zero. In this case, the
objective will reduce to the limiting independent VAE, where the modalities do
not "talk" to each other and there is no multimodal alignment.

In the most interesting case, there will be a non-trivial overlap between
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$ and $h(\bm{z}_{m}\mid\bm{x}_{m})$, i.
e. between the different unimodal posterior approximations, leading to a
multimodal alignment through the soft sharing that we wish to see.

In addition, Figure 5 empirically shows that the negative mean squared error
(MSE) of the vanilla autoencoder (AE) upper bounds the proposed objective
$\mathcal{E}$. We show that lowering the $\beta$ value of the regularizer $R$
(see Section 4) approximates the negative MSE of the vanilla AE.

Figure 5:  We compare the achieved values of the proposed objective
$\mathcal{E}$ to the vanilla Autoencoder’s negative mean squared error (MSE).
Lowering the $\beta$ value of the regularizer $R$ in the objective (see
Section 4) approximates the negative MSE bound provided by the vanilla AE.
This proves empirically that the negative MSE of the vanilla AE indeed upper
bounds the proposed objective $\mathcal{E}$.

##  Appendix B Experiments

###  B.1 Dataset Licences

  * •

PolyMNIST: originally published in [Sutter et al., 2021], downloaded the data
from https://drive.google.com/drive/folders/1lr-
laYwjDq3AzalaIe9jN4shpt1wBsYM?usp=sharing and the code from
https://github.com/thomassutter/MoPoE, published under the MIT license

  * •

Bimodal CelebA: originally published in [Sutter et al., 2020], downloaded from
https://drive.google.com/drive/folders/1lr-
laYwjDq3AzalaIe9jN4shpt1wBsYM?usp=sharing, licensed under the MIT license. The
original CelebA dataset was published in [Liu et al., 2015], license not
found.

  * •

CUB image-captions: originally published in [Shi et al., 2019], downloaded
from http://www.robots.ox.ac.uk/~yshi/mmdgm/datasets/cub.zip, licensed under
GPL 3.0.

  * •

MIMIC-CXR: originally published in [Johnson et al., 2019b], downloaded from
https://physionet.org/content/mimic-cxr/2.0.0/, licensed under PhysioNet
Credentialed Health Data License 1.5.0 (see
https://physionet.org/content/mimic-cxr/view-license/2.0.0/).

  * •

MIMIC-CXR-JPG: originally published in [Johnson et al., 2019a], downloaded
from https://physionet.org/content/mimic-cxr-jpg/2.1.0/, licensed under
PhysioNet Credentialed Health Data License 1.5.0 (see
https://physionet.org/content/mimic-cxr-jpg/2.1.0/).

  * •

Hippocampal Neural Activity data: originally published in [Shahbaba et al.,
2022], downloaded from https://datadryad.org/stash/dataset/doi:10.7280/D14X30,
licensed under CC0 1.0 Universal (CC0 1.0) Public Domain Dedication (see
https://creativecommons.org/publicdomain/zero/1.0/)

###  B.2 Evaluation Details

We evaluate the different methods based on the coherence of their imputed
samples, the quality of their latent representations, and their reconstruction
error. We assume access to the full set of modalities during training, but we
do not make this assumption at test time. Hence, there is a need for methods
that can conditionally generate samples of these missing modalities, given the
available modalities. In other words, we want to be able to impute missing
modalities. Imputed modalities should not only be of high generative quality
but also display the same shared information as the available modalities
(i.e., be coherent). For example, if we generate a sample of modality
$\bm{x}_{2}$ based on modality $\bm{x}_{1}$ from the PolyMNIST dataset (see
Figure 6), the generated sample of modality $\bm{x}_{2}$ should contain the
same digit information as modality $\bm{x}_{1}$ but show the background of
modality $\bm{x}_{2}$.

####  B.2.1 Latent Representation Evaluation

We assess the learned representations based on subsets of modalities and not
the full set. The quality of the representations serves as a proxy of how
useful the learned representations are for additional downstream tasks that
are not part of the training objective. Hence, high-quality representations of
subsets of modalities are also the basis for conditionally generating coherent
samples. We assess their quality by using the classification performance of
linear classifiers. We train independent classifiers on the unimodal
representations of the training set and evaluate them on unimodal test set
representations. To assess the learned latent representation, we train a
logistic regression classifier on $10000$ latent representations of the
training set. The accuracy is computed on all representations of the test set.

####  B.2.2 Coherence Evaluation

The coherence of conditionally generated samples shows how well the content of
the imputed modalities aligns with the content of the available modalities in
terms of the shared information. We evaluate the coherence using ResNet-based
classifiers [He et al., 2016] that are trained on samples from the original
training set of every modality. Using the described procedure, generated
samples have to be visually similar to the original samples to have high
coherence. Otherwise, the nonlinear classifier will not be able to predict
digits correctly.

To compute the coherence of conditionally generated samples, we train
additional deep classifiers on original samples of the training set. We use a
ResNet-based non-linear classifier that is trained on the full original
training set. The prediction of this classifier is then used to determine the
class (on PolyMNIST: digit) of the conditionally generated sample of a missing
modality. The nonlinear classifier reaches an accuracy of above 98% on the
original test set. For other datasets, the nonlinear classifier is trained to
predict the shared information of the multimodal dataset. Hence, it serves as
a good oracle for determining the digit of generated samples.

##### Conditional Generation with the MMVM VAE and the independent VAEs

To generate modality $\bm{x}_{\tilde{m}}$ conditioned on modality
$\bm{x}_{m}$, we proceed as follows:

  1. 1.

We encode modality $\bm{x}_{m}$ using the encoder
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$

  2. 2.

We sample a latent vector $\bm{z}\sim q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$

  3. 3.

We input the latent representation $\bm{z}$ into the decoder
$p_{\theta}^{\tilde{m}}(\bm{x}_{\tilde{m}}\mid\bm{z})$ of modality
$\bm{x}_{\tilde{m}}$

To compute the coherence numbers reported, we perform the above steps for
every modality $\bm{x}_{m}$ where $m\in\\{1,\ldots,M\\}$ and average the
achieved coherence accuracies.

##### Conditional Generation with the aggregated multimodal VAEs

All aggregation-based multimodal VAEs conditionally generate samples in the
same way. Hence, the following procedure applies to the multimodal VAEs used
in Section 5 ([AVG, Hosoya, 2018], [PoE, Wu and Goodman, 2018], [MoE, Shi et
al., 2019], and [MoPoE, Sutter et al., 2021]). To generate modality
$\bm{x}_{\tilde{m}}$ conditioned on modality $\bm{x}_{m}$, we proceed as
follows:

  1. 1.

We encode modality $\bm{x}_{m}$ using the encoder
$q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$

  2. 2.

We sample a latent vector $\bm{z}\sim q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})$

  3. 3.

We input the latent representation $\bm{z}$ into the decoder
$p_{\theta}^{\tilde{m}}(\bm{x}_{\tilde{m}}\mid\bm{z})$ of modality
$\bm{x}_{\tilde{m}}$

If we would have access to a multimodal subset $\bm{X}_{A}$ consisting of more
than one modality, i.e. $|A|>1$, we would have the joint posterior
approximation distribution $q_{\phi}(\bm{z}\mid\bm{X}_{A})$ of the subset
$\bm{X}_{A}$, sample a latent vector from that distribution, i.e. $\bm{z}\sim
q_{\phi}(\bm{z}\mid\bm{X}_{A})$, and generate modality $\bm{x}_{\tilde{m}}$
using the decoder $p_{\theta}^{\tilde{m}}(\bm{x}_{\tilde{m}}\mid\bm{z})$ of
modality $\bm{x}_{\tilde{m}}$

####  B.2.3 Generative Quality

The reconstruction error is a proxy for how well every method learns the
underlying data distribution. We evaluate the different VAE models by their
achieved reconstruction error against either the learned latent representation
classification or coherence (e.g., Figure 2). We do this for multiple values
of $\beta$, where the average performance over multiple seeds with a single
$\beta$ value leads to a scatter point in the figures. This way, we can assess
the trade-off between accurately reconstructing data samples and inferring
shared information. In addition, we compute the FID [Heusel et al., 2017]
values for modalities that can be summarized as natural images in the
appendix. A low FID score correlates with a high-quality generated image.

###  B.3 Implementation (General)

We use the scikit-learn [Pedregosa et al., 2011] package for the linear
classifiers to evaluate the learned latent representations. All code is
written using Python 3.11, PyTorch [Paszke et al., 2019] and Pytorch-Lightning
[Falcon and The PyTorch Lightning team, 2019]. We base the implementations of
the aggregation-based VAE methods on the official implementations. Hence, we
base our implementations on the following repositories:

  * •

AVG: https://github.com/HaruoHosoya/gvae

  * •

PoE: https://github.com/mhw32/multimodal-vae-public

  * •

MoE: https://github.com/iffsid/mmvae

  * •

MoPoE: https://github.com/thomassutter/MoPoE

Figure 6:  PolyMNIST (translated, scale=75%): every column is a multimodal
tuple $\bm{X}$, and every row shows samples of a single modality $\bm{x}_{m}$.
We see the random translation between samples by looking at images from a
single row or column.

###  B.4 PolyMNIST

####  B.4.1 Dataset

The dataset is based on the original MNIST dataset [LeCun et al., 1998].
Compared to the original dataset, the digits are scaled down by a factor of
$0.75$ such that there is more space for the random translation. In its
original form, the PolyMNIST consists of 5 different modalities. We only use
the first three modalities in this work. The background of every modality
$\bm{x}_{m}$ consists of random patches of size $28\times 28$ from a large
image. The digit is placed at a random position of the patch. We refer to the
original publication [Sutter et al., 2021] for details on the background
images. Using this setup, every modality has modality-specific information
given by its background image and shared information given by the digit, which
is shared between all modalities. An additional difficulty compared to the
original PolyMNIST is the random translation of the digits. The dataset can be
found at https://github.com/thomassutter/MoPoE.

####  B.4.2 Implementation & Training

We use the same network architectures for all multimodal VAEs. Every
multimodal VAE consists of a ResNet-based encoder and a ResNet-based Decoder
[He et al., 2016]. All modalities share the same architecture but are
initialized differently. We assume Gaussian distribution for all unimodal
posterior approximations, i.e.

$\displaystyle
q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})=\mathcal{N}(\bm{z}_{m};\bm{\mu}_{m},\bm{\sigma}_{m}\bm{I}),$
(9)

where the parameters $\bm{\mu}_{m}$ and $\bm{\sigma}_{m}$ are inferred using
neural networks such that we have

$\displaystyle
q_{\phi}^{m}(\bm{z}_{m}\mid\bm{x}_{m})=q_{\phi}^{m}(\bm{z}_{m};\bm{\mu}_{m}(\bm{x}_{m}),\bm{\sigma}_{m}(\bm{x}_{m}))=\mathcal{N}(\bm{z}_{m};\bm{\mu}_{m}(\bm{x}_{m}),\bm{\sigma}_{m}(\bm{x}_{m}))$
(10)

The conditional data distributions $p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})$ are
modeled using the Laplace distribution, where the location parameter is
modeled with a neural network (decoder) and the scale parameter is set to 0.75
[Shi et al., 2019], i.e.

$\displaystyle
p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})=\mathcal{L}(\bm{x}_{m};\bm{\mu}_{m},\bm{\sigma}_{m}),$
(11)

where $\mathcal{L}(\cdot)$ defines a Laplace distribution. It follows that

$\displaystyle
p_{\theta}(\bm{x}_{m}\mid\bm{z}_{m})=p_{\theta}(\bm{x}_{m};\bm{\mu}_{m}(\bm{z}_{m}),\bm{\sigma}_{m})=\mathcal{L}(\bm{x}_{m};\bm{\mu}_{m}(\bm{z}_{m}),\bm{\sigma}_{m})$
(12)

We use the method of Hosoya [2018] for the implementation of the aggregated
VAE. In this approach, a simplistic version of the joint posterior
distribution is chosen where for Gaussian distribution joint posterior
approximation $\mathcal{N}(\bm{\mu}_{s},\bm{\sigma}_{s}\bm{I})$ we have the
following distribution parameters $\bm{\mu}_{s}$ and $\bm{\sigma}_{s}$

$\displaystyle\bm{\mu}_{s}=$
$\displaystyle~{}\frac{1}{M}\sum_{m=1}^{M}\bm{\mu}_{m}$ (13)
$\displaystyle\bm{\sigma}_{s}=$
$\displaystyle~{}\frac{1}{M}\sum_{m=1}^{M}\bm{\sigma}_{s}$ (14)

$\bm{\mu}_{m}$ and $\bm{\sigma}_{m}$ are the distribution parameters of the
unimodal posterior approximations
$\mathcal{N}(\bm{\mu}_{m},\bm{\sigma}_{m}\bm{I})$.

During training and evaluation, no weight-sharing takes place, i.e. every
modality has its own encoder and decoder. We use the same architectures as in
[Daunhawer et al., 2022]. For all experiments on this dataset, we use an Adam
optimizer [Kingma and Ba, 2014] with an initial learning rate of $0.0005$, and
a batch size of $256$. We train all models for $500$ epochs.

We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with
$4$ CPU workers and $16$ GB of memory. An average run takes around $7$ hours.
To train all methods used in this paper, we had to train $11\times 5\times
6=330$ different models: $11$ different $\beta$ values, $5$ different seeds,
and $6$ different methods. Hence, the total GPU compute time used to generate
the results for the PolyMNIST dataset is equal to $330\times 7=2310$ hours.
We—of course—also had to invest development GPU time, which we did not
measure.

####  B.4.3 Additional Results

We generated Figures 2(a) and 2(d) by plotting the classification accuracy of
a linear classifier, which we trained on the learned latent representation,
against the reconstruction error on the test set. For the learned latent
representation, we train a classifier on the unimodal latent representations.
For the aggregated VAE, this means that we train the classifier on samples of
the unimodal posterior approximations and not the joint posterior
approximations. Using this procedure, we test the different methods according
to their performance in case of missing data, e.g., we only have access to a
single modality instead of the full set at test time. For the reconstruction
loss, however, we computed the error given the full set of modalities. The
idea for Figures 2(a) and 2(d) is to compare the reconstruction quality (i.e.,
how well can we learn the data distribution?) against metrics that are related
to the "generative factors" of the data and relate the different modalities to
each other, i.e. the shared information of a multimodal dataset.

In Figure 7, we evaluate the performance of individual modalities in case of
missing modalities. For that, we reconstruct every modality if it was the only
modality available at test time. Hence, the modalities in the aggregated VAE
have to be reconstructed based on the unimodal posterior approximations and
not the joint posterior approximation. For the independent VAEs and the MMVM-
VAE, the reconstruction of a modality is only based on its own unimodal
posterior approximation. Hence, for the latter two methods, nothing changes in
this setting. The performance of the learned latent representation and the
coherence of generated samples are evaluated in the same way as in Figures
2(a) and 2(d).

Figure 7 shows that the reconstruction error of the aggregated VAE increases a
lot if every modality needs to reconstruct itself. Interestingly, we can see
that the "self-reconstruction error" (the x-axis in Figures 7(a) and 7(b))
decreases with an increasing $\beta$-value, which is different to the other
two methods and also different to the aggregated VAE’s behavior in Figures
2(a) and 2(d).

In addition, we also show the conditional FID values the different multimodal
VAEs achieve. Conditional FIDs come from the conditional generation of
modality $\bm{x}_{m}$ given $\bm{x}_{m}$. From all the conditionally generated
samples of modality $\bm{x}_{m}$, we then compute the conditional FID values.
Figure 8 shows the aggregated values of all conditional FIDs. We again show
the FID values in relation to the downstream task performance and the
conditional generation coherence. We see that only the proposed MMVM VAE
reaches FID values similar to the ones of a set of independent VAEs. However,
the set of independent VAEs cannot achieve the same latent representation
classification nor coherence as the proposed MMVM VAE.

(a) Latent Representation Classification

(b) Conditional generation coherence

Figure 7:  Results on the PolyMNIST dataset for different VAE methods. We
report the performance of the latent representation classification and the
conditional generation coherence against the conditional reconstruction loss
for different $\beta$ values. Every point in the figures above is the average
of five runs over different seeds and a specific $\beta$ value where
$\beta=2^{k}$ for $k\in\\{-8,\ldots,3\\}$. Different to Figures 2(a) and 2(d),
the x-axis is the sum of the self-reconstruction losses if only a single
modality is given as input. Hence, for the aggregated VAE methods, every
modality is decoded by its own unimodal posterior approximation instead of the
joint posterior approximation.

(a) Latent Representation Classification

(b) Conditional generation coherence

Figure 8:  Results on the PolyMNIST dataset for different VAE methods. We
report the performance of the latent representation classification and the
conditional generation coherence against the conditional FID values. Every
point in the figures above is the average of five runs over different seeds
and a specific $\beta$ value where $\beta=2^{k}$ for $k\in\\{-8,\ldots,3\\}$.
Different to Figures 2(a) and 2(d), the x-axis is not the reconstruction error
but the average FID value computed from conditionally generated samples.

(a) m0 $\rightarrow$ m0

(b) m0 $\rightarrow$ m1

(c) m0 $\rightarrow$ m2

Figure 9:  Qualitative results for the conditional generation task for the set
of unimodal VAEs.

(a) m0 $\rightarrow$ m0

(b) m0 $\rightarrow$ m1

(c) m0 $\rightarrow$ m2

Figure 10:  Qualitative results for the conditional generation task for the
aggregation-based multimodal VAE (AVG).

(a) m0 $\rightarrow$ m0

(b) m0 $\rightarrow$ m1

(c) m0 $\rightarrow$ m2

Figure 11:  Qualitative results for the conditional generation task for the
proposed MMVM VAE.

(a) Latent Representation Classification

(b) Conditional generation coherence

Figure 12:  Results on the bimodal CelebA dataset for the different VAE
methods. We report the performance of the latent representation classification
and the conditional generation coherence against the conditional FID values.
Every point in the figures above is the average of five runs over different
seeds and a specific $\beta$ value where $\beta=2^{k}$ for
$k\in\\{-5,\ldots,4\\}$. Unlike in Figures 2(a) and 2(d), the x-axis is not
the reconstruction error but the average FID value computed from conditionally
generated samples.

###  B.5 Bimodal CelebA

Figure 13:  Bimodal CelebA. Three samples of image-text pairs. To introduce
another level of difficulty to the text modality, we added a random
translation to the starting point of the text attributes.

####  B.5.1 Dataset

We take the dataset from Sutter et al. [2020]. It extends the original CelebA
dataset [Liu et al., 2015] to the multimodal setting by creating a second
modality based on the attributes. In the original dataset, every image comes
with a set of $40$ labels, such as blond hair, smiling, etc. The difficulty of
the individual attributes is not only due to their visual appearance but also
their frequency in the dataset. In the multimodal extension, Sutter et al.
[2021] created a text string from the attribute words. Unlike the label
vector, absent attributes are dismissed in the text string instead of negated.
Additionally, the attributes are randomly ordered.

####  B.5.2 Implementation & Training

We use ResNet-based encoders and decoders for this experiment as well [He et
al., 2016], similar to the ones in the PolyMNIST experiment. The image encoder
and decoder consist of 2D convolutions, while the text encoder and decoder
consist of 1D convolutions. We use a character-level encoding and not a word
or token-level encoding because of the synthetic nature of the text modality.
We also use an Adam optimizer [Kingma and Ba, 2014] with a starting learning
rate of $0.0002$ and a batch size of $128$. We train all models for 400 epochs
and $3$ seeds. The implementation follows the one described in Section B.4.

We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with
$4$ CPU workers and $16$ GB of memory. An average run takes around $24$ hours.
To train all methods used in this paper, we had to train $10\times 3\times
6=180$ different models: $10$ different $\beta$ values, $3$ different seeds,
and $6$ different methods. Hence, the total GPU compute time used to generate
the results for the PolyMNIST dataset equals $180\times 24\approx 4320$ hours.
We—of course—also had to invest GPU time to develop the method, which we did
not measure.

####  B.5.3 Additional Results

Given the multilabel nature of the CelebA dataset, we evaluate the learned
latent representation with respect to the individual attributes and not only
the average performance across all attributes. Figure 14 shows the detailed
results according to the full set of $40$ attributes for the three methods:
independent VAEs, aggregated VAE, and MMVM VAE. We again train linear binary
classifiers on inferred representations of the training set and evaluate them
on representations of the test set. However, we now report the individual
performance of every classifier. In the main text (see Figures 2(b) and 2(e)),
we report the average performance of the $40$ binary classifiers.

(a) Image Representations

(b) Text Representations

Figure 14:  Attribute-level results on the bimodal CelebA datasets for the
latent representation classification. The MMVM VAE outperforms the independent
VAEs and the aggregated VAE on most attributes.

(a) Independent VAEs

(b) AVG VAE

(c) MMVM VAE

Figure 15:  Qualitative Results for the CelebA dataset on the image-to-image
generation task. The first 10 rows of every subplot show the input image and
the bottom 10 rows its conditional generation. Different to the training, we
provide only the image to every model and based on the latent representation
of that image, we generate a sample. We see that the aggregated VAE (Figure
15(b)) is not able to conditionally generate visually pleasing samples
compared to the independent VAEs (Figure 15(a)) and the MMVM VAE (Figure
15(c)).

(a) Independent VAEs

(b) PoE VAE

(c) MMVM VAE

Figure 16:  Qualitative results for the MIMIC-CXR dataset on the conditional
generation task lateral to frontal. Results are aligned with the other
datasets.  Figure 17:  MIMIC-CXR experiment dataset: every column is a bimodal
tuple $\bm{X}$, the top row shows samples of the frontal modality
$\bm{x}_{f}$, and the bottom row shows samples of the lateral modality
$\bm{x}_{l}$. The first two tuples are linked to No Findings, indicative of
healthy conditions. Tuples three and four are labeled with Consolidation
disease. The tuple five is labeled with Atelectasis disease. We can observe
that tuples three and four share the same frontal image, but they differ due
to having distinct lateral images.

###  B.6 CUB Image-Captions

####  B.6.1 Dataset

The dataset contains 11,788 images of birds and 117,880 (10 times as many)
captions, each image with 10 fine-grained captions describing the bird’s
appearance characteristics collected through Amazon Mechanical Turk (AMT). We
performed a 75-25 training-test split, with 8,855 and 2,933 images, and 88,550
and 29,330 corresponding captions in the training set and test set,
respectively. We resized the images to 3 channels and 64 by 64 pixels and
modeled the captions using embeddings.

The labels applied to downstream tasks are based on the bird’s primary colors.
Based on the primary color attribute, we remapped the original 15-class
primary color label set into a 6-class label set to address the sparsity
issue. The labels and mappings are blue-to-red (blue, iridescent, purple,
green, pink, red), brown (brown), grey (grey), yellow (yellow, olive, orange,
buff), black (black), and white (white).

####  B.6.2 Implementation Details & Training

The network structures are similar to the ones in other experiments. For both
image and caption data, we employed a CNN encoder and decoder. We used a
128-dimensional latent space with a Laplace likelihood on images and a
Categorical likelihood for captions. The activation function was chosen to be
ReLU between layers. For all experiments on this dataset, we used an Adam
optimizer with an initial learning rate of 0.001 and a batch size of 128. We
trained all models for 100 epochs.

###  B.7 MIMIC-CXR

####  B.7.1 Dataset

The dataset we use in our experiment is a multimodal interpretation of the
original MIMIC-CXR dataset. The original MIMIC-CXR dataset comprises high-
resolution chest X-ray images related to imaging studies. A study may include
multiple chest X-ray images captured from several view positions. We
categorized these views into two primary modalities: frontal (including “AP”
and “PA” views) and lateral (including “LL” and “Lateral” views). For each
study, we pair every frontal image with every lateral image in all possible
combinations. Studies lacking at least one frontal and one lateral image are
excluded. This approach formalizes a new dataset composed of image pairs, thus
offering a bimodal interpretation of the original MIMIC-CXR dataset. More
rigorously, we define a dataset $\mathbb{X}=\\{\bm{X}^{(i)}\\}_{i=1}^{n}$
where each $\bm{X}^{(i)}=\\{\bm{x}_{f}^{(i)},\bm{x}_{l}^{(i)}\\}$ is a bimodal
tuple composed of one frontal image and one lateral image of the same study.
An image may appear in multiple tuples, but we ensure that each tuple is
unique by having at least one different image. Examples of these bimodal
tuples are illustrated in Figure 17.

We use the JPG version of the MIMIC-CXR dataset, namely the MIMIC-CXR-JPG
dataset [Johnson et al., 2019a]. In our preprocessing pipeline, we apply
center cropping and downscale the images to a resolution of $224\times 224$.
We utilize the labels from the MIMIC-CXR-JPG dataset which are obtained using
the CheXpert tool [Irvin et al., 2019]. All non-positive labels (including
“negative,” “non-mentioned,” or “uncertain”) were combined into an aggregate
“negative” label following the approach adopted by Haque et al. [2023]. Each
imaging study is connected to a subject. We split the dataset into distinct
training (80%), validation (10%), and test (10%) sets based on subjects, thus
ensuring that the same image or study cannot be present in multiple sets.

####  B.7.2 Implementation & Training

##### Multimodal VAEs Implementation & Training

We use ResNet-based encoders and decoders for this experiment [He et al.,
2016], similar to those used in the PolyMNIST and the Bimodal CelebA
experiments. The image encoder and decoder consist of 2D convolution layers.
The architectural design of the image encoders and decoders is uniform for
both frontal and lateral modalities. We use an Adam optimizer [Kingma and Ba,
2014] with a learning rate of $0.00005$ and a batch size of $32$. We train all
methods for $240$ epochs and $3$ seeds. The implementation follows the one
described in Section B.4.

We use NVIDIA A100-SXM4-40GB GPUs for all our runs. An average run, evaluating
one method on one seed, takes approximately $45$ hours. To train all methods
evaluated in this experiment, we had to train $3\times 3=9$ different models:
$3$ seeds and $3$ methods. Hence, the total GPU compute time used to generate
the VAE results for the MIMIC-CXR experiment is around $45\times 9=405$ hours.
We also had to use GPU time in the development process, which we did not
measure.

##### Supervised classifier Implementation & Training

We use supervised classifiers based on Resnet. The classifier architecture is
derived from the VAEs encoder we use in this experiment. To adapt the encoder
for classification tasks, we added a linear layer equipped with $14$ neurons
corresponding to the number of labels in the MIMIC-CXR dataset. We use an Adam
optimizer with a learning rate of $0.0001$ and a batch size of $256$. We train
distinct models for both frontal and lateral modalities, training each for
$30$ epochs and $3$ seeds.

We train the classifiers using NVIDIA A100-SXM4-40GB GPUs. To train one
classifier on one seed takes approximately $1$ hour. We train a total of
$3\times 2=6$ classifiers: $3$ seeds and $2$ modalities, resulting in about
$6$ hours of GPU compute time. We also had to use GPU time in the development
process, which we did not measure.

####  B.7.3 Additional Results

In our main experiment, we evaluate the quality of the learned latent
representations using Random Forest (RF) classifiers. Specifically, we train
independent RF binary classifiers for each model and each label on the
inferred representations of the training set and evaluate them on the
representations of the test set. The RF classifiers are configured with 5,000
estimators and a maximum depth of 30. In Table 1 in Section 5.3, we report for
each model the average performance over the two modalities and three seeds,
totaling six RF classifiers per model: one for each modality and each of the
three different seeds.

Here, we provide detailed insights into the capabilities of the models to
leverage different modalities during training to improve the unimodal
representations by reporting the performance of the latent representation
classification for each modality separately in Table 2 and Table 3. These
results allow us to highlight the strengths of the MMVM model in improving
unimodal representations with additional modalities. For instance, in the main
text we mentioned that in the Cardiomegaly classification task, the MMVM
lateral representations $z_{L}$ lead to substantial improvements over the
frontal representations $z_{F}$ of the other VAEs, even though the former
modality $x_{L}$ is less informative than the latter $x_{F}$ as indicated by
the respective performance of the supervised model on each modality. This
behavior is not specific to Cardiomegaly and can be observed for other labels
such as Atelectasis, Lung Opacity, No Finding, Pleural Effusion, Pneumonia,
and Support Devices. Even though the difference between modalities for these
labels is sometimes small, the consistency of those results demonstrate the
soft-sharing capabilities of the MMVM approach. Conversely, in tasks where the
frontal view is less informative, such as Consolidation, Enlarged
Cardiomediastinum, and Pneumothorax, the MMVM’s lateral representations
outperform the frontal ones of the other VAEs.

The efficacy of the soft sharing mechanism in the MMVM VAE is also reflected
in the average performance for all labels. The performance difference between
the two unimodal latent representations produced by the MMVM VAE is
substantially smaller (0.9 percentage points on average) compared to the
independent VAEs (2.5 percentage points on average) and the PoE VAE (2.2
percentage points on average).

Table 2: Evaluation of the VAEs’ frontal latent representations $z_{F}$ classification performance on the test split. The performance of a fully-supervised non-linear deep network is included for reference. The average AUROC [%] and standard deviation over three seeds are reported. | supervised | independent | PoE | MMVM
---|---|---|---|---
Atelectasis | 79.5 $\pm$0.3 | 73.1 $\pm$0.0 | 75.7 $\pm$0.3 | 77.6 $\pm$0.1
Cardiomegaly | 81.7 $\pm$0.1 | 76.3 $\pm$0.4 | 78.5 $\pm$0.3 | 80.5 $\pm$0.1
Consolidation | 65.3 $\pm$0.7 | 62.4 $\pm$0.4 | 66.7 $\pm$0.8 | 69.1 $\pm$0.6
Edema | 88.0 $\pm$0.2 | 83.0 $\pm$0.3 | 84.5 $\pm$0.3 | 86.3 $\pm$0.1
Enlarged Cardiomediastinum | 57.9 $\pm$1.3 | 59.5 $\pm$1.2 | 66.4 $\pm$0.5 | 68.6 $\pm$1.1
Fracture | 51.4 $\pm$0.5 | 56.0 $\pm$0.1 | 58.8 $\pm$0.3 | 58.6 $\pm$0.8
Lung Lesion | 52.4 $\pm$0.4 | 61.3 $\pm$0.4 | 63.4 $\pm$0.4 | 64.1 $\pm$0.2
Lung Opacity | 69.5 $\pm$0.2 | 63.8 $\pm$0.3 | 66.3 $\pm$0.2 | 68.1 $\pm$0.1
No Finding | 73.9 $\pm$1.3 | 76.6 $\pm$0.3 | 77.2 $\pm$0.2 | 79.1 $\pm$0.1
Pleural Effusion | 88.0 $\pm$0.0 | 81.2 $\pm$0.6 | 82.8 $\pm$0.2 | 85.7 $\pm$0.3
Pleural Other | 53.9 $\pm$1.0 | 67.8 $\pm$1.1 | 68.7 $\pm$1.2 | 70.0 $\pm$2.0
Pneumonia | 61.3 $\pm$0.3 | 55.3 $\pm$0.5 | 57.5 $\pm$0.4 | 60.0 $\pm$0.6
Pneumothorax | 76.9 $\pm$1.1 | 75.3 $\pm$1.0 | 78.5 $\pm$0.7 | 81.9 $\pm$0.4
Support Devices | 77.7 $\pm$0.1 | 70.8 $\pm$0.3 | 73.8 $\pm$0.1 | 76.6 $\pm$0.2
All Labels | 69.8 $\pm$12.4 | 68.7 $\pm$9.0 | 71.3 $\pm$8.4 | 73.3 $\pm$8.9
Table 3: Evaluation of the VAEs’ lateral latent representation $z_{L}$ classification performance on the test split. The performance of a fully-supervised non-linear deep network is included for reference. The average AUROC [%] and standard deviation over three seeds are reported. | supervised | independent | PoE | MMVM
---|---|---|---|---
Atelectasis | 78.0 $\pm$0.1 | 70.7 $\pm$0.3 | 73.7 $\pm$0.2 | 77.0 $\pm$0.2
Cardiomegaly | 79.0 $\pm$0.2 | 70.8 $\pm$0.9 | 74.8 $\pm$0.1 | 78.7 $\pm$0.0
Consolidation | 68.6 $\pm$1.4 | 64.4 $\pm$1.4 | 66.7 $\pm$0.9 | 70.2 $\pm$0.8
Edema | 86.2 $\pm$0.1 | 75.4 $\pm$0.9 | 79.1 $\pm$0.1 | 84.3 $\pm$0.3
Enlarged Cardiomediastinum | 61.5 $\pm$1.0 | 60.1 $\pm$0.7 | 63.5 $\pm$0.8 | 69.0 $\pm$0.7
Fracture | 52.3 $\pm$0.2 | 57.9 $\pm$0.6 | 59.0 $\pm$0.5 | 60.9 $\pm$0.3
Lung Lesion | 57.3 $\pm$0.4 | 58.9 $\pm$0.2 | 59.3 $\pm$0.3 | 63.0 $\pm$0.7
Lung Opacity | 68.9 $\pm$0.2 | 61.9 $\pm$0.5 | 64.1 $\pm$0.4 | 68.1 $\pm$0.2
No Finding | 72.0 $\pm$0.8 | 73.9 $\pm$0.3 | 74.6 $\pm$0.1 | 78.3 $\pm$0.1
Pleural Effusion | 91.0 $\pm$0.3 | 80.2 $\pm$0.2 | 82.1 $\pm$0.1 | 85.7 $\pm$0.1
Pleural Other | 61.5 $\pm$1.9 | 62.8 $\pm$1.3 | 63.9 $\pm$1.0 | 68.5 $\pm$1.9
Pneumonia | 61.2 $\pm$0.4 | 56.4 $\pm$0.5 | 58.2 $\pm$0.1 | 59.0 $\pm$0.2
Pneumothorax | 82.4 $\pm$0.8 | 75.6 $\pm$0.5 | 78.6 $\pm$0.2 | 81.7 $\pm$0.3
Support Devices | 77.3 $\pm$0.1 | 71.9 $\pm$0.6 | 74.7 $\pm$0.5 | 77.1 $\pm$0.3
All Labels | 71.2 $\pm$11.2 | 67.2 $\pm$7.6 | 69.4 $\pm$8.0 | 73.0 $\pm$8.5

Figure 18 illustrates the performance of the average unimodal latent
representation $z$ classification described above against the reconstruction
loss for the three different VAE methods introduced in the main text
(independent, aggregate PoE VAE and the proposed MMVM VAE). As a reminder, the
AUROC performance [in %] is averaged over three seeds and the two modalities.

Figure 18: Latent representation classification for the MIMIC-CXR dataset. The
mean AUROC over all labels and averaged over three seeds is reported.

###  B.8 Hippocampal Neural Activities

####  B.8.1 Dataset

The training data was collected from 250 ms length time frames after the port
entry. Due to the behavior difference from each rat (some rats react faster to
the odors while some others react slower), the training time frames of the
five rats started from 250 ms, 250 ms, 500 ms, 500 ms, and 250 ms,
respectively. During training, we treated each data point as independent and
trained all the VAE models based on sliding windows (100 ms sub-window, 10 ms
steps; 16 data points per window on each trial). The 100 ms sub-windows
constituted the input data, with the dimension equal to the rat’s number of
neurons multiplied by 10, as the data was further binned into 10-ms
increments.

####  B.8.2 Implementation & Training

We use the same network architectures for all multimodal VAEs. Each of the
autoencoders includes its unique encoder and decoder, both containing two
hidden layers, without weight-sharing during training and evaluation. All
modalities share the same architecture but the layers’ dimensions are
different, with 920, 790, 1040, 490, 460 dimensional input and hidden layers,
respectively. The activation function was chosen to be LeakyReLU with a 0.01
negative slope. For all experiments on this dataset, we use an Adam optimizer
with an initial learning rate of $0.001$, a batch size of $128$. We train all
models for $1000$ epochs.

####  B.8.3 Additional Results

We show the 2-dimensional latent representations for every rat and the six VAE
encoders in Figure 4 and Figure 20. In these two figures, each dot is the two-
dimensional latent representation of a 100 ms sub-window of one odor trial for
one rat. Figure 4 is colored by 4 odors, and Figure 20 is colored by 5
modalities (rats). Figure 4 shows the odor stimuli separation on the latent
space and how good MMVM VAE is in separating the odors. Figure 20 shows that
the proposed MMVM VAE can best utilize the shared information between the five
rats by pulling the latent representations together. At the same time, the
independent AVG and PoE baseline models fail to extract the information shared
between rats. Although it shows separation in some views, the independent
model does not provide a connection between views. The five tiny clusters in
Figure 4 and Figure 20 show that, instead of showing a clear odor separation
on the latent space, the AVG model separated the data by rats. The results
went against the intention to share information across views. In other words,
the five rats’ latent representations were far away from each other, so the
aggregated VAE completely failed to connect the five views.

(a) independent

(b) AVG

(c) MoE

(d) PoE

(e) MoPoE

(f) MMVM

Figure 19: Latent neural representation during a memory experiment. Each
model’s performance is evaluated based on its own optimal $\beta$ value
(0.00001, 0.01, 0.001 for independent, aggregated, and MMVM respectively) in
terms of the self-conditioned latent representation classification accuracy
according to Figure 3(a). Our model can distinguish the odor stimuli in the
latent space with a clear separation of odors (4 different colors).
Conversely, the two baseline models failed to combine multi-views as the odor
separation only occurred within single views.

(a) independent

(b) AVG

(c) MoE

(d) PoE

(e) MoPoE

(f) MMVM

Figure 20: Latent Representation of Rats Brain by Each Rat. In our proposed
MMVM model, the five views shared latent representations as the latent
representation of all five views (colors) gathered together, while the two
baseline models failed to combine multi-views. As shown in the baseline
models’ latent representation, the five clusters (colors) kept away from each
other and did not share information.