(accv) Package accv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: IMT School for Advanced Studies of Lucca, 22institutetext:
Institute of Computational Perception, Johannes Kepler University
33institutetext: Mohamed bin Zayed University of Artificial Intelligence
44institutetext: Swarm Robotics Lab NCRA, University of Engineering and
Technology, Taxila 55institutetext: Dalhousie University 66institutetext:
Interuniversity Microelectronics Centre (IMEC), 77institutetext: Human-
centered AI Group, AI Lab, Linz Institute of Technology
77email<EMAIL_ADDRESS>77email<EMAIL_ADDRESS>
# Chameleon: Images Are What You Need For Multimodal Learning Robust To
Missing Modalities

Muhammad Irzam Liaqat 11 0009-0009-1265-7337 Shah Nawaz 22 0000-0002-7715-4409
Muhammad Zaigham Zaheer 33 0000-0001-8272-1351 Muhammad Saad Saeed 44
0000-0002-0893-9499 Hassan Sajjad 55 0000-0002-8584-6595 Tom De Schepper 66
0000-0002-2969-3133 Karthik Nandakumar 33 0000-0002-6274-9725 Muhammad Haris
Khan 33 0000-0001-9746-276X Markus Schedl 2277 0000-0003-1706-3406

###### Abstract

Multimodal learning has demonstrated remarkable performance improvements over
unimodal architectures. However, multimodal learning methods often exhibit
deteriorated performances if one or more modalities are missing. This may be
attributed to the commonly used multi-branch design containing modality-
specific streams making the models reliant on the availability of a complete
set of modalities. In this work, we propose a robust textual-visual multimodal
learning method, Chameleon, that completely deviates from the conventional
multi-branch design. To enable this, we present the unification of input
modalities into one format by encoding textual modality into visual
representations. As a result, our approach does not require modality-specific
branches to learn modality-independent multimodal representations making it
robust to missing modalities. Extensive experiments are performed on four
popular challenging datasets including Hateful Memes, UPMC Food-101, MM-IMDb,
and Ferramenta. Chameleon not only achieves superior performance when all
modalities are present at train/test time but also demonstrates notable
resilience in the case of missing modalities.

###### Keywords:

Multimodal learning Vision and other modalities Missing modalities

##  1 Introduction

Recent years have seen a surge in the use of multimodal data for various
applications. For instance, users combine text, image, audio, or video
modalities to sell a product over an e-commerce platform or express views on
social media platforms. Two or more of these modalities are often combined in
a multi-branch network to solve different tasks such as multimodal
classification [20, 19], cross-modal retrieval [46], cross-modal verification
[33], multimodal named entity recognition [30, 3], visual question answering
[1, 8], image captioning [45], semantic relatedness [17], and multimodal
machine translation [41, 7]. In all these tasks, multimodal methods can
experience scenarios where some modalities are missing, e.g., due to failures
in data acquisition pipelines. Several researchers have recently concluded
that multimodal learning is not inherently robust to missing modalities and
can result in a significantly deteriorated performance when modalities are
missing [27, 28, 23]. For example, as seen in Fig. 1, ViLT [22], a state-of-
the-art (SOTA)textual-visual Transformer, demonstrates a drop in performance
of $28.3$% when only $30$% of the textual modality is available (i.e., $70$%
missing) at test time. The performance deterioration may be attributed to the
fundamental design of multi-branch methods [22, 26, 46, 32] for multimodal
learning that strives to project each modality to a joint embedding space. As
such, this makes the model inherently reliant on the availability of a
complete set of modalities [27].

Figure 1: Comparisons of Chameleon with ViLT using Food-$101$. Chameleon
demonstrates better multimodal performance as well as retains superior
performance when textual modality is $70$% (30% available) missing at test
time.

To address the shortcomings of existing multi-branch networks, we propose
Chameleon, a multimodal learning method operating on textual-visual modalities
that encodes text embeddings as an image. This way, the network is able to
process either of the two inputs, visual or encoded text, as well as both
together, all as images. Since this encoding is obtained at the input level,
the weights are shared across modalities, thereby making the model independent
of modality-specific branches. To perform training, we propose two variants of
Chameleon, illustrated in Fig. 2c and Fig. 2d. While more details about these
variants are discussed in Section 3, when a modality is missing during
training or testing Chameleon shifts its attention to the available modality
to output the correct prediction scores. We evaluate Chameleon on four
textual-visual datasets, including UPMC Food-$101$ [48], Hateful Memes [19],
MM-IMDb [2], and Ferramenta [10]. It outperforms SOTA multimodal methods on
complete modalities and demonstrates superior robustness against missing
modalities at train and test times. For instance, as seen in Fig. 1, compared
to ViLT [22], Chameleon consistently demonstrates better classification
accuracy in both missing modality and complete modality scenarios. Notably,
Chameleon outperforms ViLT in the missing modality setting by a significant
margin of $13.3$%.

The key contributions of our work are as follows:

  1. 1.

We propose Chameleon, a textual-visual multimodal method inherently robust to
missing modalities.

  2. 2.

We explore an encoding approach that transforms textual information into
visual representations to carry out robust multimodal training.

  3. 3.

We explore the general applicability of our approach towards robustness to
missing modalities on Convolutional Neural Networks (CNNs) as well as Vision
Transformers (ViT). To the best of our knowledge, Chameleon is the first work
that explores the robustness to missing modality across multiple learning
paradigms including CNNs and ViTs.

  4. 4.

A wide range of experiments performed on various textual-visual datasets under
different unimodal, multimodal, and missing modality settings during training
and testing demonstrate the significance of Chameleon, making it a defacto
approach for missing modality related applications.

##  2 Related Work

###  2.1 Multimodal Learning and Missing Modality

Multiple modalities including texts and images often contain complementary
information about a common subject. The goal of multimodal learning is to
leverage complementary information across modalities to improve the
performance of various machine learning tasks. Each task may be different from
the other, however, the underlying objective remains the same: to learn joint
representations across multiple modalities [4, 50]. Existing multimodal
methods employ multi-branch networks to learn joint representations by
minimizing the distance between the representations of multiple modalities
[32, 38, 21, 2, 44, 20, 22, 26, 36, 33]. Such methods have achieved remarkable
performance using modality-complete data [15, 2, 44, 20, 51, 19, 22].

However, multimodal methods suffer from performance deterioration if some
modalities are absent either during training or testing [27, 23, 56, 42].
Considering the importance of multimodal methods, recent years have seen a
surge in tackling the missing modality problem [28, 27, 23, 54, 47]. For
example, Ma et al. [27] improved the robustness of Transformer models via
multi-task optimization. More recently, Lee et al. [23] introduced missing-
modality-aware prompts which can be plugged into multimodal Transformers to
handle missing modality. Chameleon is related to these as we also address the
missing modality problem. However, we approach the problem by converting
textual modality into a visual format to train a visual network. This makes
our approach independent of any specific component such as missing-modality-
aware prompts towards tackling the missing modality problem.

###  2.2 Rendering Non-visual Modalities into Images

Recently, some researchers have explored the idea of transforming text
information into a visual format to perform visual recognition task. For
example, Gallo et al. [12] leverage Word$2$Vec word embeddings to reconstruct
the semantics associated with text as an image to train a visual network for
text classification. Salesky et al. [39] render the raw text directly into
visual format, divide the rendered image into overlapping slices, and produce
representations with optical character recognition to train a machine
translation model. Similarly, Rust et al. [37] propose a Masked Autoencoding
Visual Transformer to reconstruct the pixels in masked image patches to train
a language model. More recently, Tschannen et al. [43] proposed CLIPPO that
renders text information as images to train a pure pixel-based model
performing multimodal learning. All approaches rendering text into visual
format are, in essence, related to Chameleon as we also encode text into
visual format. However, instead of the common approach of transforming raw
text directly into images, we use word embeddings to encode textual
information as color-coded pixels which is found to be effective in our
experiments for the missing modality problem. Further, we explore the
utilization of joint visual representations of multimodal inputs to train a
robust multimodal classification method as well as study its resilience
against missing modalities. In addition, although the scope of our approach is
textual-visual data, we acknowledge the existing approaches that encode audio
modality into visual format to train on a given task [31, 49].

###  2.3 Mixing Multiple Input Instances

The second variant of our training strategy, fused, is loosely inspired by the
recent progress in image augmentation [55, 53] where two input images are
mixed in such a way that the network is trained to predict both class labels
at the same time. Such approaches have been proven effective towards better
generalization, i.e., when only one image is provided to the network at test
time, the model is able to classify it correctly. We extend this idea to
improve the multimodal performance as well as robustness of Chameleon towards
missing modality by augmenting the encoded text with its corresponding image
to create a fused multimodal image. The network is then trained using this
fused image to predict the class labels.

Figure 2: The overall architecture of Chameleon. (a) Multimodal input consists
of images and text. (b) Word embeddings are encoded as an image. Example of
encoding three words: ‘Extol’, ‘craft’, and ‘scissors’ is provided with
embedding length $15$. (c) The joint variant of Chameleon in which visual and
encoded text inputs are fed to a weight-sharing visual network in a sequential
fashion. (d) The fused variant of Chameleon in which a fused image created by
fusing both visual and encoded text inputs are fed to the network.

##  3 Methodology

We present Chameleon, a multimodal learning method consisting of textual-
visual modalities that encodes text embeddings as images and subsequently
trains an image classification network to learn joint representation. Our
approach is built on the intuition that learning on a common input format with
weight sharing across multiple modalities results in a robust multimodal
method resilient to missing modalities. In order to carry out the training, we
explore two different input approaches namely joint and fused. Fig. 2
summarizes Chameleon whereas each of the components is discussed subsequently
in this section.

###  3.1 Problem Formulation

Formally, given $\mathcal{D}=\\{(x_{i}^{t},x_{i}^{v})\\}_{i=1}^{N}$ is the
training set where $N$ is the number of pairs of modality $t$ (text) and
modality $v$ (visual) and $x_{i}^{t}$ and $x_{i}^{v}$ are individual modality
samples of the $i^{th}$ instance respectively. Moreover, each pair
$(x_{i}^{t},x_{i}^{v})$ has a class label $y_{i}$. Typical existing multimodal
methods take multiple modalities as input by using a multi-branch network
$\mathcal{C}_{m}$ to perform the classification task:

$\tilde{y}_{i}=\mathcal{C}_{m}(\\{x^{t},x^{v}\\},y_{i})$ (1)

Such a multi-branch configuration requires modality-complete data to perform a
given task and, as seen in Fig. 1, missing modality may result in a
significant performance deterioration [22, 27].

In Chameleon, we propose to encode text input as image and subsequently
perform the task entirely in the visual domain, both during training and
testing. In other words, a visual network is trained on multimodal data
consisting of images and encoded texts through a single interface of visual
information. We hypothesize that unifying the modalities to an identical input
format will make the model independent of modality-specific branches and will
result in a method robust to missing modalities. Therefore, in our approach,
Eq. (1) takes the following form:

$\tilde{y}_{i}=\mathcal{C}_{v}(\\{\hat{x}^{t},x^{v}\\},y_{i}),$ (2)

where $\mathcal{C}_{v}$ is a visual classifier,
$\hat{x}^{t}=\mathcal{E}(x^{t})$, and $\mathcal{E}$ is the encoding scheme
that transforms $x^{t}$ into its visual format $\hat{x}^{t}$. The encoding
scheme is discussed next.

###  3.2 Text Encoding Scheme

Mikolov et al. [29] showed that words group together in high dimensional space
based on various linguistic knowledge such as semantics, morphology, and
syntax. Moreover, Dalvi et al. [5] showed the existence of multifaceted groups
in the latent space, i.e., groups that are formed based on more than one
relationship among words. We hypothesize that the knowledge of these
relationships is represented in the feature vectors. Explicitly considering
all features in the form of an image will enable the preservation of the rich
knowledge learned in the network. Building upon this, we use combinations of
three consecutive values in the feature vector as RGB pixel blocks. Formally,
for a given pair of $(x_{i}^{t},x_{i}^{v})$, the encoding scheme takes a word
in $x_{i}^{t}$ and converts it into a feature vector. The feature vector is
then normalized to assume values in the interval $[0\dots 255]$. Afterward,
consecutive three elements in the normalized feature vector are considered as
one RGB value. The process is repeated for each word in $x_{i}^{t}$ to obtain
the final visual encoding $\hat{x}_{i}^{t}$. Fig. 2b shows examples of
encoding input words into an RGB sequence. To train the network, we explore
two possible configurations:
Joint. The encoded text information ($\hat{x}_{i}^{t}$) is patched onto a
blank image to create a joint visual representation (Fig. 2c). This results in
an additional image containing only the encoded text as visual information.
For training, the two images are input in a sequential fashion to a weight-
shared visual network.
Fused. The encoded text ($\hat{x}_{i}^{t}$) is patched onto its corresponding
input image ($x_{i}^{v}$) to create a fused visual representation (Fig. 2d)
containing both modalities. The training of the visual classifier
($\mathcal{C}_{v}$) is carried out in a typical fashion.

|  |  Baseline | Chameleon
---|---|---|---
|  a. Orig. Image |  b. Unimodal |  c. Multimodal |  d. Missing Text |  e. Missing Image
Ice Cream |  [-.5] |  [-.5] |  [-.5] |  [-.5] |  [-.5]
|  0.60 |  0.99 |  0.99 |  1.0
Caser Salad |  [-.5] |  [-.5] |  [-.5] |  [-.5] |  [-.5]
|  0.98 |  0.99 |  0.99 |  1.0
Baklava |  [-.5] |  [-.5] |  [-.5] |  [-.5] |  [-.5]
|  0.98 |  0.99 |  0.99 |  0.99

Figure 3:  Grad-CAM visualizations of selected images from UPMC Food-$101$:
(a) Orignal image randomly selected from the test set. (b) Unimodal image only
training and testing. (c) Multimodal training and testing. (d) Multimodal
training; testing on image only, i.e., 100% text missing. (e) Multimodal
training; testing on text only, i.e., 100% image missing. As seen, in unimodal
image-only training (b), the model focuses on distinct features of the object.
With our multimodal training (c), the model not only retains its focus on the
object but also includes encodings representing text modality to make the
predictions. When the text modality (d) or image modality (e) is missing
during testing, the model focuses on the available modality to make accurate
final predictions demonstrating the success of Chameleon in training the
multimodal method robust to missing modalities.

###  3.3 Robustness to Missing Modalities

Chameleon inherently learns shared representations across modalities, bringing
them into a similar latent space. This results in a richer representation that
is complemented by both modalities and yields a robust model when one of the
modalities is missing. In other words, when a modality is missing, the
available modality enables the use of the multimodal knowledge from the shared
latent space to make a prediction. To verify this, we visually compare the
activation maps [40] of various different training/testing configurations of
Chameleon in Fig. 3.2, including image-only unimodal training/testing,
multimodal training/testing, multimodal training with missing text modality
testing, and multimodal training with missing image modality testing. In
image-only unimodal training and testing (Fig. 3.2b), as expected, the model
focuses on the distinct features of the object to predict the classification
score. For the case of complete modality during training and testing (Fig.
3.2c), Chameleon not only retains its focus on parts of the object but also
attends to the portions of the image where text modality is encoded.

In the case of missing modality at test time, a multimodal architecture robust
to missing modalities should ideally be able to retain the focus on the
available modality and behave like an unimodal network. We observe this case
in Fig. 3.2d for Chameleon, when the text modality is missing at test time. As
seen, the model shifts its focus on the object itself and behaves similarly to
the unimodal network. Likewise, when the image modality is missing at test
time (Fig. 3.2d), Chameleon shifts its focus to the encoded textual
representation. These visualizations highlight the internal working of
Chameleon in successfully learning multimodal representations while
demonstrating resilience against missing modalities.

##  4 Experiments and Analysis

We evaluate Chameleon on the multimodal classification task using four popular
and challenging datasets including UPMC Food-$101$ [48], Hateful Memes [18],
MM-IMDb [44], and Ferramenta [10]. Recently, Ma et al.[27] and Lee et al. [23]
introduced comprehensive protocols to study the missing modality problem on
textual-visual data at train and test time. Specifically, multimodal methods
are trained/tested on different predefined levels of missing modality to
observe deterioration. We conduct experiments using these settings to make
results comparable. Moreover, an extensive ablation study is performed to
evaluate different design choices of Chameleon.

###  4.1 Datasets

In addition to the three textual-visual datasets used in the existing
literature on missing modality [27, 23], we extend our study by selecting
another widely popular and challenging multimodal dataset, Ferramenta [10],
which is curated to resolve ambiguities among visual samples by using the
textual modality.

UPMC Food-101. It is a multimodal dataset consisting of textual and visual
modalities. The dataset is crawled from the web and each entry consists of an
image and the HTML web page on which it was found. It contains $90,704$ pairs
divided into $101$ food classes, and comes with a predefined $75$/$25$
train/test split. Hateful Memes. It is a multimodal dataset containing meme
images and their respective textual contents as the two modalities with binary
labels and is developed with the aim of identifying hate speech in memes. The
dataset contains $10,000$ memes. MM-IMDb. It is a multimodal dataset
containing textual-visual modalities to predict the genres of a movie. It
contains $25,956$ image-text pair and $23$ classes. Ferramenta. It consists of
$88,010$ adverts belonging to $52$ different product classes extracted from an
e-commerce platform representing product images and their textual
specifications. The data is divided into $66,141$ instances for train and
$2,186$ instances for test.

###  4.2 Evaluation Metrics

Following existing SOTA methods on complete and missing modalities [23, 27,
44, 48, 19], we report classification accuracy for UPMC Food-$101$ and
Ferramenta, area under the receiver operating characteristic (AUROC) for
Hateful Memes and F$1$ Macro for MM-IMDb.

###  4.3 Implementation Details

We consider both CNN (ResNet-$101$ [14]) and ViT [6] ((vit-base-
patch$16$-$224$) as visual networks. Both variants including joint and fused
are also evaluated individually by carrying out separate training on each
network. Based on the detailed empirical studies reported in Sections 4.6,
unless stated otherwise, ViT with joint representation is our default choice
of architecture for training and testing. We train Word$2$Vec for every
dataset and the embedding length is set to $36$ for encoding words to visual
format. Rather than training from scratch, we fine-tune both ResNet-$101$ and
ViT pre-trained on ImageNet.

###  4.4 Evaluations Under Complete Modalities Setting

To evaluate whether Chameleon is capable of learning from multiple input
modalities, we first evaluate when complete modalities are present during
training and testing. As seen in Table 1, Chameleon achieves SOTA performance
on three of the four datasets including UPMC Food-$101$, Hateful memes, and
Ferramenta. More specifically, Chameleon achieves classification performances
of $92.5$%, $73.9$% and $96.6$%, respectively, outperforming other state-of-
the-art methods. Discussion about the performance of Chameleon on MM-IMDb
dataset is provided in the next section. Overall, the results demonstrate that
Chameleon is capable of learning from multiple modalities on par with the
existing SOTA multimodal methods.

Table 1: Comparison of Chameleon with SOTA multimodal methods on UPMC
Food-$101$, Hateful Memes, MM-IMDb, and Ferramenta datasets using modality-
complete data. Best results are bold; second best are underlined.

UPMC Food-$101$ [48]
---
Method | Acc.
Wang et al. [48] | 85.1
Fused Rep. [34] | 85.7
CLIPPO [43] | 91.2
CentralNet [44] | 91.5
ViLT [22] | 91.9
Ma et al. [27] | 92.0
MMBT [16] | 92.1
BL [13] | 92.5
Chameleon | 92.5

Hateful Memes  [19]
---
Method | AUROC
MMBT-G [16] | 67.3
ViLT [22] | 70.2
Clippo [43] | 70.7
Ma et al.[27] | 71.8
MMBT-R [16] | 72.2
Visual BERT [24] | 73.2
ViLBERT [26] | 73.4
Chameleon | 73.9

MM-IMDb [44]
---
Method | F1 Marco
CBGP [20] | 52.9
CBP [9] | 53.2
GMU [2] | 53.9
Lee et al. [23] | 54.0
ViLT [22] | 55.3
MFAS [35] | 55.7
CentralNet [44] | 56.1
Chameleon | 51.2

Ferramenta [10]
---
Method | Acc.
Ferramenta [10] | 92.9
Fused Rep. [34] | 94.8
IeTF [11] | 95.2
Two-Branch [38] | 96.2
MHFNet [52] | 96.5
Chameleon | 96.6

Table 2: Comparison of Chameleon with different levels of available modality at test time using UPMC-Food-$101$, Hateful Memes, MM-IMDb, and Ferramenta datasets. Comparison is provided with ViLT [22]∗, Ma et al. [27], CLIPPO[43], & LLaVA [25]. ∗ViLT values are taken from Ma et al. [27]. Boldface and underline denote, respectively, the best and second-best results. $\dagger$ indicates our implementation. Data | Training | Testing | ViLT∗ | Ma et al. | CLIPPO$\dagger$ | LLaVA$\dagger$ | Chameleon
---|---|---|---|---|---|---|---
Image | Text | Image | Text
Food-101 | 100% | 100% | 100% | 100% | 91.9 | 92.0 | 91.2 | 92.2 | 92.5
100% | 100% | 100% | 90% | 88.2 | 90.5 | 89.6 | 90.3 | 90.8
100% | 100% | 100% | 70% | 80.7 | 87.1 | 86.3 | 87.2 | 87.3
100% | 100% | 100% | 50% | 73.3 | 82.6 | 81.9 | 83.0 | 83.1
100% | 100% | 100% | 30% | 65.9 | 77.5 | 78.1 | 78.9 | 79.2
100% | 100% | 100% | 10% | 58.4 | 73.3 | 74.1 | 75.4 | 76.3
Hateful Memes | 100% | 100% | 100% | 100% | 70.2 | 71.8 | 70.7 | 70.7 | 73.9
100% | 100% | 100% | 90% | 68.8 | 69.7 | 70.3 | 70.5 | 73.6
100% | 100% | 100% | 70% | 65.9 | 66.6 | 70.0 | 70.3 | 73.4
100% | 100% | 100% | 50% | 63.6 | 63.9 | 69.3 | 69.9 | 73.1
100% | 100% | 100% | 30% | 60.2 | 61.2 | 69.0 | 69.8 | 72.8
100% | 100% | 100% | 10% | 58.0 | 59.6 | 68.1 | 69.5 | 72.0
MM-IMDb | 100% | 100% | 100% | 100% | 55.3 | 55.0 | 28.4 | 59.3 | 51.2
100% | 100% | 100% | 90% | 51.8 | 53.8 | 27.5 | 56.7 | 45.7
100% | 100% | 100% | 70% | 45.1 | 52.0 | 25.6 | 51.3 | 43.1
100% | 100% | 100% | 50% | 38.9 | 46.6 | 23.6 | 44.0 | 40.4
100% | 100% | 100% | 30% | 31.2 | 41.8 | 21.2 | 36.7 | 37.9
100% | 100% | 100% | 10% | 23.1 | 37.3 | 18.9 | 27.4 | 35.2
Ferramenta | 100% | 100% | 100% | 100% | 95.9 | - | 93.4 | 94.8 | 96.6
100% | 100% | 100% | 90% | 68.1 | - | 91.6 | 94.0 | 95.8
100% | 100% | 100% | 70% | 60.8 | - | 89.9 | 93.4 | 95.4
100% | 100% | 100% | 50% | 60.4 | - | 88.7 | 93.0 | 94.9
100% | 100% | 100% | 30% | 54.2 | - | 86.1 | 92.7 | 94.8
100% | 100% | 100% | 10% | 51.3 | - | 84.6 | 92.0 | 94.6

###  4.5 Evaluations Under Missing Modalities Setting

Several researchers have shown that multimodal methods are sensitive to
missing modalities [27, 28, 23]. Chameleon addresses this issue specifically
by converting textual information into visual format. In this section, we
provide a detailed comparison with existing SOTA methods by extensively
evaluating Chameleon on missing modality scenarios, both during training and
testing.

####  4.5.1 Missing Modalities During Testing.

A common issue with multimodal methods during deployment is encountering
missing modalities at test time due to failures in the data acquisition
pipeline. Table 2 compares Chameleon with other multimodal learning methods:
ViLT [22], Ma et al. [27], CLIPPO [43], and LLaVA [25] for varying amounts of
missing modality on UPMC Food-$101$, Hateful Memes, MM-IMDb, and Ferramenta
datasets during testing. As seen, Chameleon outperforms compared methods on
UPMC Food-$101$, Hateful Memes, and Ferramenta demonstrating strong resilience
against missing modalities. For example, in the case when only $10$% of text
modality is available on the UPMC Food-$101$ dataset during testing, Chameleon
demonstrates an accuracy of $76.3$%. In comparison, ViLT [22], Ma et al. [27],
CLIPPO [43] and LLaVA demonstrate performances of $58.4$%, $73.3$%, $74.1$%,
and $75.4$%, respectively. Similar trends are noticeable on Hateful Memes and
Ferramenta datasets. In the case of MM-IMDb dataset, though the performance of
Chameleon is lower when all modalities are available, it outperforms ViLT,
CLIPPO and LLaVa when a modality is severely missing. Overall, considering all
4 datasets presenting 24 scenarios of complete and missing modality, Chameleon
outperforms all compared methods in 18 while second best in 2 scenarios.
Another interesting observation is seen when results of Chameleon are compared
to CLIPPO which can be considered a baseline to our method that renders text
directly as an image. Furthermore, we compare results with a large-scale
vision and language model (LLaVA), which converts image features to textual
tokens. Though the methodology is different compared to Chameleon, the essence
of converting one modality into the other is similar. Comparing results
directly with CLIPPO and LLaVA demonstrates the effectiveness of Chameleon. As
seen in Table 2, Chameleon performs significantly better in most scenarios
which not only signifies the importance of our proposed text encoding scheme
for an improved generalization of networks robust to missing modalities but
also demonstrates the importance of converting textual modality into visual
format.

The overall persistent robustness to missing modality of Chameleon compared to
existing methods across various challenging datasets and scenarios is owed to
the text encoding scheme used that enables the training on visual inputs. This
way, if one modality is missing during testing, Chameleon shifts its focus to
the available modality, thus yielding higher classification performance. More
insights on this are provided in Fig. 3.2 where Grad-CAM visualizations
clearly show the shifting of focus from multimodal data to the available
modality for predictions. More visual examples are provided in the
supplementary material.

Table 3: Comparison of Chameleon with ViLT [22]∗, Lee et al. [23], and CLIPPO
[43] on UPMC Food-$101$ [48], Hateful Memes [19], and MM-IMDb [2] under
different missing modality settings during training and testing. ∗ViLT values
are taken from Lee et al. [23]. Boldface and underline denote, respectively,
the best and second best results. $\dagger$ indicates our implementation.

Data | Training | Testing | ViLT [22] | Lee et al. [23] | CLIPPO [43]$\dagger$ | Chameleon
---|---|---|---|---|---|---
Image | Text | Image | Text
UPMC Food-$101$ | 100% | 100% | 100% | 100% | 91.9 | 92.0 | 91.2 | 92.5
100% | 30% | 100% | 30% | 66.3 | 74.5 | 77.5 | 79.8
30% | 100% | 30% | 100% | 76.7 | 86.2 | 83.6 | 83.9
Hateful Memes | 100% | 100% | 100% | 100% | 70.2 | 71.0 | 70.7 | 73.9
100% | 30% | 100% | 30% | 60.8 | 59.1 | 60.9 | 72.7
30% | 100% | 30% | 100% | 61.6 | 63.1 | 58.4 | 62.0
MM-IMDb | 100% | 100% | 100% | 100% | 46.0 | 54.0 | 28.4 | 51.2
100% | 30% | 100% | 30% | 35.1 | 39.2 | 23.6 | 36.8
30% | 100% | 30% | 100% | 37.7 | 46.3 | 23.3 | 45.1

####  4.5.2 Missing Modalities During Training.

Derived from the motivation that modality can be missing in any data sample,
Lee et al. [23] recently extended the evaluation protocol by introducing
scenarios of missing modality during training and testing, i.e., $30$% of one
modality is available against $100$% of the other modality at train and test
time. Table 3 compares our approach with existing methods ViLT [22], Lee et
al. [23], and CLIPPO [43] using this protocol on UPMC Food-$101$, Hateful
Memes, and MM-IMDb datasets. In most scenarios, Chameleon demonstrates
comparable or better performance than existing SOTA methods. Notably, compared
to CLIPPO, Chameleon demonstrates robustness to missing modalities during
training and testing thus reiterating the importance of our text encoding
scheme to train multimodal systems.

(a) UPMC Food-$101$

(b) Hateful Memes

Figure 4: Performance comparisons of the two variants of Chameleon (fused and
joint) with ViLT [22] and Ma et al. [27] on various levels of missing textual
modality during testing on UPMC Food-101 and Hateful Memes datasets. A smaller
drop in performance by our approach in most cases signifies its effectiveness
towards training Vision Transformers resilient to missing modalities without
dataset-centric fusion strategies.

###  4.6 Analysis and Discussion

In this section, we provide further analysis on Chameleon including its
application to different visual networks, dataset independence, and
visualizations for further insights.

####  4.6.1 Is Chameleon Agnostic to the Visual Network?

The encoding scheme of Chameleon makes it generally usable to train any visual
network. Investigating the extent to which the visual classification network
and text encoding method influence performance, we conduct a series of
experiments, using ViT and CNN (ResNet-$101$) for the former and joint and
fused variants for the latter. The results of these experiments are summarized
in Table 4. As seen, although Transformer with joint encoding outperforms all
other variants of our approach, RestNet-$101$ demonstrates comparable
performance. An interesting observation here is that the fused approach works
slightly better in the case of CNNs whereas joint works slightly better in the
case of ViTs. Overall, the comparable performance of ViTs and CNNs
demonstrates that Chameleon is model-agnostic and may be applied to CNNs as
well as ViTs.

####  4.6.2 Are Dataset-specific Transformers Necessary?

Ma et al. [27] have observed that multimodal Transformers are not only
sensitive to missing modalities but different design choices, such as fusion
strategy, should be tailored to the dataset. They further conclude that it may
not be possible to design a general multimodal Transformer architecture to be
used across datasets. In contrast, with our proposed design of encoding
textual modality into visual format and then subsequently training a visual
network such as ViT, Chameleon becomes completely independent of dataset-
centric design choices. To evaluate this, we train two separate ViTs using
both fused and joint variants of Chameleon on the UPMC Food-$101$ and Hateful
Memes datasets. Each model is then evaluated on different levels of missing
textual modality at test time. Fig. 4 shows the corresponding results and
compares the performances with ViLT [22] and Ma et al. [27]. As Ma et al. have
proposed dataset-optimal fusion strategies, their approach demonstrates
robustness to missing modality compared to ViLT. Chameleon, on the other hand,
in both variants, demonstrates superior resilience against missing modality
while training on similar ViTs without any changes across datasets. Our study
suggests that Chameleon enables the training of dataset-agnostic transformers
capable of handling severe missing modalities.

Table 4: Performance analysis of fused and joint representation with different visual networks using UPMC-Food-$101$ and Hateful Memes datasets. AUROC and accuracy are reported for Hateful Memes and UPMC-Food-$101$, respectively. Dataset (Performance Metric) | Method |  ResNet-$101$ | ViT
---|---|---|---
UPMC Food-$101$ (Accuracy) | Fused | 91.2 | 90.5
Joint | 89.5 | 92.5
Hateful Memes (AUROC) | Fused | 71.2 | 73.6
Joint | 70.9 | 73.9
Figure 5: t-SNE visualizations of the embedding space of Chameleon (a - c) and
ViLT (d - f) along with accuracy on test set of UPMC Food-$101$. Compared to
ViLT, Chameleon not only enhances the classification boundaries when complete
modalities are available at test time but also retains these boundaries when
the textual or visual modality is completely missing during test time. Note
that classes are selected randomly from the test set.

####  4.6.3 t-SNE Visualizations.

We plot results of t-SNE projections to take a peek into the embedding space
of Chameleon trained on UPMC Food-$101$ dataset and provide comparisons with
ViLT [22] in Fig. 5. Comparisons are provided under the following three
settings: Fig. 5a & d) complete modalities during training and testing, b & e)
complete modalities during training but $100$% missing textual modality during
testing, and c & f) complete modalities during training but $100$% missing
visual modality during testing. In the case of multimodal training/testing,
compared to ViLT, Chameleon is able to project modalities belonging to the
same classes closer to each other while demonstrating inter-class separability
(Fig. 5a & d). This highlights the success of Chameleon in training a robust
multimodal classifier. Finally, in the case of multimodal training but 100%
missing modality during testing (Fig. 5b, c, e & f), although some distortions
are noticeable, compared to ViLT, Chameleon successfully retains the overall
separability of the classes under such severe missing modality case. This
demonstrates the resilience of Chameleon towards missing modalities.

###  4.7 Limitations

While advantageous in most missing modality cases, utilizing text as an image
has its limitations as well. For example, in the case of longer text
descriptions, the network struggles to learn joint representation of multiple
modalities as well as achieving robustness to missing modalities. This
phenomenon can be observed in Tables 2 & 3 where CLIPPO, utilizing rendered
text as an image, performs notably lower than its counterpart networks. In the
case of Chameleon, this problem is partially handled as we convert the textual
embedding into encoded visual representations before training a network. This
way, longer texts are summarized into compact encoded visual representations
thus yielding superior performance.

##  5 Conclusion

In this work, we presented a robust textual-visual multimodal learning method
that encodes word embeddings of the textual modality into visual format to
learn modality-independent representations of the input modalities. The common
input format facilitates joint representation learning by sharing weights
across multiple modalities resulting in a multimodal network robust to missing
modalities. Extensive experiments are performed on the popular and challenging
datasets UPMC Food-$101$, Hateful Memes, MM-IMDb and Ferramenta. The proposed
method is thoroughly evaluated on complete modalities as well as missing
modalities during training and testing. The experimental results indicate that
the proposed method obtains superior performance when a complete set of
modalities is available. In the case of missing modalities, the performance
deterioration is noticeably smaller than that of the existing multimodal
learning methods, indicating significant robustness of Chameleon.

## References

  * [1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)
  * [2] Arevalo, J., Solorio, T., Montes-y Gómez, M., González, F.A.: Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992 (2017)
  * [3] Arshad, O., Gallo, I., Nawaz, S., Calefati, A.: Aiding intra-text representations with visual context for multimodal named entity recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 337–342. IEEE (2019)
  * [4] Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41(2), 423–443 (2018)
  * [5] Dalvi, F., Khan, A.R., Alam, F., Durrani, N., Xu, J., Sajjad, H.: Discovering latent concepts learned in BERT. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=POTMtpYI1xH
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [7] Elliott, D., Frank, S., Sima’an, K., Specia, L.: Multi30k: Multilingual english-german image descriptions. arXiv preprint arXiv:1605.00459 (2016)
  * [8] Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847 (2016)
  * [9] Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. In: Su, J., Duh, K., Carreras, X. (eds.) Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 457–468. Association for Computational Linguistics, Austin, Texas (Nov 2016). https://doi.org/10.18653/v1/D16-1044, https://aclanthology.org/D16-1044
  * [10] Gallo, I., Calefati, A., Nawaz, S.: Multimodal classification fusion in real-world scenarios. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 5, pp. 36–41. IEEE (2017)
  * [11] Gallo, I., Calefati, A., Nawaz, S., Janjua, M.K.: Image and encoded text fusion for multi-modal classification. In: 2018 Digital Image Computing: Techniques and Applications (DICTA). pp. 1–7. IEEE (2018)
  * [12] Gallo, I., Nawaz, S., Calefati, A.: Semantic text encoding for text classification using convolutional neural networks. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 5, pp. 16–21. IEEE (2017)
  * [13] Gallo, I., Ria, G., Landro, N., La Grassa, R.: Image and text fusion for upmc food-101 using bert and cnns. In: 2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ). pp. 1–6. IEEE (2020)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] He, X., Peng, Y.: Fine-grained image classification via combining vision and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5994–6002 (2017)
  * [16] Kiela, D., Bhooshan, S., Firooz, H., Perez, E., Testuggine, D.: Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950 (2019)
  * [17] Kiela, D., Bottou, L.: Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In: Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP). pp. 36–45 (2014)
  * [18] Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Fitzpatrick, C.A., Bull, P., Lipstein, G., Nelli, T., Zhu, R., et al.: The hateful memes challenge: competition report. In: NeurIPS 2020 Competition and Demonstration Track. pp. 344–360. PMLR (2021)
  * [19] Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D.: The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems 33, 2611–2624 (2020)
  * [20] Kiela, D., Grave, E., Joulin, A., Mikolov, T.: Efficient large-scale multi-modal classification. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence pp. 5198–5204 (2018)
  * [21] Kim, C., Shin, H.V., Oh, T.H., Kaspar, A., Elgharib, M., Matusik, W.: On learning associations of faces and voices. In: Asian Conference on Computer Vision. pp. 276–292. Springer (2018)
  * [22] Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. In: International Conference on Machine Learning. pp. 5583–5594. PMLR (2021)
  * [23] Lee, Y.L., Tsai, Y.H., Chiu, W.C., Lee, C.Y.: Multimodal prompting with missing modalities for visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14943–14952 (2023)
  * [24] Li, L., Yatskar, M., Yin, D., Hsieh, C., Chang, K.: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)
  * [25] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)
  * [26] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32 (2019)
  * [27] Ma, M., Ren, J., Zhao, L., Testuggine, D., Peng, X.: Are multimodal transformers robust to missing modality? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18177–18186 (2022)
  * [28] Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: Smil: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2302–2310 (2021)
  * [29] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of the ICLR Workshop. Scottsdale, AZ, USA (2013)
  * [30] Moon, S., Neves, L., Carvalho, V.: Multimodal named entity recognition for short social media posts. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers. pp. 852–860. Association for Computational Linguistics (2018)
  * [31] Nagrani, A., Chung, J.S., Zisserman, A.: Voxceleb: a large-scale speaker identification dataset. In: INTERSPEECH (2017)
  * [32] Nagrani, A., Albanie, S., Zisserman, A.: Learnable pins: Cross-modal embeddings for person identity. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 71–88 (2018)
  * [33] Nagrani, A., Albanie, S., Zisserman, A.: Seeing voices and hearing faces: Cross-modal biometric matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8427–8436 (2018)
  * [34] Nawaz, S., Calefati, A., Janjua, M.K., Anwaar, M.U., Gallo, I.: Learning fused representations for large-scale multimodal classification. IEEE Sensors Letters 3(1), 1–4 (2018)
  * [35] Pérez-Rúa, J.M., Vielzeuf, V., Pateux, S., Baccouche, M., Jurie, F.: Mfas: Multimodal fusion architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6966–6975 (2019)
  * [36] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [37] Rust, P., Lotz, J.F., Bugliarello, E., Salesky, E., de Lhoneux, M., Elliott, D.: Language modelling with pixels. arXiv preprint arXiv:2207.06991 (2022)
  * [38] Saeed, M.S., Khan, M.H., Nawaz, S., Yousaf, M.H., Del Bue, A.: Fusion and orthogonal projection for improved face-voice association. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7057–7061. IEEE (2022)
  * [39] Salesky, E., Etter, D., Post, M.: Robust open-vocabulary translation from visual text representations. arXiv preprint arXiv:2104.08211 (2021)
  * [40] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [41] Specia, L., Frank, S., Sima’An, K., Elliott, D.: A shared task on multimodal machine translation and crosslingual image description. In: Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers. pp. 543–553 (2016)
  * [42] Suo, Q., Zhong, W., Ma, F., Yuan, Y., Gao, J., Zhang, A.: Metric learning on healthcare data with incomplete modalities. In: IJCAI. pp. 3534–3540 (2019)
  * [43] Tschannen, M., Mustafa, B., Houlsby, N.: Clippo: Image-and-language understanding from pixels only. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11006–11017 (2023)
  * [44] Vielzeuf, V., Lechervy, A., Pateux, S., Jurie, F.: Centralnet: a multilayer approach for multimodal fusion. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)
  * [45] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2015)
  * [46] Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5005–5013 (2016)
  * [47] Wang, N., Cao, H., Zhao, J., Chen, R., Yan, D., Zhang, J.: M2r2: Missing-modality robust emotion recognition framework with iterative data augmentation. IEEE Transactions on Artificial Intelligence (2022)
  * [48] Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). pp. 1–6. IEEE (2015)
  * [49] Xie, W., Nagrani, A., Chung, J.S., Zisserman, A.: Utterance-level aggregation for speaker recognition in the wild. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5791–5795. IEEE (2019)
  * [50] Xu, P., Zhu, X., Clifton, D.A.: Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(10), 12113–12132 (2023)
  * [51] Yang, F., Peng, X., Ghosh, G., Shilon, R., Ma, H., Moore, E., Predovic, G.: Exploring deep multimodal fusion of text and photo for hate speech classification. In: Proceedings of the third workshop on abusive language online. pp. 11–18 (2019)
  * [52] Yue, T., Li, Y., Qin, J., Hu, Z.: Multi-modal hierarchical fusion network for fine-grained paper classification. Multimedia Tools and Applications pp. 1–17 (2023)
  * [53] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)
  * [54] Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., Zhao, J.: M3care: Learning with missing modalities in multimodal healthcare data. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 2418–2428 (2022)
  * [55] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [56] Zhang, X., Song, Q., Liu, G.: Multimodal image aesthetic prediction with missing modality. Mathematics 10(13), 2312 (2022)