# Text-centric Alignment for Multi-Modality Learning

Yun-Da Tsai  National Taiwan University<EMAIL_ADDRESS>,  Ting Yu
Yen  National Taiwan University<EMAIL_ADDRESS>,  Pei-Fu Guo  National
Taiwan University<EMAIL_ADDRESS>,  Zhe-Yan Li  National Taiwan
University<EMAIL_ADDRESS>and  Shou-De Lin  National Taiwan University
<EMAIL_ADDRESS>
(2018)

###### Abstract.

This research paper addresses the challenge of modality mismatch in multimodal
learning, where the modalities available during inference differ from those
available at training. We propose the Text-centric Alignment for Multi-
Modality Learning (TAMML) approach, an innovative method that utilizes Large
Language Models (LLMs) with in-context learning and foundation models to
enhance the generalizability of multimodal systems under these conditions. By
leveraging the unique properties of text as a unified semantic space, TAMML
demonstrates significant improvements in handling unseen, diverse, and
unpredictable modality combinations. TAMML not only adapts to varying
modalities but also maintains robust performance, showcasing the potential of
foundation models in overcoming the limitations of traditional fixed-modality
frameworks in embedding representations. This study contributes to the field
by offering a flexible, effective solution for real-world applications where
modality availability is dynamic and uncertain.

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn:
978-1-4503-XXXX-X/18/06††copyright: none

##  1\. Introduction

This work considers a foundation model for multimodal learning. Multimodal
learning focuses on integrating and interpreting information across various
data modalities during a model’s training and inference stages. This
interdisciplinary field intersects with domains such as computer vision,
natural language processing, audio signal processing and programming (Xu et
al., 2024; Zhang et al., 2024; Tsai et al., 2023; Song et al., 2023). The
primary goal is to enable machine learning models to comprehend and integrate
different forms of data, such as combining text, images, and audio into a
cohesive understanding. The challenge in multimodal learning lies in
effectively capturing and integrating the nuanced and often complex
relationships between different data types. This requires recognizing patterns
within each modality and understanding how these patterns interact and
complement each other across modalities. The applications of multimodal
learning are diverse and far-reaching, enhancing accessibility through
integrated speech, text, and visual information services and contributing to
more immersive experiences in areas like virtual reality and interactive
systems. As the field evolves, it holds the potential to revolutionize human-
computer interaction, content creation, and a multitude of other sectors by
enabling a more sophisticated and intuitive understanding of complex,
multimodal data.

Figure 1. Recent works for cross-modality learning focus on different modality
mismatch scenarios. This paper aims to establish a general method for all
mismatch types and combinations. Specifically, our experiments focus on
generalizability that could be transferred to an unseen modality at test time.

The motivation for this research arises from the dynamic nature of real-world
data, where modalities can unpredictably vary or be absent at inference time.
Traditional multimodal learning methods, typically fixated on static modality
combinations during both training and inference, fall short in such fluid
environments. Although foundation models pre-trained on vast datasets across
numerous modalities are frequently employed to encode these modalities into
embeddings, changes in the input modality necessitate retraining the
discriminative downstream model for accurate predictions. For instance, a
classifier trained on embeddings from images and tabular data might not be
suitable for inferring audio inputs due to disparities in embedding dimensions
and semantic content. To overcome these challenges, our research leverages
text’s capacity to serve as a unified semantic space alongside the extensive
zero-shot prediction capabilities of Large Language Models (LLMs). Text’s
modality-invariant nature provides a versatile bridge across different data
types, potentially circumventing issues like modality collapse and extending
generalizability to unseen modalities. Moreover, advanced text analysis tasks
such as translation, summarization, and explanation have been extensively
researched and integrated into Large Language Models (LLMs). This integration
offers powerful capabilities to align various modalities effectively. By
employing foundation models to convert diverse modalities into text, we
propose a novel and flexible multimodal learning framework, offering a
promising solution to the limitations of current multimodal learning
methodologies for unseen modalities.

Our research objective is to develop a downstream model that is invariant
across modalities, capable of being trained on data of certain modalities and
performing zero-shot predictions on unseen modalities at test time. For
instance, assuming only image, tabular, and text data are available during
training, we want to train a model that can take video inputs during inference
time, as shown in Figure 3. Note that, as we will demonstrate through
experiments, simply converting all modalities into text for training and
inference is insufficient, as the mismatch between different modalities does
not translate and align as seamlessly in text, especially in complex scenarios
involving multiple modality combinations. To address these challenges, our
work has extensively explored challenges such as modality alignment,
translation, missing modality imputation, and generation. Our goal is to
pioneer methods in multimodal learning that leverage text representations and
in-context learning, thereby establishing a seamless integration across
various modalities. Furthermore, we aim to explore the balance between
flexibility and performance when employing text representations for multimodal
learning, in contrast to traditional embedding approaches.

TAMML employs Large Language Models (LLMs) for data transformation across
various modalities, with the aim of creating a unified semantic space. This
process is conducted exclusively through in-context learning. Initially, we
transform different modalities into text. Recently, various solutions have
been developed, such as GPT-4 (OpenAI, 2023), Blip2 (Li et al., 2023) for
vision, and TabLLM (Hegselmann et al., 2023) for tabular data. Following this,
we engage LLMs in text-style translation across modalities, ensuring that all
modalities in their textual representation adopt a consistent linguistic
structure, thereby reducing the gap between different modalities. To further
align these modalities within a closer semantic space, remove redundant
information, and mitigate the heterogeneity inherent in text data from diverse
sources, we further conduct modality summarization. This step involves a
concise summarization of the translated data. Additionally, TAMML includes a
reasoning augmentation step akin to the Chain-of-Thought (Wei et al., 2022)
method, where we enhance the data with LLMs to boost prediction and judgment
capabilities. Moreover, we leverage LLMs as a source of large-scale external
knowledge, enriching the data understanding and interpretative depth (Chen et
al., 2023). We would like to answer several hypotheses through extensive
experiments. First, whether TAMML is more effective compared with the existing
solutions in predicting data of unseen modality. Second, although this work
mainly focuses on the prediction given unseen modality, we would like to
understand whether the proposed solution is still effective when the modality
in testing is already seen during training. Third, whether the text-as-the-
medium strategy is generally more robust compared to the solutions focusing on
embedding-based cross-modality transfer. We benchmarked TAMML against existing
methodologies in closely related tasks, particularly focusing on zero-shot
learning cross-modality translation, which involves translating unseen source
data to a different target domain. Techniques like MIDiffusion (Wang et al.,
2023) and SDEdit (Meng et al., 2022) demonstrate commendable performance in
tasks such as domain translation within images. However, these methods
encounter challenges when the source and target domains represent completely
different modalities.

Our contributions can be summarized as follows:

  * •

We investigate the potential advantage of using LLMs and text representation
for multimodal learning. We propose TAMML , an in-context cross-modality
translation method that utilizes foundation models to tackle training/testing
modality mismatch and generalize to any unseen modality at test time.

  * •

We demonstrate TAMML can significantly outperform SOTA approaches by
conducting multiple experiments on real-world datasets. We also have an
ablation study to analyze the effectiveness of each component in TAMML .

  * •

Additional experiments further verify that even when the testing modality is
already seen during training, TAMML can still outperform the competitors by a
large margin.

The remainder of this paper is organized as follows: Section 2 reviews
preliminary works on Multimodal Foundation Models and Zero-shot Cross Modality
Translation. Section 3 introduces TAMML , detailing how we utilize LLMs to
transform between different modalities and adapt to unseen inference
modalities. Section 4 presents our experimental results and demonstrates the
effectiveness of TAMML , including ablation studies on the individual
components of TAMML . Section 5 provides further analysis, comparing text and
embedding representations through visualization. The paper concludes with
Section 6, summarizing our findings and discussing limitations and future
directions.

Figure 2.  Traditional downstream training relies on embeddings extracted from
upstream foundation models, with one foundation model designated for each
modality. This approach limits the downstream model’s ability to adapt to
unseen modalities at test time without undergoing complete retraining.
Previous research has addressed this issue by implementing zero-shot cross-
modality translations during the inference phase.

##  2\. Related Works

In this section, we begin by briefly exploring the advancements of multimodal
foundation models in Section 2.1. Next, we discuss zero-shot learning cross-
modality translation in Section 2.2.

###  2.1. Multimodal Foundation Models

Recent advancements in large-scale foundation models have significantly
enhanced content generation capabilities across various modalities. These
developments span a wide range of applications, including text-to-image
(Ramesh et al., 2021; Rombach et al., 2022), text-to-video (Singer et al.,
2022), audio-to-image (Jamaludin et al., 2019), text-to-speech (Ren et al.,
2019), speech-to-gesture (Ahuja et al., 2020), speaker-to-listener (Ng et al.,
2022), language-to-pose (Ahuja and Morency, 2019), and even in the realms of
speech and music generation (Oord et al., 2018). Nevertheless, aligning the
semantic spaces of independently trained foundation models poses a significant
challenge, hindering the ability of downstream models to seamlessly switch
between upstream modalities at test time.

Extending beyond single-modality applications, large multimodal language
models (MLLMs) have shown remarkable proficiency in both reasoning and
generation tasks (Yin et al., 2023). For instance, Flamingo (Alayrac et al.,
2022) employs a vision encoder for feature extraction from images and utilizes
a resampler to reduce computational complexity. This is achieved by learning a
predefined number of latent input queries, which are then processed by a
Transformer that cross-attends to the visual features. On the other hand,
Kosmos-2 (Peng et al., 2023) adopts a more streamlined architecture,
comprising only a vision encoder, a linear projector, and a base transformer.
This model directly concatenates image and text features, using unique tokens
for differentiation. LLaVA (Liu et al., 2023) represents a novel end-to-end
trained large multimodal model that combines a vision encoder and Vicuna
(Chiang et al., 2023) for general-purpose visual and language understanding,
achieving impressive chat capabilities mimicking spirits of the multimodal
GPT-4. However, despite the impressive capabilities of MLLMs in handling
multimodal data, acquiring large-scale datasets encompassing multiple
modalities remains a significant challenge.

Figure 3.  In the training phase, each modality in raw input is first
transformed into text representations using a corresponding foundation model.
For example, we employ an image caption model for images and TabLLM
(Hegselmann et al., 2023) to transform tabular data. Following the modality
transformation, we utilize LLM for text-style translation across modalities,
applying a one-shot in-context learning approach to adapt the linguistic style
as anticipated during inference. After text-style translation, modality
summarization and reasoning augmentation are applied in parallel. Finally, the
output texts are concatenated as the training inputs to a transformer model
for downstream prediction. The inference phase follows a similar pattern, with
the notable exception of bypassing the translation module to preserve the
original linguistic style.

###  2.2. Zero-shot Learning Cross Modality Translation

The challenge of cross-modality data translation without access to source
modal data leads to a Zero-shot-Learning-based approach for this task. A
fundamental difficulty with learning-based methods is their limited capability
in handling unseen data classes (Wang et al., 2021; Bucher et al., 2017;
Kuchibhotla et al., 2022). Zero-shot-Learning emerges as a robust strategy for
scenarios where training and test classes do not overlap. Traditional Zero-
shot Learning methods often seek a direct projection from the image feature
space to a semantic space. This is achieved through discriminative methods
(Palatucci et al., 2009; Akata et al., 2015) and generative models (Long et
al., 2017; Wang et al., 2018). In the realm of GAN-based zero-shot-learning
for cross-modality data translation, models typically modify the latent
representation of a pre-trained GAN, a process known as GAN inversion (Zhu et
al., 2020; Shi et al., 2022; Brock et al., 2016; Abdal et al., 2020; Da Tsai
and De Lin, 2022). Perturbation-diffusion-based approaches to cross-modality
data translation facilitate Zero-shot learning for these tasks (Cheng et al.,
2023; Ho et al., 2020; Kawar et al., 2022; Song et al., 2020b), showing
impressive results when the numerical features of the source and target
domains align (Meng et al., 2022). However, these methods can struggle when
there is a significant disparity in the cross-domain appearance features.

##  3\. Methodologies

This section focuses on how TAMML makes training modality adapt to unseen
testing modality and unseen modality combinations. In Section 3.1, we define
the problem scenario and its notation of our research, and Section 3.2
describes the individual text transformation method to different modalities.
Next, we introduce our main algorithm to connect information from multiple
modalities. Section 3.3 translating training modalities to inference
modalities. Section 3.4 summarizes different modalities into human-written
descriptions, and Section 3.5 leverages external knowledge as data
augmentation. At last, Section 3.6 elucidates the flow of training and
inference within the downstream task.

###  3.1. Problem Formalization

Suppose we have a set $M$ of $p$ modalities
$M=\\{m_{1},m_{2},\ldots,m_{p}\\}$. In the training phase, a subset of
modalities $M_{T}\subseteq M$ is used. In the inference phase, a different
subset $M_{I}\subseteq M$ is utilized, with the critical condition that
$M_{T}\cap M_{I}=\emptyset$ to ensure no overlap in modalities between
training and inference.

Within this framework, we define two distinct datasets for the training and
inference phase. The training dataset $D_{T}$ is composed of a set of $n_{T}$
samples, and each sample $x$ is restricted to $M_{T}$, denoted as
$D_{T}=\\{(x_{M_{T}}^{i},y^{i})\\}_{i=1}^{n_{T}}$. Similarly, the inference
dataset consists of a set of $n_{I}$ samples restricted to $M_{I}$, formalized
as $D_{I}=\\{(x_{M_{I}}^{i},y^{i})\\}_{i=1}^{n_{I}}$. Our algorithms are
designed to build the model $F$ on $D_{T}$ and evaluate unseen data with
unseen modality combinations in $D_{I}$ to measure the model’s ability to
generalize knowledge in zero-shot multimodal learning. Besides, we design a
dataset $D_{E}$ containing $k$ samples, each of them is restricted to $M$.
This dataset is used when we prompt.

###  3.2. Text Transformation

For the challenge of the heterogeneity between different modalities, we
attempt to map the data of different modalities into a similar semantic space
at the raw input stage, integrating modalities with minimal processing.

For image modality, we utilize the state-of-the-art image caption model, which
allows us to take images and answer questions about them. In this stage, the
image caption model is prompted to describe images and generate detailed text
output, effectively translating visual content into descriptive text. For
table modality, we implement the simple text serialization template, which is
the textual enumeration of all features as ”The column name values,” proposed
by TabLLM(Hegselmann et al., 2023). According to TabLLM’s study, this template
is proven to outperform zero-shot tabular learning in the LLM. For text
modality, we maintain human-written sentence forms without conversion to
maintain their original natural linguistic structure. After transformation,
the sentence order of each sample $x$ is determined by the order of modalities
in $M$ to concatenate the text sentences and is used as the input for the next
step. We have included and compared various SOTA image caption models and LLMs
in Appendix C.2.

###  3.3. Text-style Translation across Modality

After text transformation, we obtain textual representation for each modality.
Next, we employ Large Language Models (LLMs) to perform text-style translation
across different modalities. This is because, although all types of data are
converted into textual representation, there are still syntactic and semantic
gaps between the transformed text across different modalities, which are
highly dependent on the foundation models used for transformation. For
instance, the textual representation of tabular data can follow a static
pattern (e.g., ”the type of the pet is Cat; the name of the pet is Kali,”
etc.), while the textual representation of the image is affected by the
language style of the image caption model (e.g., ”the image shows that a young
kitty with a coat…”). This divergence in textual representation leads to gaps
in semantic space distribution. Our solution is to adopt in-context learning
in LLM to translate the text style to the one in the target space, reducing
the mismatch between training and inference data. Text-style Translation
function $T$ can be formulated as follows:

$T:M_{T}\rightarrow M_{I}$

Specifically, we enhance the capability of the LLM to effectively interpret
the semantic content and convert the modality text from $M_{T}$ format into
$M_{I}$ format. This transformation is achieved through few-shot in-context
learning, where LLM is exposed to three examples, which are samples from
$D_{E}$, illustrating the desired translation within the context of our
modality combination for training and inference purposes. Moreover, to
mitigate information loss during conversion, we specifically emphasize that
the integrity of the original information should be maintained in our prompt.
This emphasis ensures that the essential information content and cross-modal
connections are preserved during the cross-modal translation process. Appendix
A Fig. 5 demonstrates the details of the Text-style Translation prompt.

To achieve the two effects of reduction of the distance gap between modalities
and information preservation, the LLM needs to achieve $T$ is optimized to
minimize the semantic distance $d$ between the $D_{T}$ and $D_{I}$, while
preserving the core information content $I_{c}$. Thus, the LLM optimization
objective can be formulated as:

$\min_{T}d(T(D_{T}),D_{I})\quad\text{subject to}\quad
I_{c}(T(D_{T}))=I_{c}(D_{T})$

###  3.4. Modality Summarization

Compared with text-style translation, which maps the training distribution
directly to the inference distribution, modality summarization involves
utilizing an LLM to summarize the translated data. This step aims to achieve
two main objectives: (1) It facilitates the alignment of different modalities
in a closer semantic space and further eliminates the heterogeneity of text
representation of diverse modalities. As discussed in the previous section,
varying text transformation methods cause gaps in semantic space. TAMML
further generalizes the text representation across all modalities with similar
linguistic styles and formats. (2) It identifies the connections and
interactions among elements of different modalities. This process enhances
information quality by promoting interaction, such as generating novel
information, highlighting shared core information, and eliminating redundant
information.

The modality summarization is also generated by LLM. Our experiment is divided
into two steps. First, we extract a sample in $D_{E}$ with the inference
modality combination. We pre-define the linguistic styles in the prompt,
instructing the LLM to integrate information across diverse modalities to
produce a summarized output. Second, the output is merged back to our original
prompt to form a demonstration, and then one-shot in-context learning is
applied to each sample x in $T(D_{T})$. Each sample may follow the textual
structure of the demonstration established in the first step, aiming to reduce
heterogeneity. Appendix A Fig. 6 demonstrates the details of the modality
summarization prompt.

###  3.5. LLM Reasoning Augmentation

We apply LLMs in terms of reasoning similar to Chain-of-Thought (Wei et al.,
2022) method and leverage LLMs as a source of large-scale external knowledge
similar to (Chen et al., 2023) to achieve data augmentation.

First, the LLM was assigned a specific prediction task accompanied by
straightforward instructions and examples to leverage its external knowledge
while analyzing the concepts and information present in the original textual
inputs. Throughout the task, the LLM will make predictions and provide
comprehensive explanations for each input sample. By utilizing the outputs
derived from this predictive reasoning process, we enhance the original input
information and achieve data augmentation. Next, for each reasoning process,
the LLM was furnished with pertinent background information and examples
related to its prediction tasks. It is subsequently assigned an assistant role
to provide informative reasoning and accurately predict either objective
values in regression tasks or labels in classification tasks. Appendix A Fig.
7 demonstrates the details of the reasoning augmentation prompt.

###  3.6. Downstream Training

This section elucidates the entire flow of training and inference within
downstream tasks. The process is shown in Fig. 3. Initially, the raw input
contains different modalities, which are each very heterogeneous in
expression. First, we transform each modality into text and conduct the text-
style translation step. In parallel, the text representation is then subjected
to two distinct yet complementary processes: modality summarization and
reasoning augmentation. The summarization aligns textual representation in
diverse modalities, while the reasoning augmentation step integrates these
points with external knowledge, thereby enhancing the overall informational
context.

The transformed text representation is then fed into a transformer. More
specifically, we adopted Longformer(Beltagy et al., 2020) as the transformer
architecture. After mean pooling is applied to aggregate the contextual
representations, a Multilayer Perceptron (MLP) is employed to process the
pooled features and output the predictions. The loss function employed is
contingent on the downstream task.

##  4\. Experiments

Here, we articulate our hypotheses and address the research questions to
evaluate the effectiveness of TAMML . Q1: Under modality mismatch scenarios,
is TAMML better than the embedding-based SOTA solutions? Q2 (follow Q1): Is
TAMML still effective for situations in which the testing modality has been
involved during training? (i.e. training: all modalities, testing: some of the
modalities) Q3: Is text representation generally more robust than embedding
representation for cross-modality translation? Detailed descriptions of our
experimental setup, including model checkpoints, hyperparameters, and dataset
specifics, are provided in Appendix B. Additional inquiries, Q4 and Q5, are
addressed in Appendix D.

Setup In this section, we primarily shown results from GPT-3.5-turbo as our
large language models (LLMs) and GPT-4-vision for image captioning, unless
specified otherwise. For additional results involving other LLMs and image
caption models, please see Appendix C.2.

###  4.1. Q1: Under Modality Mismatch Scenarios, How Does TAMML Compare To the
SOTA?

In this section, we focus on modality mismatch scenarios where training and
testing modalities are completely different. We mainly compare our results to
several zero-shot cross-modality data translation methods.

####  4.1.1. Dataset

PetFinder.my Adoption Prediction (Addison Howard, 2018) examines the
predictive factors of pet adoptability, with a particular focus on the
rapidity of adoption post-listing. The dataset is a composite of the following
modalities:

  * •

Text modality contains the human-written description of the status of the pet

  * •

Image modality contains a profile photo for the pet

  * •

Tabular modality contains basic information, such as gender and age, of the
pet.

Airbnb Pricing Prediction (ins, 2023) is a composite of the following
modalities that we used for making a regression prediction of housing prices
in the Hawaii market:

  * •

Text modality contains the human-written description of the homestay, the
neighborhood description, and the host’s profile.

  * •

Image modality contains images of the homestay

  * •

Tabular modality delivers essential details such as location, rating score,
and review counts.

Training | Testing | PetFinder — Accuracy $\uparrow$ | Airbnb — MSE $\downarrow$
---|---|---|---
|  | SDEdit | DDRM | Idinvert | TAMML | SDEdit | DDRM | Idinvert | TAMML
text+image | tabular | 0.282 | 0.291 | 0.279 | 0.356 | 0.935 | 0.600 | 0.799 | 0.244
text+tabular | image | 0.289 | 0.277 | 0.286 | 0.355 | 0.656 | 0.778 | 0.643 | 0.394
image+tabular | text | 0.281 | 0.297 | 0.279 | 0.330 | 0.514 | 0.565 | 0.781 | 0.285
text | image+tabular | 0.291 | 0.283 | 0.289 | 0.343 | 1.548 | 0.914 | 0.915 | 0.216
text | image | 0.289 | 0.276 | 0.287 | 0.350 | 1.513 | 0.895 | 1.010 | 0.404
text | tabular | 0.293 | 0.259 | 0.277 | 0.342 | 1.061 | 0.824 | 0.931 | 0.233
image | text+tabular | 0.290 | 0.297 | 0.284 | 0.328 | 0.556 | 0.530 | 0.602 | 0.391
image | text | 0.288 | 0.282 | 0.280 | 0.324 | 0.678 | 0.589 | 0.759 | 0.423
image | tabular | 0.291 | 0.287 | 0.284 | 0.364 | 0.592 | 0.538 | 0.516 | 0.428
tabular | text+image | 0.290 | 0.271 | 0.285 | 0.355 | 0.637 | 0.675 | 0.662 | 0.290
tabular | text | 0.289 | 0.265 | 0.280 | 0.326 | 0.569 | 0.693 | 0.707 | 0.309
tabular | image | 0.289 | 0.263 | 0.277 | 0.357 | 0.609 | 0.715 | 0.615 | 0.408
Average | 0.289 | 0.279 | 0.282 | 0.344 | 0.822 | 0.693 | 0.745 | 0.335
Table 1. This table presents a detailed comparison, highlighting TAMML ’s
performance against all baseline models under modality mismatch scenarios.
PetFinder dataset uses accuracy as the key evaluation metric. Airbnb dataset
uses Mean Squared Error (MSE) as the key evaluation metric. Avito dataset uses
Root Mean Squared Error (RMSE) as the key evaluation metric.

####  4.1.2. Competitors

Here, we adopt the embedding-based cross-modality translation solutions. A
high-level illustration of conventional embedding-based downstream training is
demonstrated in Figure 2. In our experiments, we applied various foundation
models, each for different modalities, as an encoder that extracts embedding
representations, after which alignment layers are used for learning and
concatenating each representation. The details of each foundation model and
its usage are in Appendix B.4. Following these modality encoders, we
concatenate the representative outputs from all modalities and feed them into
a transformer encoder. After fine-tuning, the model generates cross-modality
alignment outputs.

In the experiments, we compared TAMML to several zero-shot cross-modality data
translation methods built on top of embedding-based downstream model training.
We include two perturbation-diffusion-based methods, SDEdit (Meng et al.,
2022), DDRM (Kawar et al., 2022), and one GAN inversion method, In-domain
inversion GAN (Idinvert) (Zhu et al., 2020). All these methods include
training a generative model (diffusion or a GAN) that can translate the input
embedding distribution into a distribution similar to training modalities. At
test time, we use the trained generative models to generate synthesis modality
embedding (similar to the distribution of training modalities) for various
unseen modalities. For diffusion models, we train a score-based generative
model (Ho et al., 2020) and a backbone model for every modality combination.
In our implementation, we leverage DDIM (Song et al., 2020a) as the pre-
trained model for both SDEdit and DDRM. For GAN, we leverage the StyleGAN
(Karras et al., 2019) implementation as the backbone model.

####  4.1.3. Results

The key findings outlined in Table 1 underscore the superior performance of
TAMML , which achieves substantial gains over competing baselines across
various modality combinations. These results suggest that the proposed
strategy, which integrates LLMs’ in-context learning with foundation models,
holds a decisive edge over all existing methods. Specifically, on the
PetFinder dataset, TAMML enhances accuracy by an average of approximately 21%,
significantly outperforming other methods. Similarly, in the Airbnb dataset,
TAMML achieves an average reduction in mean square error of around 54%,
dwarfing the maximum 16% error reduction seen with alternative baselines.

###  4.2. Q2: Is the Proposed Solution Still Effective when there is no
modality mismatching?

This section presents experiments designed to assess the performance of TAMML
given all modalities are available during training. This setup evaluates
whether TAMML can surpass embedding-based solutions even in scenarios without
modality mismatches. The analysis primarily focuses on comparing embedding-
based methods that leverage pre-trained foundation models for modality
processing.

Besides the two datasets introduced in Section 4.1.1, we have included one
additional dataset. Avito Demand Prediction
111https://www.kaggle.com/competitions/avito-demand-prediction/data predicts
the likelihood of an ad selling something based on user item and context
features. The dataset is a composite of the following modalities:

  * •

Text modality contains the ad title and description.

  * •

Image modality contains a profile photo of the item.

  * •

Tabular modality contains basic information, such as region, city, item
category, ad price, etc.

####  4.2.1. Baseline

In this section, ”embedding-based solution” refers to the process of
extracting each input modality as embeddings, rather than text, for the
downstream discriminative model from pre-trained foundation models, given that
all modalities are known and available at training time. The foundation models
utilized to extract these embedding representations are the same as those
discussed in Section 4.1.2. During the experiments, these embeddings are
concatenated for downstream training.

Testing | Pet — Acc $\uparrow$ | Airbnb — MSE $\downarrow$ | Avito — RMSE $\downarrow$
---|---|---|---
| Emb | TAMML | Emb | TAMML | Emb | TAMML
tabular | 0.297 | 0.370 | 0.336 | 0.208 | 0.107 | 0.045
image | 0.286 | 0.360 | 0.501 | 0.359 | 0.097 | 0.045
text | 0.284 | 0.338 | 0.764 | 0.264 | 0.110 | 0.046
image+tabular | 0.336 | 0.404 | 0.298 | 0.226 | 0.093 | 0.044
text+tabular | 0.384 | 0.374 | 0.224 | 0.212 | 0.102 | 0.043
text+image | 0.296 | 0.383 | 0.487 | 0.247 | 0.096 | 0.044
all | 0.396 | 0.413 | 0.236 | 0.230 | 0.090 | 0.043
Average | 0.326 | 0.377 | 0.407 | 0.249 | 0.099 | 0.044
Table 2. This table presents a detailed comparison, highlighting TAMML ’s
performance against embedding-based solutions when the model is trained on all
modalities and tested on different subset modalities.

####  4.2.2. Results

The key findings outlined in Table 2 underscore the superior performance of
TAMML , which still achieves substantial gains over competing baselines across
various modality combinations. These results suggest that despite no modality
mismatching, our strategy holds a decisive edge over embedding-based methods.
Specifically, on the PetFinder dataset, our technique enhances accuracy by an
average of approximately 16%, significantly outperforming embedding-based
methods. Similarly, in the Airbnb dataset, TAMML achieves a decrease of
approximately 38% in mean square error, indicating a significant improvement
in prediction accuracy. Moreover, in the Avito dataset, the decrease is even
more pronounced, with a reduction of approximately 55% in root mean square
error when applying TAMML .

###  4.3. Q3: Is Text Representation Generally More Robust Than Embedding
Representation For Cross Modality Translation?

In this section, we aimed to understand the trade-off between performance and
flexibility when converting various modalities from embedding into text,
especially under modality mismatch conditions.

####  4.3.1. MLLMs Baseline

Previous experiments in Section 4.1 and Section 4.2 cannot compare text
representation and embedding representation since converting modalities into
text involves different foundation models, each with different capabilities.
For a fair comparison of performance between text representation and embedding
representation, the most appropriate approach is to utilize multimodal
Language Model Models (MLLMs). This ensures fairness in the comparison because
all modalities are converted from the same foundation model. Therefore, we
applied the following SOTA MLLMs in our experiments: Kosmos-2 (Peng et al.,
2023) and Flamingo (Alayrac et al., 2022). In our experiments, we leverage
MLLMs as both pre-trained feature extractors and text decoders. We then employ
mean pooling to aggregate representations, followed by using an MLP as a
backbone model to generate predictions. We aim to compare the performance gaps
between a downstream model trained on images and another trained on image
captions (attributed to the dataset).

####  4.3.2. Results

Results in Table 3 consistently reveal that downstream models trained on image
captions exhibit less performance degradation compared to those trained on
image embeddings in scenarios of modality mismatch. This observation holds
true across all state-of-the-art multimodal LLMs we investigated. Such results
strongly suggest that cross-modality translation within text representations,
as facilitated by TAMML , proves to be a more effective and robust strategy
than utilizing embedding representations when faced with modality mismatch
conditions.

Pet — Acc $\uparrow$ | Flamingo | Kosmos2
---|---|---
Test/Train | caption | image | caption | image
text | -0.07 | -0.10 | -0.09 | -0.11
tabular | -0.08 | -0.10 | -0.12 | -0.21
text+tabular | -0.08 | -0.11 | -0.10 | -0.15
Air — MSE $\downarrow$ | Flamingo | Kosmos2
Test/Train | caption | image | caption | image
text | -0.00 | -0.06 | -0.01 | -0.03
tabular | -0.03 | -0.07 | -0.05 | -0.05
text+tabular | -0.02 | -0.07 | -0.04 | -0.03
Table 3. Text representation shows consistently less performance degradation
for cross-modality translation when explicitly compared to embedding
representation. Both representations are derived from the same Multimodal LLMs
for fair comparison. Nevertheless, transforming from image to caption has a
slight performance drawdown.

###  4.4. Ablation Studies

This section explores the contribution of individual components within TAMML
by conducting ablation studies. We incrementally added modules to evaluate
their impact on performance, with findings summarized in Table 4.

Training | Testing | PetFinder — Accuracy $\uparrow$ |
---|---|---|---
|  | SDEdit | Text | +Modality | +Text-style | +Reasoning
|  | Transformation | Summarization | Translation | Augmentation
text+image | tabular | 0.282 | 0.310 | 0.321 | 0.361 | 0.356
text+tabular | image | 0.289 | 0.329 | 0.365 | 0.374 | 0.355
image+tabular | text | 0.281 | 0.305 | 0.295 | 0.331 | 0.330
text | image+tabular | 0.291 | 0.282 | 0.296 | 0.345 | 0.343
text | image | 0.289 | 0.293 | 0.298 | 0.357 | 0.350
text | tabular | 0.293 | 0.297 | 0.318 | 0.311 | 0.342
image | text+tabular | 0.290 | 0.314 | 0.289 | 0.330 | 0.328
image | text | 0.288 | 0.306 | 0.330 | 0.327 | 0.324
image | tabular | 0.291 | 0.300 | 0.307 | 0.365 | 0.364
tabular | text+image | 0.290 | 0.194 | 0.366 | 0.351 | 0.355
tabular | text | 0.289 | 0.193 | 0.306 | 0.309 | 0.326
tabular | image | 0.289 | 0.196 | 0.357 | 0.357 | 0.357
Average $\pm$ Variance ($\times 10^{-4}$) | $0.289\pm 0.12$ | $0.277\pm 25.91$ | $0.321\pm 7.84$ | $0.343\pm 4.53$ | $\textbf{0.344}\pm\textbf{1.97}$
Table 4. Ablation studies on various components of TAMML . Our observations
reveal that text transformations significantly enhance performance across all
modality combinations except for tabular data, which are in fixed formatted
text. The formatting issue is effectively solved by incorporating a
summarization module, resulting in a substantial enhancement in performance.
Furthermore, the inclusion of both the translation module and the reasoning
augmentation module leads to further improvements in overall performance.

####  4.4.1. Text Transformation

Compared to the embedding-based methods SDEdit, analysis from Table 4 shows
converting modality features into text enhances performance by approximately
2%, indicating less modality mismatch during training and inference compared
to embedding representations. This improvement is consistent across most data
modalities, except for tabular data, which sees about a 10% decline. This
discrepancy is attributed to the fixed format of tabular text transformation,
highlighting a significant style gap with more fluid, human-like writing,
particularly impacting tabular data’s inference performance.

####  4.4.2. Modality Summarization

Table 4 results indicate modality summarization improves tabular data accuracy
significantly from 0.277 to 0.321 on average. After this stage, TAMML has
already outperformed the strongest competitor SDEdit. This suggests that
summarization effectively standardizes text formats into a cohesive style,
mitigating heterogeneity in text transformation and enhancing data format
alignment.

####  4.4.3. Text-Style Translation across Modality

According to Table 4, text-style translation bridges training and inference
phase gaps, with about 7% improvements from 0.321 to 0.343. This enhancement
is particularly notable when the gap in textual style remains consistent
across phases, as seen in the image-to-table scenarios. Such consistency aids
in more accurate mapping function determination by the model.

####  4.4.4. Reasoning Augmentation

Incorporating reasoning augmentation, as indicated in Table 4, may not
necessarily yield significantly better performance compared to other
components. However, we have observed that it contributes to a more stable
performance across different scenarios. The variance value with augmentation
is substantially lower than that without it.

##  5\. Analysis and Discussion

In this section, we delve into a series of analyses and discussions,
extracting valuable insights from our discoveries. Specifically, we provide
more supportive evidence with visualization and distribution distance
measurements.

###  5.1. Visualization for Distribution Alignment

In Section 4.3, we have validated the effectiveness of text transformation in
TAMML through experimental performance. Further, we visualized 1,400 data
points in these modalities with their position-aware embeddings using UMAP
(McInnes et al., 2018) in Fig. 4. The left figure illustrates the original
distributions of image and text embeddings, while the right figure displays
the corresponding distributions after the summarization module in TAMML . We
observe that the distribution boundaries between image and text modalities
become less distinct, which indicates they are closer in the semantic space.
To be more precise, TAMML significantly reduces the average instance Euclidean
distance between image and text in the semantic space from $10.213$ to
$0.411$.

###  5.2. Effects of the Foundation Models

Some might argue that the exceptional performance of TAMML could be attributed
to the superior GPT-based foundation models. To investigate this, we replaced
the different foundation model in our architecture with smaller open-source
models like LLaMA for text and Blip2 for image captioning. We conducted
ablation studies focusing on the performance of four various open-source LLMs
and four image foundation models. Specifically, we showed that our approach
maintains strong performance even with a small model size. Table 5 showcases
the results averaged across twelve training-inference modality combinations
(detailed shown in Appendix C.2, Tables 9 and 10). The results suggest that
using smaller foundation models does not necessarily result in significantly
inferior performance within TAMML .

Figure 4. The left and right pictures of the figure illustrate the
visualizations of embeddings for image and text data, respectively, before and
after our processes. LLMs
---
Pet — Acc $\uparrow$ | LLaMA-13b | LLaMA-70b | Mixtral-8x7b | GPT-3.5
Average | 0.302 | 0.317 | 0.320 | 0.321
Image Caption Models
Pet — Acc $\uparrow$ | Blip2 | Kosmos2 | Flamingo | GPT4
Average | 0.303 | 0.299 | 0.293 | 0.307
Table 5. Foundation model comparison: Each number presented here is an average
derived from detailed experiments outlined in Appendix C.2. In general, we can
infer that the foundation model has only a limited impact on TAMML .

##  6\. Conclusion and Future Directions

Our study has effectively harnessed Large Language Models (LLMs) for
multimodal learning, creating a unified semantic space that integrates various
data modalities through text. Through techniques such as text transformation,
text-style translation, summarization, and reasoning augmentation, we have
demonstrated that operations performed in the text domain using in-context
learning with LLMs can achieve comparable performance to traditional methods
operating in embedding space. This approach not only opens new avenues in
multimodal learning but also underscores the significant potential and
advantages of text as a unifying medium. Future efforts will focus on refining
TAMML for broader multimodal tasks and overcoming prompt sensitivity to fully
leverage text in multimodal learning.

## References

  * (1)
  * ins (2023) 2023\.  Inside Airbnb : Hawaii.  http://insideairbnb.com/get-the-data Accessed on: 10 September, 2023.
  * Abdal et al. (2020) Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020.  Image2stylegan++: How to edit the embedded images?. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8296–8305.
  * Addison Howard (2018) Mongrel Jedi Addison Howard, MichaelApers. 2018.  PetFinder.my Adoption Prediction.  https://kaggle.com/competitions/petfinder-adoption-prediction
  * Ahuja et al. (2020) Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. 2020.  Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_. Springer, 248–265.
  * Ahuja and Morency (2019) Chaitanya Ahuja and Louis-Philippe Morency. 2019.  Language2pose: Natural language grounded pose forecasting. In _2019 International Conference on 3D Vision (3DV)_. IEEE, 719–728.
  * Akata et al. (2015) Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2015.  Label-embedding for image classification.  _IEEE transactions on pattern analysis and machine intelligence_ 38, 7 (2015), 1425–1438.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ 35 (2022), 23716–23736.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Brock et al. (2016) Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. 2016.  Neural photo editing with introspective adversarial networks.  _arXiv preprint arXiv:1609.07093_ (2016).
  * Bucher et al. (2017) Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. 2017.  Generating visual representations for zero-shot classification. In _Proceedings of the IEEE International Conference on Computer Vision Workshops_. 2666–2673.
  * Chen et al. (2023) Haoxing Chen, Yaohui Li, Yan Hong, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Huijia Zhu, and Weiqiang Wang. 2023.  Boosting Audio-visual Zero-shot Learning with Large Language Models.  _arXiv preprint arXiv: 2311.12268_ (2023).
  * Cheng et al. (2023) Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hung-Yu Tseng, and Hsin-Ying Lee. 2023.  Adaptively-realistic image generation from stroke and sketch with diffusion model. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 4054–4062.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023\.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ (2023).
  * Da Tsai and De Lin (2022) Yun Da Tsai and Shou De Lin. 2022.  Fast online inference for nonlinear contextual bandit based on Generative Adversarial Network.  _arXiv preprint arXiv:2202.08867_ (2022).
  * Dosovitskiy et al. (2010) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2010\.  An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020.  _arXiv preprint arXiv:2010.11929_ (2010).
  * Gorishniy et al. (2021) Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021.  Revisiting deep learning models for tabular data.  _Advances in Neural Information Processing Systems_ 34 (2021), 18932–18943.
  * Hegselmann et al. (2023) Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023.  Tabllm: Few-shot classification of tabular data with large language models. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 5549–5581.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Jamaludin et al. (2019) Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. 2019.  You said that?: Synthesising talking faces from audio.  _International Journal of Computer Vision_ 127 (2019), 1767–1779.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4401–4410.
  * Kawar et al. (2022) Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. 2022.  Denoising diffusion restoration models.  _Advances in Neural Information Processing Systems_ 35 (2022), 23593–23606.
  * Kuchibhotla et al. (2022) Hari Chandana Kuchibhotla, Sumitra S Malagi, Shivam Chandhok, and Vineeth N Balasubramanian. 2022.  Unseen Classes at a Later Time? No Problem. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9245–9254.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual Instruction Tuning.
  * Long et al. (2017) Yang Long, Li Liu, Fumin Shen, Ling Shao, and Xuelong Li. 2017.  Zero-shot learning using synthesised unseen visual data with diffusion regularisation.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 10 (2017), 2498–2512.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ (2018).
  * Meng et al. (2022) C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. 2022.  SDEdit:Guided image synthesis and editing with stochastic differential equations.  _International Conference on Learning Representations_ (2022).
  * Ng et al. (2022) Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. 2022.  Learning to listen: Modeling non-deterministic dyadic facial motion. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20395–20405.
  * Oord et al. (2018) Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. 2018\.  Parallel wavenet: Fast high-fidelity speech synthesis. In _International conference on machine learning_. PMLR, 3918–3926.
  * OpenAI (2023) OpenAI. 2023.  OpenAI models api.  (2023).  https://platform.openai.com/docs/models
  * Palatucci et al. (2009) Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. 2009.  Zero-shot learning with semantic output codes.  _Advances in neural information processing systems_ 22 (2009).
  * Peng et al. (2023) Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023.  Kosmos-2: Grounding Multimodal Large Language Models to the World.  _arXiv preprint arXiv:2306.14824_ (2023).
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation. In _International Conference on Machine Learning_. PMLR, 8821–8831.
  * Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019.  Fastspeech: Fast, robust and controllable text to speech.  _Advances in neural information processing systems_ 32 (2019).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10684–10695.
  * Shi et al. (2022) Yichun Shi, Xiao Yang, Yangyue Wan, and Xiaohui Shen. 2022.  Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11254–11264.
  * Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022\.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ (2022).
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020a.  Denoising Diffusion Implicit Models.  _arXiv:2010.02502_ (October 2020).  https://arxiv.org/abs/2010.02502
  * Song et al. (2023) Shezheng Song, Xiaopeng Li, and Shasha Li. 2023.  How to bridge the gap between modalities: A comprehensive survey on multimodal large language model.  _arXiv preprint arXiv:2311.07594_ (2023).
  * Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020b.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ (2020).
  * Tsai et al. (2023) YunDa Tsai, Mingjie Liu, and Haoxing Ren. 2023.  Rtlfixer: Automatically fixing rtl syntax errors with large language models.  _arXiv preprint arXiv:2311.16543_ (2023).
  * Tzeng et al. (2017) Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017.  Adversarial discriminative domain adaptation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7167–7176.
  * Wang et al. (2018) Wenlin Wang, Yunchen Pu, Vinay Verma, Kai Fan, Yizhe Zhang, Changyou Chen, Piyush Rai, and Lawrence Carin. 2018.  Zero-shot learning via class-conditioned deep generative models. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 32.
  * Wang et al. (2021) Zihao Wang, Clair Vandersteen, Charles Raffaelli, Nicolas Guevara, François Patou, and Hervé Delingette. 2021.  One-shot learning for landmarks detection. In _Deep Generative Models, and Data Augmentation, Labelling, and Imperfections: First Workshop, DGM4MICCAI 2021, and First Workshop, DALI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings 1_. Springer, 163–172.
  * Wang et al. (2023) Zihao Wang, Yingyu Yang, Maxime Sermesant, Hervé Delingette, and Ona Wu. 2023.  Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion.  _arXiv preprint arXiv:2301.13743_ (2023).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Xu et al. (2024) Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. 2024\.  A survey of resource-efficient llm and multimodal foundation models.  _arXiv preprint arXiv:2401.08092_ (2024).
  * Yin et al. (2023) Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023.  A Survey on Multimodal Large Language Models.  _arXiv preprint arXiv:2306.13549_ (2023).
  * Zhang et al. (2024) Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024.  Mm-llms: Recent advances in multimodal large language models.  _arXiv preprint arXiv:2401.13601_ (2024).
  * Zhu et al. (2020) Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020.  In-domain gan inversion for real image editing. In _European conference on computer vision_. Springer, 592–608.

##  Appendix A Prompt Templates

User Prompt:
task description
The following is a dataset about Pet Adoption Prediction. We will predict the
speed at which a pet is adopted based on the pet’s listing on the dataset.
Input contains tabular modality. We previously converted this modality into
text, and we tried to use the combined text to do classification tasks.
modality description
I would like you to rewrite all items in the input as text that represents
image modality
That is, transfer tabular text format ”The (column meaning) is (item)” into a
format that describes the image, i.e., pretending that there is an image in
front of you, and you go to describe it.
Ensure that:
1\. All information should naturally blend together from different modalities,
just like the contents of the same paragraph, and the order should be switched
and blended but not missing any information.
2\. Please start with ”The image”
Few-Shot Examples (Optional):
Input: The type of pet is a Cat. The name of pet is Kali. The age of the pet
when listed (in months) is 3…
Answer: The image shows an adorable grey and black striped cat wearing a tiny
yellow crown with…
Figure 5. Example prompt for text-style translation across modalities. User
Prompt:
task description
The following is a dataset about Pet Adoption Prediction. We will predict the
speed at which a pet is adopted based on the pet’s listing on the dataset.
The input contains multiple modalities, which are text and image modalities.
We previously converted this modality into text, and we tried to use the
combined text to do classification tasks.
I would like you to rewrite all items in the input as natural text.
modality description
Ensure that:
1\. If the original table contains entries under the ”secondary breed” and
”color” columns that are marked as ”Unknown,” the conversion into natural text
will omit any mention of those particular attributes.
2\. Avoid fabricating or inventing any content!
3\. All information should naturally blend together from different modalities,
just like the contents of the same paragraph, and the order should be switched
and blended but not missing any information.
4\. Please start with ”This pet is”.
One-Shot Examples (Generated by LLM in the first step):
Input: The type of pet is Cat. The name of pet is Kali. The age of pet when
listed (in months) is 3…
Answer: This pet is Kali, an adorable grey and black striped cat who carries …
Figure 6. Example prompt for modality summarization. User Prompt:
Background Information
Airbnb revolutionized travel by connecting guests with unique accommodations
worldwide. Its diverse offerings, from cozy homes to exotic locales, cater to
individual preferences. Airbnb offers a range of accommodation options with
varying prices, allowing users to find lodging that suits their budget and
preferences. Prices are determined by hosts based on factors such as location,
amenities, and property type.
Task Description
Imagine you are an assistant booking Airbnb, and you need to assist me since I
want to predict a suitable price range for each Airbnb. I will provide you
with some Airbnb examples, and you need to answer my question about the price
of the given Airbnb information. For every question, explain as detailed as
possible so I can evaluate the Airbnb value reasonably. Few-Shot Examples
Following are examples of different Airbnb items and their price.
Example 1:
Input: The host’s response time to booking inquires…
Answer: Price Range within $\$90$ to $\$110$
Example 2: …
Inference
Now, you are given information on Airbnb data.
Sample Input: The detailed description of listing Airbnb is ….
What price range do you think Airbnb falls within? Give me your answer and
elaborate on your reason. Figure 7. Example prompt for LLM reasoning
augmentation.

##  Appendix B Experiment Detail Setup

###  B.1. Model Checkpoints

We conduct all experiments with GPT-3.5-turbo as the LLM and GPT-4-vision as
the image caption model through OpenAI APIs (OpenAI, 2023), except for the
analysis experiment that compares different LLMs and foundation models.

Model | Checkpoints
---|---
GPT-3.5-turbo | gpt-3.5-turbo-0613
GPT-4-vision | gpt-4-vision-preview
BLIP2 | huggingface: Salesforce/blip-image-captioning-large
Kosmos2 | huggingface: microsoft/kosmos-2-patch14-224
Vision Transformer | huggingface: google/vit-base-patch16-224
Flamingo | huggingface: openflamingo/OpenFlamingo-9B-vitl-mpt7b
Longformer | huggingface: allenai/longformer-base-4096
LLAMA-2-7b-chat | huggingface: meta-llama/Llama-2-7b-chat
LLAMA-2-13b-chat | huggingface: meta-llama/Llama-2-13b-chat
LLAMA-2-70b-chat | huggingface: meta-llama/Llama-2-70b-chat
Mixtral-8x7b | huggingface:mistralai/Mixtral-8x7B-Instruct-v0.1
Table 6. Model checkpoints.

###  B.2. Hyperparameters

Model | Hyperparameters
---|---
GPT-3.5-turbo | temperature=1,max_tokens=4096
GPT-4-vision | temperature=0.8,max_tokens=300
BLIP2 | default parameter
Kosmos2 | default parameter
Vision Transformer | default parameter
Flamingo | default parameter
Longformer | max_length=2048
LLAMA-2-7b-chat | temperature=1,max_tokens=4096
LLAMA-2-13b-chat | temperature=1,max_tokens=4096
LLAMA-2-70b-chat | temperature=1,max_tokens=4096
Mixtral | temperature=1,max_tokens=4096
SDEdit | see folder in https://github.com/0Frett/LLM_Prediction_Baselines
DDRM | see folder in https://github.com/0Frett/LLM_Prediction_Baselines
Idinvert | batch_size=64,gradient_accumulate=8,network_capacity=32
Table 7. Hyper parameters.

###  B.3. Dataset

PetFinder
---
Field | Value
url | https://www.kaggle.com/competitions/petfinder-adoption-prediction
$\\#$ instances | 13453
tabular columns | 23
Airbnb
---
Field | Value
url | http://insideairbnb.com/get-the-data/
$\\#$ instances | 12184
tabular columns | 30
Avito
---
Field | Value
url | https://www.kaggle.com/competitions/avito-demand-prediction/data
$\\#$ instances | 7000
tabular columns | 18
Table 8. Dataset Meta Info

###  B.4. Foundation Models

For image modality, we utilize the embedding layer and tokenization method of
the Vision Transformer (Dosovitskiy et al., 2010). This process splits the
image into fixed-size patches and then projects each patch to obtain
embeddings. For tabular modality, we employ the FT-Transformer(Gorishniy et
al., 2021) method to encode, dividing tabular features into numeric and
categorical with separate projection layers for dimension enhancement. For
text modality, the embedding layer of Longformer(Beltagy et al., 2020) is used
for projection.

##  Appendix C Analysis and Discussion

###  C.1. In-context Modality Transfer Outperforms Zero-shot Learning Based
Methods

Text-style translation across modalities in TAMML transforms the training
modality combination into the testing modality combination to reduce the
semantic gap between them using LLMs. Similar concepts are used in zero-shot
learning baselines, which create a generative model for modality translation.
For comparison, we collected different pairs of training and testing data and
created visualizations for each one of them.

Orange is the source modality, blue is the target modality, and purple is the
source modality after transformation. Visualization results of Ours are shown
in Figure 8. Visualization results of SDEdit are shown in Figure 9. As the
results indicate, our translation effectively maps to closely align with the
target modality in semantic space.

Figure 8. Cross Modality Translation (Ours): training data map to the
distribution of target modality. Figure 9. Cross Modality Translation
(PetFinder / SDEdit): training data map to the distribution of target
modality.

###  C.2. Effects of Foundation Model

To assess the impact of different foundational model capabilities in our
architecture, we conducted ablation studies focusing on the performance of
various LLMs and foundation models.

####  C.2.1. Large Language Model

In our experiments, LLMs primarily performed the bulk of text transformations.
Therefore, the generation quality of these LLMs merits discussion. For this
purpose, we compared the performance of LLaMA_13b, LLaMA_70b, Mixtral, and our
baseline results in a simplified workflow focused solely on summarization
tasks. The outcomes are detailed in Table 9. The data demonstrates that
LLaMA_70b consistently surpasses LLaMA_13b, highlighting the influence of
model size on summarization quality. However, when juxtaposed with Mixtral and
our GPT-3.5 in various combinations, each model displayed unique strengths.
Notably, LLaMA_70b showed superior performance in generalizing human-written
texts. In contrast, Mixtral was more adept at processing texts related to
images, while GPT-3.5 excelled in dealing with structured tabular data. These
findings suggest that choosing the appropriate model based on the specific
data modality can lead to more effective outcomes.

####  C.2.2. Image Caption Model

The quality of image captions significantly influences the outcomes of text
transformation processes. In our simplified workflow, we focus on converting
text with various image caption models. For this experiment, we selected three
distinct image caption models: BLIP2 (Li et al., 2023), Kosmos2, and Flamingo.
Each of them has undergone specialized pretraining for image captioning. As
shown in Table 10, it is evident that the GPT-4 model maintains the highest
stability across all mismatch combinations with at least a 30% accuracy rate.
In contrast, other models drop to approximately 28% accuracy in tabular
modality. However, in other combinations, the performance of these models is
observed to be at similar levels. This indicates the feasibility of replacing
image caption models in such frameworks.

Training | Testing | LLaMA-13b | LLaMA-70b | Mixtral-8x7b | GPT-3.5
---|---|---|---|---|---
text+image | tabular | 0.315 | 0.316 | 0.336 | 0.321
text+tabular | image | 0.302 | 0.304 | 0.320 | 0.365
image+tabular | text | 0.300 | 0.308 | 0.295 | 0.295
text | image+tabular | 0.280 | 0.313 | 0.298 | 0.296
text | image | 0.318 | 0.320 | 0.311 | 0.298
text | tabular | 0.268 | 0.311 | 0.297 | 0.318
image | text+tabular | 0.322 | 0.300 | 0.341 | 0.289
image | text | 0.311 | 0.323 | 0.314 | 0.330
image | tabular | 0.316 | 0.332 | 0.334 | 0.307
tabular | text+image | 0.300 | 0.335 | 0.352 | 0.366
tabular | text | 0.292 | 0.308 | 0.308 | 0.306
tabular | image | 0.299 | 0.336 | 0.332 | 0.357
Average | 0.302 | 0.317 | 0.320 | 0.321
Table 9. Compare different LLMs: The results show GPT-3.5 provides the most stable performance, while other models display unique strengths in different modality combinations, substantiating the feasibility of LLM replacement. Training | Testing | Blip2 | Kosmos2 | Flamingo | GPT4
---|---|---|---|---|---
image | text+tabular | 0.313 | 0.314 | 0.291 | 0.314
| text | 0.311 | 0.303 | 0.308 | 0.306
| tabular | 0.284 | 0.281 | 0.280 | 0.300
Average | 0.303 | 0.299 | 0.293 | 0.307
Table 10. Compare different image caption models: The results show that GPT-4
provides the most stable performance, especially under tabular data, but under
other combinations, all image caption models can achieve similar performance
levels, substantiating the feasibility of image caption model replacement.

##  Appendix D Extra Experiment Results

In this section, we try to answer additional inquiries, Q4 and Q5, where Q4
explores the performance of text-based solutions versus embedding-based
solutions when training and testing modalities are identical, and Q5 compares
our zero-shot in-context learning approach to non-zero-shot methods, such as
domain adaptation.

###  D.1. Q4: Text-based Solutions Versus Embedding-based Solutions When
Training And Testing Modalities Are Identical

Table 11 provides the experiment results under no train/test modality
mismatch.

Train & Test | Regular (Embedding) | TAMML (Text)
---|---|---
text | 0.352 | 0.382
image | 0.273 | 0.369
tabular | 0.429 | 0.394
text+image | 0.286 | 0.400
text+tabular | 0.411 | 0.404
image+tabular | 0.403 | 0.408
Table 11. Experiment results under no train/test modality mismatch condition.
Under this condition, TAMML does not show performance degradation and even
performs better in several modality combinations. The regular method means the
downstream model is trained on embedding representations. Note that this
result differs from the result in Table 3 because the foundation models used
for generating embedding and text representations are not the same.

###  D.2. Q5: How does TAMML compare to non-zero-shot methods?

Table 12 provides the experiment results under modality mismatch with
different test time finetuning settings (not zero-shot). The settings are as
follows:

  * •

no finetuning: complete mismatch scenario same as main result experiments.

  * •

unsupervised domain adaptation: finetune- the downstream model given the
information of inference modality but without labels. We adopted the ADDA
(Tzeng et al., 2017) method.

  * •

supervised training (with all modalities): the downstream model given the
information of paired train/inference time modality with labels. This means
that the modality used in testing is fully trained.

Train | Test | no finetuning: Emb | no finetuning: TAMML | unsupervised domain adaptation | supervised training (all modalities)
---|---|---|---|---|---
text | image | 0.288 | 0.350 | 0.195 | 0.338
text | tabular | 0.289 | 0.342 | 0.281 | 0.359
image | text | 0.270 | 0.324 | 0.276 | 0.306
image | tabular | 0.273 | 0.364 | 0.276 | 0.359
tabular | text | 0.289 | 0.326 | 0.195 | 0.306
tabular | image | 0.279 | 0.357 | 0.195 | 0.338
Table 12. The experiment results showed a condition with no train/test
modality mismatch. Under this condition, TAMML shows no performance
degradation and even performs better in several modality combinations. The
regular method means the downstream model is trained on embedding
representations. Note that this result differs from the one in Table 3 because
the foundation models used for generating embedding and text representations
are not the same.