for omni-modal representation learning.
A) consistently enhances the performance of understanding tasks, such as classification, zero-shot classification (ZS) and linear probing (LP), across 3D point cloud([Liu et al., 2023]), depth([Girdhar et al., 2023]), audio([Girdhar et al., 2023]), tactile([Yang et al., 2022]), and EEG([Bai et al., 2023]) modalities. The citations represent the compared previous methods. Further details in <ref>.
B) By plugging into multimodal foundation models, it enables emergent applications “out-of-the-box”, including Any-modality Captioning/QA, Any-modality-to-Image Generation and text-guided Any-modality-to-Image editing, to name a few.

$^*$This work extends [Lei et al., 2023] with added modalities and applications.

Aiming to advance AI agents, large foundation models significantly improve reasoning and instruction execution, yet the current focus on vision and language neglects the potential of perceiving diverse modalities in open-world environments.
However, the success of data-driven vision and language models is costly or even infeasible to be reproduced for rare modalities.
In this paper, we present that facilitates efficient omni-modal representation learning by perceiving novel modalities with a and aligning them to a pre-defined space.
Specifically, the modality-specific lens is tuned to project any-modal signals to an intermediate embedding space, which are then processed by a strong ViT with pre-trained visual knowledge.
The encoded representations are optimized toward aligning with the modal-independent space, pre-defined by off-the-shelf foundation models.
provides a unified solution for representation learning of increasing modalities with two appealing advantages:
(i) Unlocking the great potential of to novel modalities effectively with efficient parameters and data regime;
(ii) Enabling emergent downstream capabilities through modality alignment and shared parameters.
We tailor to learn representations for 3D point cloud, depth, audio, tactile and EEG, and set new state-of-the-art results across various understanding tasks, such as zero-shot classification.
By seamlessly integrating into Multimodal Foundation Models, we enable Any-modality to Text and Image Generation in a zero-shot manner.
Code and models are available at <https://github.com/TencentARC/ViT-Lens>.

§ INTRODUCTION

Humans interact with the world through various sensory systems like vision, audition, touch, smell, and taste. To advance versatile AI agents, deep learning models need to replicate these human-like multi-sensory abilities and tackle varied user-specified tasks.
For instance, visually interpreting road signs to ensure our safe driving, listening to sirens to respond to emergency vehicles, and tactually assessing clothing fabric quality to offer shopping guidance.
Among these applications, omni-modal representation learning has become a focal point, which enables comprehensive perception in open-world environments.

On the way to pursuing omni-modal AI agents, the research community has utilized large-scale web data to make substantial strides in language [Devlin et al., 2018, Liu et al., 2019, Radford et al., 2018, Radford et al., 2019, Brown et al., 2020, OpenAI, 2021, Touvron et al., 2023] and vision [Dosovitskiy et al., 2020, He et al., 2022, Bao et al., 2021, Peng et al., 2022, Fang et al., 2023, Fang et al., 2023, Radford et al., 2021, Jia et al., 2021, Cherti et al., 2022, Yu et al., 2022].
Consequently, Multimodal Foundation Models (MFMs) [Alayrac et al., 2022, Zhu et al., 2023, Liu et al., 2023, Chen et al., 2022, Driess et al., 2023, Dai et al., 2023, Ge et al., 2023, Ge et al., 2023] that integrate vision representations with Large Language Models (LLMs) have made great progress in both vision-language comprehension and generation.

However, extending the success of unleashing LLMs to comprehend and interact with a broader array of modalities remains challenging.
Despite recent initiatives [Girdhar et al., 2023, Wang et al., 2023] in pursuing omni-modal intelligence, their capabilities on certain modalities are often constrained by limited data used in the training phase.
In contrast to image, video, and text data, which are abundant on the internet, acquiring large-scale datasets for less common modalities can be non-trivial. This scarcity of data leads to sub-optimal models with poor generalization, particularly when encountering novel categories, thereby limiting their broader real-world applications.

In this work, we present a novel perspective.
Given the exceptional generalization and transfer learning capabilities of the  [Radford et al., 2021, Fang et al., 2023, Fang et al., 2023, Caron et al., 2021, Oquab et al., 2023], there is promise in adapting their inherent knowledge to comprehend novel modalities.
This eliminates the necessity of collecting large-scale data to train models from scratch for each modality, which demands substantial time and resources.
Recognizing the rich knowledge encoded in a , we conjecture that a is able to function as a multi-modal processor – it possesses the capacity to sense and comprehend a spectrum of modalities as it interprets images.

From this standpoint,
we introduce , which encodes the out-of-image modalities through a set of parameters, with the goal of maximizing the utilization of pretrained model weights and the knowledge they encapsulate.
Specifically, employs a modality-specific Lens along with a lightweight modality embedding module to transform input data into an intermediate space. Subsequently, a frozen is applied for further encoding.
This approach enables the encoding of diverse modalities, aligning their features with the established features of anchor data, which can range from images, text to others, from off-the-shelf foundation models.

Our proposed method offers several advantages in advancing omni-modal representation learning:
(1) Parameters and data efficient approach. Our method adopts a shared set of parameters across various modalities, enabling an efficient utilization of model parameters.  Moreover, it efficiently enhances representations for less common modalities by leveraging the advanced model, reducing the demand for extensive data collection.
(2) Emergent capability. By training with the used in an off-the-shelf MFM, we can seamlessly obtain an Any-Modality MFM via integration.
The integrated model extends the original MFM's capabilities to various modalities, without any specific instruction tuning.
For instance, without direct training for tactile data, the model broadens its image generation capability to include tactile-to-image generation. As a result, it can generate an image of a sofa upon receiving tactile signals indicating “leather”.

We conducted comprehensive experiments across multiple modalities, extending beyond images and videos to encompass 3D point cloud, depth, audio, tactile, and EEG. This experiments were evaluated across 11 benchmarks.
As is shown in <ref>A, demonstrates state-of-the-art performance in 3D zero-shot classification. Particularly, when LVIS classes are excluded during training, achieves an impressive zero-shot classification accuracy of 50.1% on Objaverse-LVIS [Deitke et al., 2023], surpassing the prior SOTA by 11.0%.
It consistently outperforms ImageBind [Girdhar et al., 2023] on depth and audio benchmarks, and surpasses previous works on tactile [Yang et al., 2022] and EEG [Bai et al., 2023] related tasks.

Beyond understanding tasks, we plug into two recent MFMs, InstructBlip [Dai et al., 2023] and SEED [Ge et al., 2023, Ge et al., 2023].
As illustrated in <ref>B, this empowers the MFMs to comprehend any modality in a zero-shot manner, making Any-Captioning, Any-QA, Any-to-Image Generation and text guided Any-to-Image editing right out of the box, all without the need for specific instruction tuning.

§ RELATED WORK

Vision Language Pretraining: Advancements and Impacts.
Recent advancements in vision-language pretraining, including models such as CLIP [Radford et al., 2021], ALIGN [Jia et al., 2021], CoCa [Yu et al., 2022], Flamingo [Alayrac et al., 2022], and LiT [Zhai et al., 2022], have leveraged image-text pairs to achieve remarkable zero-shot performance on a wide range of vision and language tasks. Meanwhile, pretrained CLIP models have served as influential teachers and their joint embedding space has demonstrated efficacy in diverse zero-shot tasks such as segmentation [Li et al., 2022], detection [Gu et al., 2021, Zhou et al., 2022], 3D shape understanding [Xue et al., 2023, Xue et al., 2023, Liu et al., 2023, Zhang et al., 2022, Zhu et al., 2022], 3D open-vocabulary segmentation [Peng et al., 2023], mesh animation [Youwang et al., 2022], audio understanding [Guzhov et al., 2022] and more [Zhang et al., 2023]. extends these models' capacities to diverse modalities by integrating , enhancing its omni-modal understanding ability and enabling superior performance across various tasks and modalities.

Multimodal Learning.
Previous studies explored joint training across multiple modalities in both supervised [Girdhar et al., 2022, Likhosherstov et al., 2021, Gao et al., 2020] and self-supervised settings [Girdhar et al., 2023, Arandjelovic and Zisserman, 2017, Tian et al., 2020, Morgado et al., 2021, Lin et al., 2022].
Several approaches aim at aligning various modalities to CLIP for multimodal zero-shot learning. AudioCLIP [Guzhov et al., 2022] adds audio to CLIP for zero-shot audio classification, while ImageBind [Girdhar et al., 2023] aligns six modalities to CLIP using paired image data.
Besides, ONE-PEACE [Wang et al., 2023] introduces a unified encoder that is pretrained from scratch to align vision, language, and audio.
Zhang  [Zhang et al., 2023] pretrain a transformer with LAION-2B, following CLIP's methodology, for downstream supervised tasks across modalities.
In contrast, stands out by leveraging a to understand and unite diverse modalities without manual annotations. Its seamlessly integration with Multimodal Foundation Models (MFM) allows easy plug-and-play in emergent applications.

Multimodal Foundation Models.
Recent advancements in Large Language Models (LLMs) [OpenAI, 2021, Touvron et al., 2023] have demonstrated remarkable language understanding and reasoning abilities. Afterwards, substantial efforts [Alayrac et al., 2022, Zhu et al., 2023, Liu et al., 2023, Liu et al., 2023, Li et al., 2023] have been directed towards enabling LLMs to perceive and interact with the visual world with the help of visual representation models. Similar paradigms enable LLMs to understand more modalities by algning the well-trained encoders of various modalities to the textual space of LLMs [Su et al., 2023, Han et al., 2023, Guo et al., 2023].
Beyond understanding tasks, recent works [Sun et al., 2023, Ge et al., 2023, Ge et al., 2023] empower LLMs with the ability to generate images, and NextGPT [Wu et al., 2023] extends the generative capabilities to encompass audio and video.
Most of these Multimodal Foundation Models (MFMs) require specific instruction-following data within particular domains for training. In this study, we demonstrate that can seamlessly integrate with an MFM without additional training, extending its capabilities to various modalities.

§ METHOD

Overview. advances omni-modal representation learning by leveraging parameters to encode features for various modalities, utilizing the pre-existing knowledge in ViT.
Specifically, a modality-specific encoder, composed of a modality embedding module, the modality-specific Lens, and the , is optimized to embed robust representations through the training objective of cross-modal alignment.
For each modality, we consider its associated anchor modalities, like image and text, as reference points for learning. For alignment, we leverage foundation models, such as CLIP [Radford et al., 2021], to extract features from the anchor modalities.
Our approach leverages the extensive knowledge embedded in both the foundation models and , providing a strong basis for representation learning for each modality. This compensates for the shortage of large-scale training data available for certain modalities.
We illustrate our approach in <ref>.

Training Pipeline. extends the capabilities of a to diverse modalities. For each novel modality, it firstly employs a Modality Embedding (ModEmbed) and a Lens to learn mapping modality-specific data into an intermediate embedding space.
It subsequently employs a set of layers to encode the feature.
Finally, the output feature is aligned with the feature extracted from the anchor data (image, text, etc.) of the new modality using an off-the-shelf foundation model.

§.§ Architecture

Foundation Models for alignment.
In , the new modalities are aligned to a unified feature space established by a robust foundation model. Various options exist for this model, ranging from language models [Devlin et al., 2018, Liu et al., 2019, Radford et al., 2018, Radford et al., 2019, Touvron et al., 2023], vision models [Caron et al., 2021, Oquab et al., 2023, He et al., 2022, Fang et al., 2023] to vision-language models [Radford et al., 2021, Cherti et al., 2022, Li et al., 2022, Li et al., 2023]. During training, we fix the foundation model's parameters and utilize it to encode features for the anchor data, which serves as supervision for feature alignment.

Modality Encoder. As is shown in <ref>, the modality encoder in consists of a Modality Embedding Module (ModEmbed), a Lens and a set of layers.
Due to the distinct characteristics of various modalities, raw signals may not match the input space. This mismatch can result in suboptimal performance, despite utilizing a powerful model.
Therefore, we employ some heuristic designs:
(1) Obtain modality token embeddings: for each modality, we adopt a specific tokenization scheme to transform raw input signals into token embeddings.
(2) Map modality token embeddings to the input space: the Lens learns to map the modality embeddings into a group of latent embeddings, thereby constructing the input for the .
Subsequently, the latent embeddings are forwarded to frozen layers to obtain the final representation.

During training, the component remains frozen, and only the parameters of ModEmbed and Lens are updated. More details can be found in Supp.

Lens Architecture used in .

Lens: Connecting Modalities to . We introduce two variants of Lens to link modality token embeddings to . We show their architectures in <ref>.

* Self-attention blocks (S-Attn). This variant involves a stack of self-attention layers [Vaswani et al., 2017] that transforms the input token embeddings into intermediate embeddings with equal indices. We can potentially enhance this variant's capability by initializing it with pretrained weights from existing layers. It suits modalities structured with image-like inputs, such as depth maps.

* Iterative cross-self-attention blocks (Iter-CS-Attn). This variant's basis block involves a cross-attention module coupled with a self-attention tower, inspired by [Jaegle et al., 2021]. It maps a latent array and input embeddings to a latent embedding of matching length within the input latent array. This manner condenses inputs of varied sized into a latent bottleneck,
making it apt for lengthy input modalities like 3D point clouds.
Similar architectures are employed in Vision-Language Models (VLMs) [Alayrac et al., 2022, Li et al., 2023] to extract visual information for Large Language Models (LLMs). Our innovation lies in utilizing this structure to map signals from diverse modalities into the 's input space, enabling the to understand modalities beyond images.

§.§ Training Objective

In this work, we study modalities of 3D, depth, audio, tactile and EEG, among which all the data samples are associated with text descriptions, image appearances, or both.
We use the pretrained CLIP [Radford et al., 2021, Cherti et al., 2022] as the foundation model. By default, we employ pretrained layers of in the foundation CLIP as part of the modality encoder. Following the approach in previous works [Xue et al., 2023, Xue et al., 2023, Liu et al., 2023, Guzhov et al., 2022], we adopt multimodal contrastive learning for representation alignment.

We denote $X = \left\{x_1, \ldots, x_N \right\}$ as the collection of modality data to be learned, $\mathcal{A} = \left\{A_1, \ldots, A_M\right\}$ as the set of anchor modalities, $a_n^m$ as the anchor data of $x_n$ from modality $A_m$, $\ve{G}_A$ as the foundation model for anchor modality $A$, and $\ve{F}$ as the modality encoder to be learned. The contrastive loss for alignment is formulated as:

\begin{align*}
\mathcal{L} = -\frac{1}{2B|\mathcal{A}|}\sum_{i=1}^B \sum_{k=1}^{|\mathcal{A}|}
\left(
\log \frac{\exp(h_i^X \cdot h_i^{A_k} / \tau)}{\sum_j \exp (h_i^X \cdot h_j^{A_k} / \tau)} \right. \\ \left.
+ \log \frac{\exp( h_i^{A_k} \cdot h_i^X  / \tau)}{\sum_j \exp (h_i^{A_k} \cdot h_j^{X} / \tau)}
\right),
\end{align*}

where $B$ is the batch size; $\tau$ is a learnable temperature; $h_i^X = {\rm Norm}\left(\ve{F}(x_i)\right)$, $h_i^{A_k}={\rm Norm}\left(\ve{G}_{A_k}(a_i^k)\right)$ are normalized features of data $x_i$ and its anchor data $a_i^k$ from $A_k$.

§.§ Free Lunch for Multimodal Foundation Models

Demonstration of integrating to MFM. (A) Original overall pipeline of MFM for vision; (B) Illustration of plugging well-trained Lenses of different modalities to MFM, without additional instruction-following training.

Plug into MFM.
Recent MFMs for vision [Zhu et al., 2023, Dai et al., 2023, Liu et al., 2023, Liu et al., 2023, Ge et al., 2023, Ge et al., 2023] enables LLMs to understand visual content. As shown in <ref>A, this process begins with the use of a frozen to extract visual features. Subsequently, a well-trained abstractor module processes these features, constructing inputs that can be understood by the LLM.

By incorporating the from MFM as part of the modality encoder and as the foundation model in training, the yielded modality Lenses can be seamlessly integrated into the MFM for plug-and-play application, as depicted in <ref>B.
In later experiments, we showcase the emergent abilities facilitated by this tuning-free adaptation.

§ EXPERIMENTS

§.§ Experimental Setup

For this part, we describe the main experimental setup and provide full details in Supp.

Pretraining Datasets.
Beyond image/video and text, we train on a variety of modalities, including 3D point cloud, depth, audio, tactile, and EEG data. These datasets are anchored to text descriptions, images, or both for feature alignment.

For 3D point cloud experiments, we utilize a combination of ShapeNet [Chang et al., 2015], 3D-FUTURE [Fu et al., 2021], ABO [Collins et al., 2022], and Objaverse [Deitke et al., 2023].  We incorporate rendered images and text captions from previous works, resulting in three pretraining datasets: ULIP-ShapeNet [Xue et al., 2023], ULIP-2-Objaverse [Xue et al., 2023], and OpenShape-Triplets [Liu et al., 2023].
Depth data is sourced from the SUN RGB-D dataset [Song et al., 2015], utilizing paired image and scene labels for alignment.
Audio data is obtained from the Audioset dataset [Gemmeke et al., 2017], accompanied by associated video and text label metadata.
Tactile data is sourced from the Touch-and-go dataset [Yang et al., 2022], featuring paired frame and material label text.
Finally, EEG data from [Spampinato et al., 2017] is aligned with paired ImageNet image and text labels.

Evaluation on Downstream Understanding Tasks.
We  evaluate across diverse modalities and protocols via a comprehensive set of downstream tasks. The primary datasets used for evaluation are summarized in <ref>.

Main Implementation Details.
We use the pretrained vision and text encoders from OpenCLIP [Cherti et al., 2022]. We apply different model sizes: based on -B/16, based on -L/14, and based on -bigG/14.

For 3D point cloud data, we follow the baseline methods [Xue et al., 2023, Liu et al., 2023] to uniformly sample 8,192 or 10,000 points and grouping them into sub-clouds through Farthest Point Sampling (FPS) followed by KNN grouping of neighboring points.
For depth input, we follow [Girdhar et al., 2023] to use in-filled depth values and convert them to disparity for scale normalization.

For audio data, we sample 5-second clips and extract a single frame randomly from the video clip if video serves as anchor data. The audio waveform is converted into a sequence of 128-dimensional log Mel filterbank features using a 25ms Hamming window every 10ms, following [Gong et al., 2021].

For tactile input, we use RGB data collected from GelSight [Johnson and Adelson, 2009].

For EEG signals, we employ 128-channel temporal sequences and use the frequency range of 5-95Hz following [Bai et al., 2023].

§.§ Results on Understanding Tasks

Zero-shot 3D Classification.
We follow [Xue et al., 2023, Xue et al., 2023, Liu et al., 2023] to use (point cloud, image, text) triplets to train . We conduct zero-shot classification on downstream benchmarks. The overall results can be found in <ref>.
In particular, when pretrained on ULIP-ShapeNet or ULIP2-Objaverse, outperforms ULIP with different 3D encoders [Qi et al., 2017, Ma et al., 2022, Qian et al., 2022, Yu et al., 2022], as is shown in <ref>.

We present the results of training on OpenShape-Triplets in <ref>. To align with [Liu et al., 2023], we adopt and train on both “NO LVIS” (excluding all shapes from the Objaverse-LVIS subset) and the entire set for comparison. outperforms models adopted in OpenShape [Liu et al., 2023].
Notably, significantly improves the accuracy on the long-tail categories of Objaverse-LVIS, from 46.8% to 52.0%. Additionally, when trained on the NO LVIS subset, achieves a top-1 accuracy of 50.1%. This performance beats ULIP by roughly 30% and surpasses OpenShape-PointBERT trained on the entire set by 3.3%, demonstrating the data-efficient merit of .
Regarding ModelNet40, achieves an 87.4% accuracy, surpassing previous SOTA. Moreover, on ScannetObjectNN, containing challenging real scans with noise and occlusion, our method exhibits decent sim-to-real transfer ability. It achieves a 60.1% zero-shot accuracy without specific sim-to-real training, surpassing the previous SOTA.

Audio Classification and Retrieval.
In our comparison presented in <ref>, consistently outperforms prior approaches in both audio classification and text-to-audio retrieval tasks.  When aligned to images (I), outperforms ImageBind based on Huge CLIP [Cherti et al., 2022], and AVFIC [Nagrani et al., 2022], which leverages automatically mined audio-text pairs for alignment.
When aligned to images and texts (I+T), demonstrates stronger performance and significantly outperforms AudioCLIP [Guzhov et al., 2022]. Although AudioCLIP uses a audio encoder pretrained with Audioset supervised classification, it falls behind .
Additionally, on zero-shot VGGSound classification, surpasses the SOTA [Wu* et al., 2023] when class names are used as text supervision for alignment.

Audio and Video Retrieval.
We use the MSR-VTT [Xu et al., 2016] benchmark to evaluate the text to audio and video retrieval performance, as presented in <ref>. We follow [Girdhar et al., 2023] to combine audio (A) and video (V) modalities. outperforms several prior methods, even surpassing those that incorporate video data for training [Miech et al., 2019, Patrick et al., 2021, Nagrani et al., 2022].

Depth-only Scene Classification.
In <ref>, we present our results for depth-only classifications. outperforms ImageBind across SUN-D and NYU-D. By using image and text as anchor data, further improves the performance and narrows the gap with the supervised SOTA model [Girdhar et al., 2022] with extra training data.

Tactile Classification Tasks.
Results for tactile tasks are displayed in <ref>. Across various tactile classification tasks like material, hard/soft, and rough/smooth classification, demonstrates superior performance compared to our implementation of ImageBind-B.
Even trained with appearance or text labels for material, can perform well on the hard/soft and rough/smooth classification tasks. This underscores the extensive knowledge transfer by CLIP during training.
Furthermore, scaling up to a larger model and incorporating text during training can further boost the performance.
In comparing the image-aligned with CMC [Yang et al., 2022] using linear probing, we observe significantly superior performance by .

EEG Visual Concept Classification.
Results in <ref> show that consistently outperforms our implemented ImageBind-B. Additionally, when compared to the EEG encoder from [Bai et al., 2023], which used more EEG data for MAE-style pretraining [He et al., 2022] and then aligned with the CLIP-L14 image encoder, demonstrates superior performance.

§.§ Few-shot Linear Probing

We evaluate the label-efficiency of through few-shot linear probing using (I) depth encoder and 3D encoder, as shown in <ref>. Linear classifiers are trained on fixed representation features.
For few-shot depth linear probing, we compare with ImageBind [Girdhar et al., 2023] and MultiMAE [Bachmann et al., 2022] trained on images, depth, and semantic segmentation data. consistently outperforms both methods in zero-shot and few-shot settings.
For few-shot 3D classification, we compare with SparseConv and PointBERT trained in [Liu et al., 2023]. significantly outperforms all methods by a large margin in all few-shot settings, showcasing its robust generalization capabilities.

§.§ Results on MFMs

In this section, we plug across various modalities into off-the-shelf MFMs, and show in our experimental results that the MFMs' capabilities can be transferred to novel modalities and their combinations, without instruction-following training.

MFM Selection in Practice.
In this work, we select InstructBLIP [Dai et al., 2023] and SEED [Ge et al., 2023, Ge et al., 2023] to probe the emergent capabilities of the MFMs with our plugged in. Both InstructBLIP and SEED utilize EVA01-g14  [Fang et al., 2023] as the visual encoder. Following the practice in <ref>, we use the same for training in MFMs experiments. More details can be found in Supp.

InstructBLIP with .
InstructBLIP [Dai et al., 2023] introduced a framework for instruction tuning in a vision-language model, demonstrating its capabilities in tasks like complex visual reasoning and image descriptions.  We show in our experiment that these capabilities can be effectively extended to novel modalities through the integration of .
Qualitative examples in <ref> (A-B) showcase the model's ability to follow instructions across various modalities, enabling Any-modality QA, captioning, etc.
Additionally, the model demonstrates precise and detailed descriptions, such as identifying a small “chair” next to a giant piano in (A), emphasizing the superior alignment achieved by .

SEED with .
SEED-LLaMa [Ge et al., 2023] is an MFM distinguished by its capacity for multimodal comprehension and image generation. This is achieved through multimodal pretraining and instruction tuning along with its SEED tokenizer [Ge et al., 2023].
We present qualitative results of integrating with SEED in <ref> (E-G). The outcomes illustrate how the combined model extends SEED's capabilities to diverse modalities.
Examples in (E-G) show the ability of compositional any-to-image generation [Ge et al., 2023].
It can translate input from any modality into an image, generate an image based on a text prompt given input from any modality, and seamlessly blend visual concepts from combinations of any modalities into a coherent and plausible image.

§.§ Ablation Study

We conduct ablations studies to investigate the effectiveness of various designs for in omni-modal learning. We report the main results here and full details are in Supp.

Lens designs for different modalities.
We study the effect of Lens designs as outlined in <ref> for different modalities. We use and set comparable amount of trainable parameters for the two variants. We also examine the effectiveness of initializing S-Attn type Lens with pretrained weights.
We train 3D point cloud on ULIP-ShapeNet and follow the main settings for other modalities.
The results are shown in <ref>.
We observe consistent performance enhancement by initializing S-Attn Lens with pretrained weights.
For image-like inputs such as depth maps and RGB-based tactile data, the S-Attn design exhibits superiority.
Conversely, modalities significantly different from image inputs, like 3D point clouds, audio spectrograms, and EEG, benefit more from Iter-CS-Attn design. Additionally, it reduces the computational overhead by reducing the input length for . Further details are available in the Supp.

Modality encoder designs and settings.  We investigate the efficacy of integrating a set of layers into the modality encoder. We use the same datasets for training and testing as in the Lens design ablation.
We compare with an architecture that combines ModEmbed and and employing different settings for the component, as detailed in <ref>.
Results indicate that simply adding the ModEmbed to a cannot fully exploit the the potential of the (#2). Training the entire encoder with pretrained weights outperforms training from scratch, highlighting the effectiveness of utilizing the pretrained weights for learning (#1 vs #3).
In comparison to #3, achieves comparable or better performance, especially for the less common modalities.
Moreover, our employs fewer trainable parameters than training the entire encoder and reduces computational overhead for modalities with lengthy inputs. Consequently, by introducing Lens, effectively and efficiently transfers the capabilities of to various modalities.

Scaling up foundation model and . We explore the effectiveness of scaling up for feature alignment. We conduct experiments to pretrain for 3D on ULIP-ShapeNet, and depth on SUN-D.
While previous works [Girdhar et al., 2023, Liu et al., 2023] show that scaling to a large encoder($>$100M) degrades the performance, we show in <ref> that scaling up can improve the 3D and depth representation and enhance performance.

Different pretrained-ViTs for . We evaluate different variants for omni-modal representation learning. We use CLIP-ViT-bigG/14 as the teacher foundation model and apply different ViTs for the modalitiy encoder. We use the same datasets for training and testing as in the model scaling ablation. Results in <ref> demonstrate that the use of including the self-supervised and CLIP pretrained variants, outperforms training from scratch on both depth and 3D modalities. This indicates that different possess the potential to serve as effective omni-modal learners.

§ CONCLUSION

In this paper, we introduce , a straightforward yet effective method to advance omni-modal representations.
employs a to encode features for diverse modalities, eliminating the need for separate modality-specific architectures.
The rich knowledge from large-scale data within the also reduces the burden of extensive data collection.
We train for various modalities, including 3D point cloud, depth, audio, tactile, and EEG. Experimental results in understanding tasks demonstrate that consistently achieves leading performance.
Moreover, we integrate to off-the-shelf MFMs, i.e., InstructBLIP and SEED, unlocking emergent capabilities such as any-modality instruction following, any-modality-to-image generation and text-guided any-modality-to-image editing.
Finally, we believe that will stimulate further research and innovation in the field of omni-modal representation learning, paving the way for more versatile and robust AI systems.

§ APPENDIX

§ MORE DETAILS OF METHOD

§.§ ModEmbed for Modality Encoder

As describe in <ref>, we adopt a specific tokenization scheme to transform raw input signals into token embeddings for each modality. In this section, we introduce the modality embedding modules for 3D point cloud, depth, audio, tactile and EEG in our work.
3D point cloud. For 3D point cloud embedding, we utilize the approach introduced in [Yu et al., 2022]. We initially sample $g$ center points from the input point cloud $p$ using farthest point sampling (FPS). Subsequently, we utilize the k-nearest neighbors (kNN) algorithm to select $k$ nearest neighbor points for each center point, forming $g$ local patches $\left\{ p_i \right\}_{i=1}^{g}$. To extract the structural patterns and spatial coordinates of these local patches, we normalize them by subtracting their center coordinates. Further, we employ a mini-PointNet [Qi et al., 2017] to project these sub-clouds into point embeddings. Additionally, we incorporate learnable positional embeddings on top of these embeddings, serving as inputs to standard Transformers or Lens models.

Audio. For audio embedding, following [Gong et al., 2021], we firstly convert the input audio waveform into a sequence of log Mel filterbank (fbank) features, forming a spectrogram with time and frequency dimensions. This spectrogram is then partitioned into a sequence of $P \times P$ patches with a stride of $S$ in both time and frequency dimensions. Each $P \times P$ patch is flattened and projected into a 1D embedding of size $d$ using a linear projection layer. Subsequently, we introduce learnable positional embeddings to capture the spatial structure of the spectrogram. These embeddings are utilized as inputs for subsequent processing by the model.

Depth. For depth embedding, we firstly follow [Girdhar et al., 2022, Girdhar et al., 2023] to convert depth maps into disparity for scale normalization. We then utilize patch embedding similar to the mechanism in . This involves partitioning the disparity into $P \times P$ patches with a stride of $S$ ($S = P$) to handle the single-channel input. Each $P \times P$ patch undergoes flattening and projection into a 1D embedding of size $d$ using a linear projection layer. To capture positional information, we incorporate learnable positional embeddings. These embeddings serve as inputs for the subsequent module.

Tactile. For tactile embedding, since we use RGB data from GelSight [Johnson and Adelson, 2009], we apply the same patch embedding as in . Specifically, we partition the RGB input into $P \times P$ patches with a stride of $S$ ($S = P$). Each $P \times P$ patch undergoes flattening and projection into a 1D embedding of size $d$ using a linear projection layer.
We integrate learnable positional embeddings for position information. These embeddings are forwarded as inputs for the subsequent module.

EEG. For EEG embedding, we use the $C$ channel EEG with $T$ timestamps. We then group every $t$ time steps into a token and transformed it into a $d$-dimensional embedding. We further add positional embeddings on top and use the yielded embeddings as inputs for the subsequent module.

§.§ More Details for Lens

Reducing computational comprexity with Iter-CS-Attn.
As is shown in <ref> in the main paper,
the cross attention mechanism generates an output with equal length of the latent query input.
In practice, we typically configure the Lens with less parameters (fewer attention layers) compared to the component. Consequently, the majority of computational overhead is incurred during the forward pass of the layers.
Consider the input latent query length $\ve{n}$ and the modality embedding length $\ve{m}$. For modalities with lengthy input ($\ve{m}>\ve{n}$), utilizing the Iter-CS-Attn Lens reduces the computational cost of to $\mathcal{O}(\ve{n}^2)$, compared to encoding embeddings of the same length as the input, which has a complexity of $\mathcal{O}(\ve{m}^2)$. This strategy significantly lowers the computational overhead for processing lengthy inputs.

§.§ Utilizing Pretrained ViT Layers

The core of enhancing omni-modal representation with is to leverage the rich knowledge encoded in the that is pretrained on large-scale data.
To integrate into the modality encoder, we apply the last $l$ out of the total $L$ transformer layers while maintaining a relatively high ratio $\frac{l}{L}$.
This strategy draws inspiration from recent research exploring interpretation [Raghu et al., 2021, Ghiasi et al., 2022]. These studies revealed that captures higher-level semantic concepts in its deeper layers while encoding general edges and textures in the shallower ones.
Building upon these insights, we posit that the shared high-level knowledge among different modalities is mostly preserved in the deeper layers of the architecture. Consequently, we propose the utilization of a set of layers within the modality encoder in our pipeline.
Notably, when $\frac{l}{L} < 1$, we either discard the initial $L - l$ transformer layers or integrate them for S-Attn type Lens learning  if applicable.

§ MORE EXPERIMENTAL DETAILS AND RESULTS

§.§ Datasets and Metrics

ULIP-ShapeNet Triplets [Xue et al., 2023].
The ULIP-ShapeNet Triplets training data for 3D point cloud is derived from ShapeNet55 [Chang et al., 2015] by Xue  [Xue et al., 2023]. All the 3D point clouds are generated from CAD models. Anchor images are synthesized using virtual cameras positioned around each object, and texts are obtained by filling metadata into a predefined prompt template. This dataset comprises approximately 52.5k 3D point cloud instances.

ULIP2-Objaberse Triplets [Xue et al., 2023].
The ULIP2-Objaverse Triplets training data for 3D point cloud is developed by Xue  [Xue et al., 2023], utilizing the recently released Objaverse [Deitke et al., 2023]. For each 3D object, 12 rendered images are obtained, spaced equally by 360/12 degrees. Each rendered image has 10 detailed captions generated using BLIP2-opt6.7B [Li et al., 2023]. It includes around 798.8k 3D point cloud instances.

OpenShape Triplets [Liu et al., 2023].
The OpenShape Triplets training data for 3D point clouds encompasses four prominent public 3D datasets: ShapeNet [Chang et al., 2015], 3D-FUTURE [Fu et al., 2021], ABO [Collins et al., 2022] and Objaverse [Deitke et al., 2023]. For each 3D object, 12 color images are rendered from preset camera poses, and thumbnail images are included as candidates if provided. OpenShape employs various strategies to obtain high-quality text descriptions, including filtering noisy metadata using GPT4 [OpenAI, 2023], generating captions using BLIP [Li et al., 2022] and Azure cognition services, and conducting image retrieval on LAION-5B to retrieve relevant texts with paired images closely resembling the object's rendered image, leading to a wider range of text styles. This dataset comprises approximately 876k 3D point cloud instances.

ModelNet40 [Wu et al., 2015].
The ModelNet40 dataset is a widely used benchmark in the field of 3D object recognition. It consists of 12,311 CAD models from 40 categories, with 9,843 training samples and 2,468 testing samples. It includes everyday objects such as chairs, tables, desks, and other household items. Each object is represented as a 3D point cloud and has been manually annotated with the object's category. The dataset is commonly used for tasks like shape classification and shape retrieval. In this work, we only use the test samples for zero-shot classification. The evaluation is performed using Top-K accuracy.

ScanObjectNN [Uy et al., 2019].
The ScanObjectNN dataset is a significant resource in the domain of 3D object recognition and segmentation. It encompasses a diverse array of 3D object instances acquired through a commodity RGB-D camera. This dataset exhibits a wide spectrum of household items, furniture, and common indoor objects. Each individual object instance is annotated with fine-grained semantic and instance-level labels. In total, it contains 2,902 objects distributed across 15 distinct categories.
In this work, we follow [Liu et al., 2023] to use the variant provided by [Yu et al., 2022] for zero-shot classification, which contains 581 test shapes with 15 categories. The evaluation is performed using Top-K accuracy.

Objaverse-LVIS [Deitke et al., 2023]. This dataset is an annotated subset of Objaverse [Deitke et al., 2023] and consists of 46,832 shapes from 1,156 LVIS [Gupta et al., 2019] categories. With a larger base of classes compared to other benchmarks, Objaverse-LVIS presents a challenging long-tailed distribution, making it a better reflection of the model's performance in open-world scenarios. In this work, we follow [Liu et al., 2023] to use this dataset for zero-shot classification, and the evaluation is performed using Top-K accuracy.

SUN-RGBD [Song et al., 2015]. We utilize paired RGB and depth maps along with associated class labels from the SUN-RGBD dataset. For training , we employ the train set comprising approximately 5k samples.
To evaluate classification performance, we use the test set (SUN Depth-only), which contains 4,660 samples. Specifically for testing, we only utilize depth as input and construct classification templates using the 19 scene classes available in the dataset. The evaluation process involves Top-K accuracy metrics.

NYU-Depth v2 [Nathan Silberman and Fergus, 2012].
We utilize the depth maps from NYU-Depth v2 test set (NYU-v2 Depth-only) containing 654 samples for evaluation. We use 16 semantic classes in the dataset and follow previous work [Girdhar et al., 2023] to conduct 10-class classification. Concretely, for classification, there is an “others” class corresponding to 7 different semantic classes – [`computer room', `study', `playroom', `office kitchen', `reception room', `lobby', `study space']. For classification, we compute the similarity of the “others” class as the maximum cosine similarity among these 7 class names. We report Top-K accuracy.

Audioset [Gemmeke et al., 2017].
This dataset is utilized for both training and evaluation in our work. It contains 10-second videos sourced from YouTube and is annotated across 527 classes. It consists of 3 pre-defined splits – unbalanced-train split with about 2M videos, balanced-train with about 20k videos and test split with about 18k videos.
Due to the unavailability of some videos for download, we finally have 1.69M/18.7k/17.1k for these three splits. We use the train splits for training and the test split for evaluation. During evaluation and when textual data serves as anchor data during training, we make use of the textual class names along with templates. The evaluation metric employed is mean Average Precision (mAP).

ESC 5-folds [Gemmeke et al., 2017].
The ESC50 dataset is a widely used benchmark dataset in the field of environmental sound classification. It comprises a collection of 2,000 sound recordings, categorically organized into 50 classes, including animal vocalizations, natural soundscapes, and human-made sounds. Each class in the dataset contains 40 audio samples that are five seconds long. It has pre-defined 5 fold evaluation, each consisting of 400 test audio clips. In this work, we evaluate the zero-shot prediction on across the 5 folds and report the overall Top-1 accuracy.

Clotho [Drossos et al., 2020].
The Clotho dataset is an audio collection paired with rich textual descriptions, comprising a development set of 2,893 audio clips and a test set of 1,045 audio clips. Each audio clip is associated with five descriptions. In this study, we focus on the text-to-audio retrieval task. For evaluation, we treat each of the five associated captions as an individual test query, searching within the set of audio clips. We employ recall@K as the evaluation metric, where a query is considered successful if the ground truth audio is retrieved among the top-K returned audio clips.

AudipCaps [Kim et al., 2019].
This dataset comprises audio-visual clips sourced from YouTube, accompanied by textual descriptions. It features clips extracted from the Audioset dataset. In this study, we employed the dataset splits outlined in [Oncescu et al., 2021], specifically excluding clips that intersected with the VGGSound dataset.   We end up with 813 clips in the test split for zero-shot evaluation. The task is text-to-audio retrieval and is evaluated by the recall@K metric.

VGGSound [Chen et al., 2020].
This is an audio-visual dataset sourced from YouYube. It contains more around 200k video clips of 10s long. These clips are annotated into 309 classes, covering a spectrum from human actions to sound-emitting objects and human-object interactions.
Since some videos are no long available for downloading, we finally end up with 162k clips for train set and 15.5k for test set. In this work, the audio from the test set is utilized specifically for zero-shot classification tasks, evaluating model performance using the Top-1 accuracy metric.

Touch-and-go [Yang et al., 2022].
The Touch-and-Go dataset comprises real-world visual and tactile data gathered by human data collectors probing objects in natural settings using tactile sensors while simultaneously recording egocentric video. It offers annotations for 20 material classes, and provide hard/soft (H/S) and rougH/Smooth (R/S) labels.
The dataset is organized into distinct splits: train-material and train-H/S with 92k samples, test-material and test-H/S with 30k samples, train-R/S with 35k samples and test-R/S with 8k samples.
In our work, we utilize the train-material split for training and perform classification on the test-material subset. For zero-shot classification, we employ test-H/S and test-R/S subsets. In the context of linear probing, we fine-tune the model using the corresponding train set for a particular task. Our evaluation of model performance utilizes the Top-1 accuracy metric.

ImageNet-EEG [Spampinato et al., 2017].
This dataset comprises EEG recordings obtained from six subjects while they were presented with 2,000 images across 40 categories from the ImageNet dataset [18]. Each category contains 50 distinct images, resulting in a total of 12,000 128-channel EEG sequences.
Recorded using a 128-channel Brainvision EEG system, the dataset covers diverse object categories, including animals (such as dogs, cats, elephants), vehicles (including airliners, bikes, cars), and everyday objects (such as computers, chairs, mugs).
We leverage the observed image and/or its corresponding text label as anchor data. We conduct classification tasks on both the validation set (consisting of 1,998 samples) and the test set (consisting of 1,997 samples).
Our evaluation of model performance is based on the Top-1 accuracy metric.

§.§ Data Input and Augmentation

Image and Video.
When handling modalities such as images, videos, or tactile sensor data with RGB or RGBT inputs, we adopt the standard input representation used in the vanilla model. Specifically, for image input, we partition it into patches of size $P \times P$. For video input, we employ 2-frame clips similar to the approach outlined in [Girdhar et al., 2023]. We construct patches of size $T \times P \times P$. Notably, $T = 2$, $P=16$ for , and $P=14$ for and . We inflate the visual encoder's weights to to accommodate spatiotemporal patches for video inputs. During inference, we aggregate features over multiple 2-frame clips. This adaptation enables models initially trained on image-text data to effectively handle videos.

3D point cloud. For 3D point cloud input, we follow previous work to uniformly sample 8,192 points [Xue et al., 2023, Xue et al., 2023] or 10,000 points [Liu et al., 2023] as the input for 3D shape.
During training, we apply standard augmentation [Xue et al., 2023] for the point clouds. As mentioned in <ref>, we construct local patches by sampling 512 sub-clouds, each comprising 32 points. This is accomplished by employing Farthest Point Sampling (FPS) and the k-Nearest Neighbors (kNN) algorithm.

For the single-view depth, we follow [Girdhar et al., 2023] to use the in-filled depth and convert them into disparity. During training, when image is used as anchor data, we apply strong data augmentation for the anchor image, including RandAug [Cubuk et al., 2020] and RandErase [Zhong et al., 2020]. We used aligned spatial crop for image and depth. For embedding module, we follow the to set $P=16$ for and $P=14$ for .

Audio. For audio input, we process each raw audio waveform by sampling it at 16kHZ, followed by extracting a log mel spectrogram with 128 frequency bins using a 25ms Hamming window with the a hop length of 10ms. Consequently, for an audio duration of $t$ seconds, our input dimensionality becomes $128 \times 100t$.
During training, we randomly sample a 5-second clip for audio input, and apply spectrogram masking [Park et al., 2019] with max time mask length of 48 frames and max frequency mask length of 12 bins.
When image is used as anchor data, we randomly sample 1 frame from the corresponding clip and apply RandAug [Cubuk et al., 2020] for the sampled frame. We also apply Mixup [Zhang et al., 2017] during training for both audio and its anchor data, with a mixup ratio of 0.5.
For embedding module, we set $P=16$ for and $P=14$ for , and $S = 10$.
At inference time, we uniformly sample multiple clips to cover the full length of the input sample and aggregate the features extracted from these clips.

For data from tactile sensors, we treat it similarly to RGB images. During training, we introduce random flips along the horizontal and vertical directions to augment the tactile input. Additionally, random rotations are applied to further augment the input data.
When image is used as anchor data for training, we apply RandAug [Cubuk et al., 2020] to augment the image.
For embedding module, we follow the to set $P=16$ for and $P=14$ for .

EEG. For EEG input data, we follow [Bai et al., 2023] to use the 128-channel EEG sequences. These EEG signals are filtered within the frequency range of 5-95Hz and truncated into a common length of 512.
For embedding module,we utilize Conv1D, configuring the kernel size to 1 and the stride to 1.

§.§ Model Configuration

In this section, we provide the configurations for encoders of different modalities in .
Details are specified in <ref>.

§.§ Training Setup

In <ref>, we list the hyper-parameters used in training for each modality. Our experiments were done on 32GB V100 GPU clusters.

§.§ Additional Results on Understanding Tasks

Additional results on audio tasks.
We merged Audioset [Gemmeke et al., 2017] and VGGSound [Chen et al., 2020], resulting in a combined training dataset with 1.86M samples. Results in <ref> show improved performance across all benchmarks due to the inclusion of additional training data.

§.§ More Details and Results for MFMs

Architectural details for MFM integration.
Both InstructBLIP [Dai et al., 2023] and SEED [Ge et al., 2023, Ge et al., 2023] apply the pretrained EVA01-g14 [Fang et al., 2023] to perceive and encode images for the subsequent LLM input. Concretely, they use the first 39 transformer layers of the 40-layer for visual feature extraction.
Adhering to this configuration, we employ the EVA01-g14 CLIP as the foundation model and utilize its as an integral part of the modality encoder for the training of multimodal alignment. We tune the parameters of ModEmbed and Lens.
During inference, we directly plug the ModEmbed and Lens prior to the , enabling the yielded MFM to handle inputs of various modality without specific instruction following.

§.§.§ Additional Results: InstructBLIP with

Comparison of InstructBLIP with against other methods on 3D data instruction following.
We train for 3D point cloud using ULIP2-Objaverse and integrate it into InstructBLIP. Beyond capturing the high-level semantics of the input data, we observed that leveraging the EVA01-g14 within the modality encoder further enhanced the model's ability to capture local details.

Our qualitative evaluation involves a comparison with: (1) PointBERT [Yu et al., 2022] aligned with EVA01-g14 CLIP, replacing the vision encoder used in InstructBLIP; and (2) CLIPCap [Mokady et al., 2021] from OpenShape [Liu et al., 2023].
We present a snapshot of qualitative outcomes across different models in <ref>,  <ref> and  <ref>.
These examples showcase several capabilities exhibited by integration without specific tuning using 3D-related instructional data. Notably, the examples demonstrate that empowers InstructBLIP to accurately describe 3D objects. For instance, the plant example in  <ref> is characterized as “sitting in a ceramic pot” and “bamboo-like”. Moreover, excels in capturing local visual concepts beyond the most prominent ones. For instance, the piano example in <ref> describes the observation of a “chair”.

For PointBERT integrated InstructBLIP, although PointBERT achieves decent performance for zero-shot classification, it fails to provide accurate information for the InstructBLIP as does. We can see that in <ref>, although it recognizes the piano, it fails to provide accurate brief and detailed description since it includes “person” in its description, which does not exist in the 3D input. Also, it fails to recognize the plant in <ref> and the toilet in <ref>.

CLIPCap-OpenShape, while occasionally displaying some relevant entities in captions (“vase” in  <ref> and “toilet” in  <ref>), often generates hallucinations and inaccurate captions.

The overall results demonstrate that excels not only at classifying the salient object of the 3D input, but also capturing the visual details.
This merit is surprising: despite the fact that we only explicitly use the  for alignment, the integrated model exhibits the ability to capturing local information.
This ability might stem from potentially inheriting information captured by other tokens, which could carry local details to the input of InstructBLIP. This capability indicates that the model might leverage the collective knowledge present in various tokens, not limited to the , contributing to its robustness in encoding rich visual information.

InstructBLIP with for input of multiple modalities.
We demonstrate that the versatile omni-modal encoder, coupled with an array of specialized Lenses, functions as a sensor adept at concurrently perceiving and understanding multiple modalities.
To achieve this, we concatenate the outputs from diverse modality Lenses prior to inputting them into the transformer. Subsequently, the encoded embeddings from this concatenation are forwarded to the LLM within InstructBLIP for text generation.

Qualitative results[Photos credited to <https://www.pexels.com/>.] are showcased in  <ref> for dual-modality input and in  <ref> for tri-modal input. The outputs produced by InstructBLIP with underscore its remarkable ability to concurrently interpret multiple modalities, akin to perceiving an image.
Notably, as evident in the qualitative results, the incorporation of enhances the resulting MFM's capacity to digest multi-modal inputs, discover unconventional co-occurrence of concepts from different modalities, and craft stories based on the aggregated information from multiple modalities without specific instruction tuning.

§.§.§ Additional Results: SEED with

Integrating the well-trained modality Lenses into SEED enables seamless any-modality to image generation. In addition to the primary paper, we present further qualitative results in <ref>.

In (S-A), we showcase the capability of generating images from any modality input by encoding the modality information using Lens and the SEED tokenizer [Ge et al., 2023]. We append to this with a text prompt, “” The results demonstrate that our integrated MFM successfully generates images across diverse modalities, including 3D point clouds, audio, EEG, tactile, and depth. Notably, in the 3D point cloud examples (Row 1 in S-A), the model retains the local structure of the 3D shapes.
Furthermore, the model exhibits the ability to generate diverse images for different inputs within the same category, exemplified by the audio examples for “sea waves”, “engine”, “crackling fire” and more. This showcases the robustness of our .

In (S-B), we show that our integration extends SEED's capability for compositional image generation to any modality. For better visualization, we show examples for 3D point clouds. We show the main instruction under each example.
In practice, we feed any modality input into the LLM via Lens and the SEED tokenizer, supplementing it with the prompt “” to guide text-based generation.
The presented results highlight the model's ability to retain visual concepts accurately in the generated images. Additionally, it preserves the local structure rather than merely focusing on high-level semantics. Notably, in examples such as “guitar” and “car” the model successfully retains the shape and some local structures, showcasing its nuanced understanding beyond high-level semantic understanding.

Moreover, the model demonstrates the capability to intake inputs from various modalities and subsequently generate an image that combines all the conveyed concepts in a coherent manner. In practice, we employ the prompt “” to facilitate this process. For a visual examples, please refer to <ref>-(E) in the main paper.

§.§ Applications

The versatility of in binding diverse modalities into a unified space unlocks a multitude of applications, including cross-modal retrieval and semantic search.
This section demonstrates the application of in the domain of any-modality to 3D scene understanding, leveraging the capabilities of the recent OpenScene framework [Peng et al., 2023]. OpenScene aligns 3D point features within the CLIP embedding space, enabling text-based and image-based searches within a 3D scene. Building upon OpenScene, extends this understanding of 3D scenes to encompass more modalities.

The qualitative results in <ref> demonstrate this application's ability to utilize inputs from multiple modalities to identify relevant areas within the scene. It effectively highlights objects like the toilet flush based on toilet audio, the sink area using 3D point cloud data of a water sink, the kitchen area from the depth map, and the presence of sofas inferred from tactile input indicating a leather sofa.

Application for any-modality to 3D scene understanding.
This application facilitates scene exploration by accepting inputs from diverse modalities and subsequently highlighting relevant areas within the scene. In the visualization, the color gradient represents the relevance level within the scene (yellow is the highest, green is moderate, blue is low, and uncolored is lowest).

§.§ Additional Ablation Studies

This section presents additional ablation experiment findings regarding training.

§.§.§ Anchor Data for Alignment

We study the effect of using different anchor data for multimodal alignment during training. We employ in experiments. We train for 3D point cloud on ULIP-ShapeNet and follow the main settings for other modalities. The results are shown in <ref>.
Our observations reveal that employing both image and text as anchor data yields superior performance for tasks involving 3D point clouds, depth, and audio. In contrast, utilizing only image or text alone results in comparatively lower accuracy.
For tactile and EEG tasks, aligning with text produces the best results. Our speculation is that in the case of tactile data, the aligned images depict close-up views of objects, differing from those used in CLIP training. Consequently, the CLIP image encoder might not offer the optimal alignment space.
As for EEG, due to the very limited scale of data, employing text-only alignment seems to be the most effective approach.

§.§.§ Different Ratio of Training Data

Our investigation delves into the influence of training data by performing ablation studies using different ratios of OpenShape Triplets [Liu et al., 2023] for training a 3D point cloud encoder. Specifically, we compare the performance of against OpenShape PointBERT in zero-shot classification on O-LVIS. The results, presented in  <ref>, demonstrate that consistently outperforms OpenShape PointBERT across all ratios.
Remarkably, in scenarios with limited training data (e.g., 1% training data), showcases a significant performance advantage over PointBERT. This suggests the data-efficient nature of , attributed to the rich knowledge encapsulated within .

§.§.§ Additional Architectural Ablations

This section focuses on additional ablation experiments centered around architectural designs. We focus on 3D tasks in this section. By default, our pretraining phase utilizes ULIP-ShapeNet Triplets [Xue et al., 2023], followed by evaluation on the ModelNet40 [Wu et al., 2015] benchmark on zero-shot classification.

Comparison with PointBERT.
We conduct experiments to compare with PointBERT [Yu et al., 2022], a transformer based architecture for 3D point cloud understanding. This comparison involves aligning to the feature space of different CLIP variants and employing distinct pretraining datasets (ULIP-ShapeNet, ULIP2-Objaverse and OpenShape Triplets).
As is shown in <ref>, outperforms PointBERT over all combinations of pretraining datasets and CLIP model for alignment.  This substantiates the efficacy of harnessing a pretrained to advance 3D shape understanding.

Configuration of Iter-CA-Attn type Lens in . We delve into the impact of different design choices of the Iter-CA-Attn type employed in for 3D encoder.
Our study encompasses the ablation of number of basis blocks (depth), as well as the exploration of parameter sharing beyond the second basis block (included), following [Jaegle et al., 2021].
The results outlined in  <ref> indicate that, beyond a certain threshold, notably four in our setting, increasing the number of basis blocks does not yield improvements in performance. Moreover, parameter sharing among blocks demonstrates its capability to reduce parameters while achieving comparable performance. This emphasizes the efficacy and efficiency of the Iter-CA-Attn Lens architecture within for establishing connections between the 3D input and a .

Other hyper-parameters in .
We vary the number of latents used in the Lens of . Note that the number of latents equals to the sequence length of the input.
As delineated in <ref>, employing a larger number of latents, such as 384 and 512, shows slightly improved performance while concurrently increasing computational complexity measured in GFlops.
This observation underscores the inherent capability of the CA-Iter-Attn type Lens to extract information from inputs of variable sizes and seamlessly connect them to the , mitigating computational complexity.
Additionally, we investigate whether the inclusion of the position embedding influences model performance. Specifically, we interpolate the original position embedding while varying the number of latents. The results presented in <ref> suggest that omitting the pretrained position embedding does not notably degrade performance. This suggests that the Lens is able to implicitly assimilate position information.

PointEmbed $\rightarrow$ Lens. To validate the efficacy of the , we investigate the performance of the “PointEmbed $\rightarrow$ Lens” paradigm. In this setup, the mean pooling feature of the CA-Iter-Attn Lens aligns directly with the CLIP feature space.
We conduct experiments with various hyper-parameter configurations, and the comprehensive outcomes are presented in <ref>.
Specifically, the configuration featuring a “depth of 6, with no parameter sharing” possesses a total parameter count comparable to the default setting of (approximately 119M parameters).
Despite having less trainable parameters, outperforms this variant of “PointEmbed $\rightarrow$ Lens” by a significant margin. Besides, also outperforms the rest variants.
This observation underscores the importance of harnessing the capabilities of the .

PointEmbed $\rightarrow$ . We also delve into the paradigm of “PointEmbed $\rightarrow$ ”. As detailed in  <ref>, training only the PointEmbed yields a zero-shot accuracy of 50%, significantly lower than that achieved by due to the restricted number of trainable parameters.
Subsequently, enabling the training of transformer blocks results in an improved zero-shot performance. However, this specialized training approach tailored specifically for enhancing 3D understanding might limit the adaptability of the resulting to other modalities, potentially impacting the overall generalization ability of the .
In contrast, achieves commendable performance while largely preserving the core parameters of the . This strategy effectively harnesses the extensive knowledge embedded within the across diverse modalities, with only a marginal increase in new parameters, showcasing its robustness and adaptability.

§ FURTHER DISCUSSION

Beyond using .
The core of in advancing representations across diverse modalities relies on leveraging the profound knowledge embedded within the . Given the significant enhancements facilitated by the , an initial exploration involves employing the powerful Large Language Model (LLM) to encode inputs across various modalities.
In this endeavor, we replace the with Flan-T5 XL [Chung et al., 2022] within the architecture. To facilitate alignment, we introduce an additional trainable token. Training the model on ULIP-ShapeNet and ULIP2-Objaverse under various experimental configurations, we report the zero-shot classification performance on MN40.
Results are show in <ref>. Notably, when trained on ULIP-ShapeNet, the model exhibits proficient alignment with CLIP (I+T), achieving a notable top-1 zero-shot accuracy of 62.6% on MN40.
Moreover, upon scaling the model to the ULIP2-Objaverse dataset enriched with textual captions, a remarkable improvement is observed. Specifically, it achieves an outstanding top-1 accuracy of 79%, surpassing the performance obtained by training PointBERT from scratch with the same CLIP model for alignment. This outcome underscores the potential of this approach for omni-modal learning.
We leave further exploration of this promising avenue to future work.

[Alayrac et al., 2022]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
arXiv, 2022.

[Arandjelovic and Zisserman, 2017]
Relja Arandjelovic and Andrew Zisserman.
Look, listen and learn.
In ICCV, 2017.

[Bachmann et al., 2022]
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir.
Multimae: Multi-modal multi-task masked autoencoders.
In ECCV, 2022.

[Bai et al., 2023]
Yunpeng Bai, Xintao Wang, Yanpei Cao, Yixiao Ge, Chun Yuan, and Ying Shan.
Dreamdiffusion: Generating high-quality images from brain eeg signals.
arXiv, 2023.

[Bao et al., 2021]
Hangbo Bao, Li Dong, and Furu Wei.
Beit: Bert pre-training of image transformers.
arXiv, 2021.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
NeurIPS, 2020.

[Caron et al., 2021]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In ICCV, 2021.

[Chang et al., 2015]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.
Shapenet: An information-rich 3d model repository.
arXiv, 2015.

[Chen et al., 2020]
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.
Vggsound: A large-scale audio-visual dataset.
In ICASSP, 2020.

[Chen et al., 2023]
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu.
Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.
arXiv, 2023.

[Chen et al., 2022]
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.
Pali: A jointly-scaled multilingual language-image model.
arXiv, 2022.

[Cherti et al., 2022]
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.
Reproducible scaling laws for contrastive language-image learning.
arXiv, 2022.

[Chung et al., 2022]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
Scaling instruction-finetuned language models.
arXiv, 2022.

[Collins et al., 2022]
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al.
Abo: Dataset and benchmarks for real-world 3d object understanding.
In CVPR, 2022.

[Cubuk et al., 2020]
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.
Randaugment: Practical automated data augmentation with a reduced search space.
In CVPR Workshop, 2020.

[Dai et al., 2023]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
Instructblip: Towards general-purpose vision-language models with instruction tuning.
arXiv, 2023.

[Deitke et al., 2023]
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.
Objaverse: A universe of annotated 3d objects.
In CVPR, 2023.

[18]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR.

[Devlin et al., 2018]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv, 2018.

[Dosovitskiy et al., 2020]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv, 2020.

[Driess et al., 2023]
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.
Palm-e: An embodied multimodal language model.
arXiv, 2023.

[Drossos et al., 2020]
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.
Clotho: An audio captioning dataset.
In ICASSP, 2020.

[Fang et al., 2023]
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.
Eva-02: A visual representation for neon genesis.
arXiv, 2023a.

[Fang et al., 2023]
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.
Eva: Exploring the limits of masked visual representation learning at scale.
In CVPR, 2023b.

[Fu et al., 2021]
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.
3d-future: 3d furniture shape with texture.
In IJCV, 2021.

[Gao et al., 2020]
Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen.
Multi-modal graph neural network for joint reasoning on vision and scene text.
In CVPR, 2020.

[Ge et al., 2023]
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.
Planting a seed of vision in large language model.
arXiv, 2023a.

[Ge et al., 2023]
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan.
Making llama see and draw with seed tokenizer.
arXiv, 2023b.

[Gemmeke et al., 2017]
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter.
Audio set: An ontology and human-labeled dataset for audio events.
In ICASSP, 2017.

[Ghiasi et al., 2022]
Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein.
What do vision transformers learn? a visual exploration.
arXiv, 2022.

[Girdhar et al., 2022]
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra.
Omnivore: A single model for many visual modalities.
In CVPR, 2022.

[Girdhar et al., 2023]
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.
Imagebind: One embedding space to bind them all.
In CVPR, 2023a.

[Girdhar et al., 2023]
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.
Omnimae: Single model masked pretraining on images and videos.
In CVPR, 2023b.

[Gong et al., 2021]
Yuan Gong, Yu-An Chung, and James Glass.
AST: Audio Spectrogram Transformer.
In Interspeech, 2021.

[Gu et al., 2021]
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.
Open-vocabulary object detection via vision and language knowledge distillation.
arXiv, 2021.

[Guo et al., 2023]
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al.
Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following.
arXiv, 2023.

[Gupta et al., 2019]
Agrim Gupta, Piotr Dollar, and Ross Girshick.
Lvis: A dataset for large vocabulary instance segmentation.
In CVPR, 2019.

[Guzhov et al., 2022]
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel.
Audioclip: Extending clip to image, text and audio.
In ICASSP, 2022.

[Han et al., 2023]
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al.
Imagebind-llm: Multi-modality instruction tuning.
arXiv, 2023.

[He et al., 2022]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners.
In CVPR, 2022.

[Jaegle et al., 2021]
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.
Perceiver: General perception with iterative attention.
In ICML, 2021.

[Jia et al., 2021]
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with noisy text supervision.
In ICML, 2021.

[Johnson and Adelson, 2009]
Micah K. Johnson and Edward H. Adelson.
Retrographic sensing for the measurement of surface texture and shape.
In CVPR, 2009.

[Kim et al., 2019]
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim.
Audiocaps: Generating captions for audios in the wild.
In NAACL-HLT, 2019.

[Lei et al., 2023]
Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dylan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou.
Vit-lens: Towards omni-modal representations.
arXiv, 2023.

[Li et al., 2022]
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.
Language-driven semantic segmentation.
In ICLR, 2022a.

[Li et al., 2022]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In ICML, 2022b.

[Li et al., 2023]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
arXiv, 2023a.

[Li et al., 2023]
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.
Videochat: Chat-centric video understanding.
arXiv, 2023b.

[Likhosherstov et al., 2021]
Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani.
Polyvit: Co-training vision transformers on images, videos and audio.
arXiv, 2021.

[Lin et al., 2022]
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al.
Egocentric video-language pretraining.
NeurIPS, 2022.

[Liu et al., 2023]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.

[Liu et al., 2023]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
arXiv, 2023b.

[Liu et al., 2023]
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su.
Openshape: Scaling up 3d shape representation towards open-world understanding.
arXiv, 2023c.

[Liu et al., 2019]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv, 2019.

[Ma et al., 2022]
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu.
Rethinking network design and local geometry in point cloud: A simple residual mlp framework.
arXiv, 2022.

[Miech et al., 2019]
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.
Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.
In ICCV, 2019.

[Mokady et al., 2021]
Ron Mokady, Amir Hertz, and Amit H Bermano.
Clipcap: Clip prefix for image captioning.
arXiv, 2021.

[Morgado et al., 2021]
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra.
Audio-visual instance discrimination with cross-modal agreement.
In CVPR, 2021.

[Nagrani et al., 2022]
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid.
Learning audio-video modalities from image captions.
In ECCV, 2022.

[Nathan Silberman and Fergus, 2012]
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.
Indoor segmentation and support inference from rgbd images.
In ECCV, 2012.

[Oncescu et al., 2021]
Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie.
Audio retrieval with natural language queries.
arXiv, 2021.

[OpenAI, 2021]
Introducing chatgpt.
OpenAI Blog, 2021.

[OpenAI, 2023]
Gpt-4 technical report, 2023.

[Oquab et al., 2023]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
arXiv, 2023.

[Park et al., 2019]
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.
Specaugment: A simple data augmentation method for automatic speech recognition.
arXiv, 2019.

[Patrick et al., 2021]
Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi.
Support-set bottlenecks for video-text representation learning.
In ICLR, 2021.

[Peng et al., 2023]
Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser.
Openscene: 3d scene understanding with open vocabularies.
In CVPR, 2023.

[Peng et al., 2022]
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei.
Beit v2: Masked image modeling with vector-quantized visual tokenizers.
arXiv, 2022.

[Piczak, 2015]
Karol J. Piczak.
ESC: Dataset for Environmental Sound Classification.
In ACM MM, 2015.

[Qi et al., 2017]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3d classification and segmentation.
In CVPR, 2017a.

[Qi et al., 2017]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.
Pointnet++: Deep hierarchical feature learning on point sets in a metric space.
NeurIPS, 2017b.

[Qian et al., 2022]
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem.
Pointnext: Revisiting pointnet++ with improved training and scaling strategies.
NeurIPS, 2022.

[Radford et al., 2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.
Improving language understanding by generative pre-training.
OpenAI blog, 2018.

[Radford et al., 2019]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners.
OpenAI blog, 2019.

[Radford et al., 2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In ICML, 2021.

[Raghu et al., 2021]
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.
Do vision transformers see like convolutional neural networks?
NeurIPS, 2021.

[Song et al., 2015]
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.
Sun rgb-d: A rgb-d scene understanding benchmark suite.
In CVPR, 2015.

[Spampinato et al., 2017]
Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, and Mubarak Shah.
Deep learning human mind for automated visual classification.
In CVPR, 2017.

[Su et al., 2023]
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.
Pandagpt: One model to instruction-follow them all.
arXiv, 2023.

[Sun et al., 2023]
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
Generative pretraining in multimodality.
arXiv, 2023.

[Tian et al., 2020]
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
Contrastive multiview coding.
In ECCV, 2020a.

[Tian et al., 2020]
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
Contrastive multiview coding.
In ECCV, 2020b.

[Touvron et al., 2023]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv, 2023.

[Uy et al., 2019]
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung.
Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.
In ICCV, 2019.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
NeurIPS, 2017.

[Wang et al., 2023]
Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou.
One-peace: Exploring one general representation model toward unlimited modalities.
arXiv, 2023.

[Wu et al., 2023]
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.
Next-gpt: Any-to-any multimodal llm.
arXiv, 2023.

[Wu* et al., 2023]
Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.
Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.
In ICASSP, 2023.

[Wu et al., 2015]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.
3d shapenets: A deep representation for volumetric shapes.
In CVPR, 2015.

[Xu et al., 2016]
Jun Xu, Tao Mei, Ting Yao, and Yong Rui.
Msr-vtt: A large video description dataset for bridging video and language.
CVPR, 2016.

[Xue et al., 2023]
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.
Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding.
In CVPR, 2023a.

[Xue et al., 2023]
Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.
Ulip-2: Towards scalable multimodal pre-training for 3d understanding.
arXiv, 2023b.

[Yang et al., 2022]
Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens.
Touch and go: Learning from human-collected vision and touch.
In NeurIPS, 2022.

[Youwang et al., 2022]
Kim Youwang, Kim Ji-Yeon, and Tae-Hyun Oh.
Clip-actor: Text-driven recommendation and stylization for animating human meshes.
In ECCV, 2022.

[Yu et al., 2022]
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.
Coca: Contrastive captioners are image-text foundation models.
arXiv, 2022a.

[Yu et al., 2022]
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.
Point-bert: Pre-training 3d point cloud transformers with masked point modeling.
In CVPR, 2022b.

[Zhai et al., 2022]
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.
Lit: Zero-shot transfer with locked-image text tuning.
In CVPR, 2022.

[Zhang et al., 2017]
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.
mixup: Beyond empirical risk minimization.
arXiv, 2017.

[Zhang et al., 2023]
Jianfeng Zhang, Hanshu Yan, Zhongcong Xu, Jiashi Feng, and Jun Hao Liew.
Magicavatar: Multi-modal avatar generation and animation.
In arXiv, 2023a.

[Zhang et al., 2022]
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.
Pointclip: Point cloud understanding by clip.
In CVPR, 2022.

[Zhang et al., 2023]
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue.
Meta-transformer: A unified framework for multimodal learning.
arXiv, 2023b.

[Zhong et al., 2020]
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.
Random erasing data augmentation.
In AAAI, 2020.

[Zhou et al., 2022]
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.
Detecting twenty-thousand classes using image-level supervision.
In ECCV, 2022.

[Zhu et al., 2023]
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
arXiv, 2023.

[Zhu et al., 2022]
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao.
Pointclip v2: Adapting clip for powerful 3d open-world learning.
arXiv, 2022.