# TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding

Zhihao Zhang1* Shengcao Cao2 Yu-Xiong Wang2
1Xi’an Jiaotong University 2University of Illinois at Urbana-Champaign
<EMAIL_ADDRESS><EMAIL_ADDRESS>equal contribution

###### Abstract

The limited scale of current 3D shape datasets hinders the advancements in 3D
shape understanding, and motivates multi-modal learning approaches which
transfer learned knowledge from data-abundant 2D image and language modalities
to 3D shapes. However, even though the image and language representations have
been aligned by cross-modal models like CLIP, we find that the image modality
fails to contribute as much as the language in existing multi-modal 3D
representation learning methods. This is attributed to the domain shift in the
2D images and the distinct focus of each modality. To more effectively
leverage both modalities in the pre-training, we introduce TriAdapter Multi-
Modal Learning (TAMM) – a novel two-stage learning approach based on three
synergetic adapters. First, our CLIP Image Adapter mitigates the domain gap
between 3D-rendered images and natural images, by adapting the visual
representations of CLIP for synthetic image-text pairs. Subsequently, our Dual
Adapters decouple the 3D shape representation space into two complementary
sub-spaces: one focusing on visual attributes and the other for semantic
understanding, which ensure a more comprehensive and effective multi-modal
pre-training. Extensive experiments demonstrate that TAMM consistently
enhances 3D representations for a wide range of 3D encoder architectures, pre-
training datasets, and downstream tasks. Notably, we boost the zero-shot
classification accuracy on Objaverse-LVIS from 46.8 to 50.7, and improve the
5-way 10-shot linear probing classification accuracy on ModelNet40 from 96.1
to 99.0. Project page: https://alanzhangcs.github.io/tamm-page.

Figure 1: Our TriAdapter Multi-Modal Learning (TAMM) significantly enhances 3D
shape understanding. Left: When aligning features of 3D point clouds (P) with
2D images (I) and/or texts (T), prior methods (_e.g_., ULIP [51] and OpenShape
[26]) _under-utilize the image modality_ , due to the overlooked or unsolved
image domain gap. TAMM better exploits the image modality and brings more
gains when learning from both image and text data. The results are produced by
pre-training Point-BERT [53] on ShapeNet [4]. Middle: Our CLIP Image Adapter
(CIA) _re-aligns_ the images rendered from 3D shapes with the text
descriptions. The rendered images are inaccurately matched with texts when the
image features are directly extracted by CLIP, and CIA can correct the
matching. Right: Our Image Alignment Adapter (IAA) and Text Alignment Adapter
(TAA) _decouple 3D features_ with complementary visual and semantic focuses.
In the visualized examples, features from one single adapter are matched with
classes whose appearance or semantics resemble the true class; using both
adapters leads to the correct class.

##  1 Introduction

Despite the recent success of 3D shape understanding [53, 28, 48, 57, 61, 21],
the limited scale of existing 3D shape datasets hinders the learning of more
robust and generalizable 3D representations. Due to the significant human
labor and expertise required in collecting and annotating 3D shape data,
building large-scale 3D shape datasets is a prohibitive endeavor [4, 30, 10].

To mitigate the challenge of the limited data scale, recent research in 3D
shape representation learning shows a promising direction of multi-modal
learning [26, 51]. By forging bonds among data of 3D shapes, 2D images, and
texts, it becomes possible to pre-train 3D shape representations by
_distilling learned knowledge from image and language modalities_. There
exists a well-established literature of massive datasets [41], self-supervised
pre-training methodologies, and high-quality pre-trained models in the 2D
image [5, 17, 3] and language [11, 2, 44] modalities, and furthermore, the
cross-modality between images and language [38, 6]. For instance, ULIP [51]
creates triplets of 3D point clouds, 2D images, and texts by rendering images
from 3D shapes and generating text descriptions from their metadata. Then,
ULIP adopts contrastive learning to align the 3D shape features with both the
image features and text features extracted by CLIP [38].

While the image and language modalities are pre-aligned by CLIP pre-training
and share the same feature space, we observe that directly aligning 3D
features with image features leads to _considerably worse representation
quality_ as compared with aligning 3D features with text features, as shown in
Figure 1-left. This phenomenon suggests that existing methods (_e.g_., ULIP
[51] and OpenShape [26]) are not optimally leveraging the 2D image modality in
3D representation pre-training.

There are two reasons for this counter-intuitive observation: 1) The 2D images
in the generated triplets follow a data distribution _different from natural
images_ , on which CLIP is pre-trained. The images are rendered or projected
from 3D to 2D, usually lacking a realistic background and texture. Since the
image domain is shifted, the image features are no longer well-aligned with
the text features, as exemplified in Figure 1-middle. 2) Intuitively, the
_image_ features represent more _visual_ attributes including the shape,
texture or color, while the _text_ features have a focus on _semantics_ such
as the function of the object. As shown in Figure 1-right, if the 3D features
are specifically aligned with one single modality of images or texts, they pay
attention to different aspects of the 3D shape. Therefore, enforcing the 3D
shape to simultaneously align with two modalities conveying subtly distinct
information would be challenging and fail to fully exploit the multi-modal
learning signals.

Aiming at addressing these two issues hindering multi-modal pre-training for
3D shape understanding, we propose TriAdapter Multi-Modal Learning (TAMM), a
two-stage pre-training approach based on three synergetic adapters. In the
first stage, to mitigate the domain gap between the 2D images rendered from 3D
shapes and the natural images on which CLIP is pre-trained, we adapt the
visual representations of CLIP based on the synthetic image-text pairs. More
specifically, we fine-tune a lightweight CLIP Image Adapter on top of CLIP’s
visual encoder through contrastive learning and re-align the adapted image
features with the text features in the new domain. This CLIP Image Adapter
allows us to establish more accurate relations between 3D shapes, 2D images,
and language in an updated feature space, without learning 3D representations
from mismatched image features and text features.

In the following second stage, to avoid the vision-semantic feature disparity
from impairing our 3D representation pre-training, we choose to embrace this
disparity and adapt the 3D representations into two sub-spaces. To
comprehensively encode a 3D shape, the 3D encoder needs to capture both the
visual and semantic aspects of its representation, which are centric to the
corresponding image and text features, respectively. Therefore, we decouple
the 3D feature space for the two focuses on visual and semantic
representations. In particular, we attach two independent feature adapters to
the 3D backbone and transform the 3D features into two sub-spaces. One sub-
space focuses more on the visual representations, and is aligned with the 2D
image feature space; the other sub-space focuses more on the semantic
representations, and is aligned with the language feature space. This approach
of decoupled feature spaces makes the learned 3D representations more
comprehensive and expressive.

To summarize, our main contribution includes:

  * •

We identify the under-utilization of the 2D image modality in existing multi-
modal methods. The image domain gap and feature disparity in image-text pairs
hinders representation learning in 3D shape understanding.

  * •

We propose a novel multi-modal learning framework with two learning stages and
three unified adapter modules. Our proposed TAMM better exploits both image
and language modalities and improves 3D shape representations.

  * •

Our TAMM consistently enhances 3D representations for a variety of 3D encoder
architectures (_e.g_., Point-BERT [53], SparseConv [7]), pre-training datasets
(_e.g_., ShapeNet [4], an ensembled dataset [26]), and downstream tasks
(_e.g_., zero-shot and linear probing shape classification on Objaverse-LVIS
[10], ModelNet40 [49], and ScanObjectNN [45]).

##  2 Related Work

3D Shape Understanding. There are two mainstreams for 3D shape representation
learning: 1) Projecting 3D shapes into voxel or grid-based formats [42] and
then using 2D/3D convolutions [7] for feature extraction. 2) Directly modeling
3D point clouds with point-centric architectures [34, 35, 58, 37, 48, 28, 32,
56]. In this work, to ensure a fair and comprehensive comparison with previous
methods [51, 26] on the pre-training scheme, we follow their selection and
utilize two representative 3D encoders from these two mainstreams: SparseConv
[7] and Point-BERT [53]. SparseConv is designed for efficiently processing
sparse voxels using specialized convolutions that focus computations on non-
zero data points. Point-BERT [48] utilizes a Transformer-based architecture
[46] and can be self-supervised by masked modeling.

Multi-Modal Representation Learning. Contrastive Language–Image Pre-training
(CLIP) [38] has enabled various downstream applications including object
detection [54, 59, 60] and language grounding [25]. Recently, CLIP has been
extended to 3D-based tasks, such as zero-shot text-to-3D generation [19, 43,
31, 22, 29, 50] and scene-level 3D segmentation [16, 33, 52, 40]. Meanwhile,
developing general and robust representations for 3D shape understanding with
the foundation of CLIP [57, 21, 61, 51, 26] becomes a major focus. Among these
methods, ULIP [51], as a pioneering work, utilizes contrastive learning to
distill CLIP features into 3D representations. OpenShape [26] follows this
learning paradigm with a focus on building a larger pre-training dataset with
enriched and filtered text data. Unlike OpenShape [26], we focus on improving
the multi-modal learning paradigm by more effectively leveraging both the
image and text modalities via a two-stage pre-training approach based on three
synergetic adapters.

Figure 2: Overview of TAMM. Left: In Stage 1, TAMM fine-tunes a lightweight
CLIP Image Adapter (CIA) through contrastive learning and re-aligns the image
features with the text features to alleviate the domain shift originated from
rendered images. Middle: In Stage 2, TAMM introduces Image Alignment Adapter
(IAA) and Text Alignment Adapter (TAA) to decouple 3D representations into two
sub-spaces: one focusing more on visual attributes and the other for semantic
understanding, ensuring a more comprehensive and effective multi-modal pre-
training strategy. Right: TAMM adaptively utilizes decoupled 3D features for
various downstream tasks including linear probing classification (top) and
zero-shot classification (bottom), achieving more robust classification
results.

##  3 Method

Figure 2 shows our proposed two-stage approach, TriAdapter Multi-Modal
Learning (TAMM), for pre-training robust and generalizable 3D representations
by leveraging 2D images and texts. We first revisit the triplet data
generation and problem formulation in multi-modal 3D representation learning.
Section 3.1 delves into our fine-tuning of the original CLIP model for better
fitting 3D understanding tasks. Section 3.2 details our strategy to enhance
alignment across the 3D, 2D, and text feature spaces. Problem Formulation.
Given the triplets $\\{(P_{i},I_{i},T_{i})\\}$, where $P_{i}$ represents a 3D
point cloud, $I_{i}$ is the corresponding image produced by projecting the 3D
point cloud $P_{i}$ into 2D images from various perspectives, $I_{i}$ denotes
the associated text generated using advanced vision-language models such as
BLIP [24], the objective is to learn high-quality 3D representations from the
triplets. The basic framework to achieve this is proposed by ULIP [51] (and
followed by OpenShape [26]) with the help of CLIP. Formally, the 3D feature
$f_{i}^{P}=E_{P}(P_{i})$ is produced by a 3D encoder $E_{P}$, and the
corresponding image feature $f_{i}^{I}=E_{I}(I_{i})$ and text feature
$f_{i}^{T}=E_{T}(T_{i})$ are generated by CLIP encoders $E_{I},E_{T}$ [6].
Then the 3D encoder $E_{P}$ is optimized by aligning the 3D feature space
$f^{P}$ simultaneously to the pre-aligned CLIP image feature space $f^{I}$ and
text feature space $f^{T}$ through contrastive learning. The corresponding
contrastive loss $L_{\text{contrast}}(f^{M_{1}},f^{M_{2}})$ between two
modalities $M_{1},M_{2}$ is formulated as:

$\scriptsize-\frac{1}{2n}\sum_{i}\left(\log\frac{\exp(f_{i}^{M_{1}}\cdot
f_{i}^{M_{2}}/\tau)}{\sum_{j}\exp(f_{i}^{M_{1}}\cdot
f_{j}^{M_{2}}/\tau)}+\log\frac{\exp(f_{i}^{M_{2}}\cdot
f_{i}^{M_{1}}/\tau)}{\sum_{j}\exp(f_{i}^{M_{2}}\cdot
f_{j}^{M_{1}}/\tau)}\right),$ (1)

where $\tau$ denotes the temperature hyperparameter.

###  3.1 Image-Text Re-Alignment

Unlike ULIP [51] and OpenShape [26], which align the 3D feature space with the
image-text feature spaces _directly produced by CLIP_ , we introduce an image
feature space tuning strategy, aiming to _foster better alignment_ across 3D,
2D, and language modalities. We argue that the image feature space produced
directly by CLIP’s visual encoder does not perfectly align with the text
feature space and is thus sub-optimal. The reason is that the 2D images in the
triplets, which originate from projections of 3D point clouds, lack
backgrounds and are visually different from natural images on which CLIP is
pre-trained.

Therefore, when using such 2D images from a shifted data domain to perform
multi-modal pre-training, it becomes necessary to further fine-tune CLIP and
re-align its image and language feature spaces. Inspired by adapter modules in
transfer learning [20], we introduce a CLIP Image Adapter (CIA), to adapt the
image feature space for the rendered 2D images in our data triplets. We append
a lightweight, learnable multi-layer perceptron (MLP) to CLIP’s image encoder,
with a residual connection allowing for a seamless integration of the new
knowledge acquired from fine-tuning with the existing knowledge from the pre-
trained CLIP backbone. To avoid heavy computation and over-fitting, we only
fine-tune the additional parameters in the CIA, instead of the whole CLIP
backbone. Formally, given the image feature $f_{i}^{I}$ and text feature
$f_{i}^{T}$ extracted from the triplet $(P_{i},T_{i},I_{i})$, we use a
learnable, two-layer CIA $A_{C}(\cdot)$ to adapt image feature, formulated as:

$A_{C}(f_{i}^{I})=\sigma(f_{i}^{I}W_{1})\cdot W_{2},$ (2)

where $W_{1}$ and $W_{2}$ are the parameters associated with the linear
transformation layers, and $\sigma$ is the non-linear activation function. The
refined image feature $\widetilde{f}_{i}^{I}$ can be computed with a residual
connection:

$\widetilde{f}_{i}^{I}=\alpha A_{C}(f_{i}^{I})+(1-\alpha)f_{i}^{I},$ (3)

where $\alpha$ is a hyperparameter. Finally, the CIA $A_{C}(\cdot)$ is
optimized by minimizing the contrastive loss function which is difined as:

$\mathcal{L}_{\text{realign}}=L_{\text{contrast}}(\widetilde{f}^{I},f^{T}).$
(4)

###  3.2 Decoupled Tri-Modal Pre-Training

Although the image and language feature spaces have been aligned by CLIP and
our CLIP Image Adapter, they still encode _subtly different information_. For
instance, the color of a 3D shape may be inferred from the image feature, but
it cannot be possibly included in the text feature if the text description
does not contain such information. Similarly, the image feature may not
include semantic information such as the function or name of an object. In
such cases, enforcing the 3D feature to align with both the image and text
features _simultaneously_ is challenging.

In order to alleviate the impediment introduced by aligning the 3D shape
feature space with two distinct modalities, we propose a novel decoupled tri-
modal pre-training framework, which aligns two decoupled 3D shape feature
spaces with the refined image feature space and the text feature space,
respectively. We encourage the 3D encoder to cover both the _visual_
representation and the _semantic_ information inherent to the 3D shape,
escaping from the dilemma of aligning with disparate image and text features
at the same time. Formally, given the triplets $\\{(P_{i},T_{i},I_{i})\\}$, we
first generate the image feature $\widetilde{f}_{i}^{I}$ using the frozen,
adapted 2D image encoder (Equation 3) and text feature $f_{i}^{T}$ using the
original CLIP text encoder [6]. We introduce a pair of lightweight Dual
Adapters: Image Alignment Adapter (IAA) $A_{V}(\cdot)$ and Text Alignment
Adapter (TAA) $A_{S}(\cdot)$. They split the 3D feature $f_{i}^{P}$ originated
from the 3D encoder $E_{P}$ into a vision-focusing feature $f_{i}^{VP}$ and a
semantic-focusing feature $f_{i}^{SP}$, respectively. The Dual Adapters
$A_{V}(\cdot)$ and $A_{S}(\cdot)$ share the same architecture as the CLIP
Image Adapter in Section 3.1 and can be formulated as:

$A_{V}(f_{i}^{P})=\sigma(f_{i}^{P}W_{1}^{V})\cdot W_{2}^{V},$ (5)
$A_{S}(f_{i}^{P})=\sigma(f_{i}^{P}W_{1}^{S})\cdot W_{2}^{S},$ (6)

where $W_{1}^{V}$, $W_{2}^{V}$, $W_{1}^{S}$ and $W_{2}^{S}$ are the parameters
associated with the linear layers, and $\sigma$ represents the activation
function. By decoupling 3D features into these sub-spaces, the 3D encoder
$E_{P}$ interprets the 3D shape with a more comprehensive visual and semantic
understanding, improving the expressivity of the learned representations.

Finally, instead of enforcing the 3D shape feature space to directly mimic the
pre-aligned image-text feature space, we use the decoupled 3D features
$f^{VP}$ , $f^{SP}$ and align them with the adapted image feature space
$\widetilde{f}^{I}$ and the text feature space $f^{T}$, respectively.
Moreover, since one single image can only capture the 3D shape from one
perspective, we align the 3D feature with the adapted features of _multi-view
images_ , to fully exploit the image modality and achieve a better alignment
between 3D and 2D. The overall loss function is defined as:

$\mathcal{L}_{\text{trimodal}}=L_{\text{contrast}}(f^{SP},f^{T})+\frac{1}{m}\sum_{k}^{m}L_{\text{contrast}}(f^{VP},\widetilde{f}^{I,k}),$
(7)

where $m$ represents the number of rendered images and $\widetilde{f}^{I,k}$
is the image feature from the $k$-th view.

Application in Downstream Tasks. The learned 3D feature sub-spaces, $f^{VP}$ ,
$f^{SP}$ can be adaptively applied to a variety of downstream tasks. More
specifically, in the context of zero-shot 3D classification, we leverage both
the 3D vision-focusing feature $f^{VP}$ and the 3D semantic-focusing feature
$f^{SP}$. We calculate the similarity between these features and category
embeddings generated from the CLIP text encoder, respectively. After summing
up the per-category similarity scores, the category with the highest
similarity is chosen as the predicted class, yielding more robust and enhanced
classification results compared to using a single sub-space alone. In the
linear probing classification task, we concatenate the 3D vision-focusing
feature $f^{VP}$ and 3D semantic-focusing feature $f^{SP}$ as the input to the
learnable linear classification layer, providing a more comprehensive and
robust representation for 3D understanding.

##  4 Experiments

Pre-Training Dataset | Models |  Pre-Training Method | Objaverse-LVIS [10] | ModelNet40 [49] | ScanObjectNN [45]
---|---|---|---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
- | PointCLIP [57] | - | 1.9 | 4.1 | 5.8 | 19.3 | 28.6 | 34.8 | 10.5 | 20.8 | 30.6
- | PointCLIP v2 [61] | - | 4.7 | 9.5 | 12.9 | 63.6 | 77.9 | 85.0 | 42.2 | 63.3 | 74.5
ShapeNet | ViT-B/32 [12] | CLIP2Point [21] | 2.7 | 5.8 | 7.9 | 49.5 | 71.3 | 81.2 | 25.5 | 44.6 | 59.4
Transformer [46] | ReCon [36] | 1.1 | 2.7 | 3.7 | 61.2 | 73.9 | 78.1 | 42.3 | 62.5 | 75.6
SparseConv [7] | OpenShape [26] | 11.6 | 21.8 | 27.1 | 72.9 | 87.2 | 93.0 | 52.7 | 72.7 | 83.6
TAMM (Ours) | 13.6 | 24.2 | 29.3 | 74.6 | 88.2 | 94.0 | 57.9 | 75.3 | 83.1
Point-BERT [53] | ULIP [51] | 6.2 | 13.6 | 17.9 | 60.4 | 79.0 | 84.4 | 51.5 | 71.1 | 80.2
OpenShape [26] | 10.8 | 20.2 | 25.0 | 70.3 | 86.9 | 91.3 | 51.3 | 69.4 | 78.4
TAMM (Ours) | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9 | 54.8 | 74.5 | 83.3
Ensembled (no LVIS) | SparseConv [7] | OpenShape [26] | 37.0 | 58.4 | 66.9 | 82.6 | 95.0 | 97.5 | 54.9 | 76.8 | 87.0
TAMM (Ours) | 39.8 | 62.0 | 70.4 | 85.7 | 96.8 | 98.3 | 57.5 | 81.3 | 90.0
Point-BERT [53] | ULIP [51] | 21.4 | 38.1 | 46.0 | 71.4 | 84.4 | 89.2 | 46.0 | 66.1 | 76.4
OpenShape [26] | 39.1 | 60.8 | 68.9 | 85.3 | 96.2 | 97.4 | 47.2 | 72.4 | 84.7
TAMM (Ours) | 42.0 | 63.6 | 71.7 | 86.3 | 96.6 | 98.1 | 56.7 | 78.3 | 86.1
Ensembled | SparseConv [7] | OpenShape [26] | 43.4 | 64.8 | 72.4 | 83.4 | 95.6 | 97.8 | 56.7 | 78.9 | 88.6
TAMM (Ours) | 43.8 | 66.2 | 74.1 | 85.4 | 96.4 | 98.1 | 58.5 | 81.3 | 89.5
Point-BERT [53] | ULIP [51] | 26.8 | 44.8 | 52.6 | 75.1 | 88.1 | 93.2 | 51.6 | 72.5 | 82.3
OpenShape [26] | 46.8 | 69.1 | 77.0 | 84.4 | 96.5 | 98.0 | 52.2 | 79.7 | 88.7
TAMM (Ours) | 50.7 | 73.2 | 80.6 | 85.0 | 96.6 | 98.1 | 55.7 | 80.7 | 88.9

Table 1: Zero-shot 3D classification results. TAMM sets a new state of the art
in zero-shot classification accuracy across Objaverse-LVIS, ModelNet-40, and
ScanObjectNN benchmarks, outperforming existing methods in diverse settings of
pre-training datasets and 3D model architectures. The performance gain is more
significant on the most challenging long-tailed Objaverse-LVIS dataset.

Pre-Training Datasets. Following the state-of-the-art method, OpenShape [26],
our TAMM is pre-trained on the triplets generated from four datasets:
ShapeNetCore [4], 3D-FUTURE [13], ABO [8], and Objaverse [10]. Our training
sets are defined as follows: “ShapeNet” is a triplet set derived exclusively
from the ShapeNetCore dataset, containing $52,470$ 3D shapes with
corresponding images and texts; “Ensembled (no LVIS)” is a set of $829,460$
triplets from the above datasets excluding Objaverse-LVIS; “Ensembled” denotes
the triplet set comprising data from all four datasets, containing $875,665$
3D shapes and their associated images and texts.

Evaluation Datasets. Our TAMM is evaluated on the following datasets:
Objaverse-LVIS [10], ModelNet40 [49], ScanObjectNN [45], and ScanNet [9].
Objaverse-LVIS encompasses a broad range of categories, featuring $46,832$
high-quality shapes distributed across $1,156$ LVIS [15] categories.
ModelNet40 is a synthetic indoor 3D dataset comprising $40$ categories. We use
the test split of $2,468$ shapes in our experiments. ScanObjectNN consists of
scanned objects from $15$ common categories. It has three main variants: OBJ-
BG, OBJ-ONLY and PB-T50-RS. ScanNet, characterized by its real-world scans,
includes 1,513 indoor scenes containing 36,213 objects. We conduct experiments
on four tasks including zero-shot 3D classification, linear probing 3D
classification, few-shot linear probing 3D classification, and real-world
recognition, to demonstrate the advantages of our TAMM.

###  4.1 Zero-Shot 3D Classification

Zero-shot ability is a key metric for reflecting the quality of learned 3D
representations, which requires the representations to be _directly applicable
to datasets where the model has never been explicitly supervised_. Without any
further tuning, the 3D representations are compared with text embeddings of
categories to predict the classes of 3D shapes. To make a fair comparison and
keep consistency with prior work, we adopt the same settings as OpenShape [26]
on three benchmarks: Objaverse-LVIS [10], ModelNet40 [49] and OBJ-ONLY
(ScanObjectNN) [45].

The results are summarized in Table 1. First, we observe that our method,
benefited from multi-modal pre-training, significantly outperforms PointCLIP
[57] and PointCLIP v2 [61] by a large margin. Furthermore, it is evident that
our pre-trained models consistently outperform contemporary pre-training
methodologies like ULIP and OpenShape, _irrespective of_ whether they are pre-
trained on smaller datasets like ShapeNet or more expansive ones such as the
Ensembled dataset. For instance, Point-BERT [53] pre-trained by our TAMM on
the Ensembled (no LVIS) dataset surpasses ULIP and OpenShape by margins of
$+20.6\%$ and $+2.9\%$ in Top-1 accuracy on the long-tailed Objaverse-LVIS
benchmark, which validates the effectiveness of our multi-modal pre-training
scheme.

Pre-Training | Method | O-LVIS | M-40 | ScanObjectNN [45]
---|---|---|---|---
Dataset | [10] | [49] | OBJ-BG | OBJ-ONLY | PB-T50-RS
ShapeNet | ULIP [51] | 34.6 | 90.6 | 75.4 | 75.4 | 64.8
OpenShape [26] | 29.3 | 88.5 | 77.8 | 78.5 | 64.1
TAMM (Ours) | 39.1 | 91.0 | 80.6 | 81.1 | 68.5
Rel. Improv. | +9.8 | +2.5 | +2.8 | +2.6 | +4.4
Ensembled | OpenShape [26] | 48.3 | 91.3 | 85.9 | 85.4 | 78.0
TAMM (Ours) | 59.5 | 93.5 | 88.5 | 88.0 | 80.3
Rel. Improv. | +11.2 | +2.2 | +2.6 | +2.6 | +1.7

Table 2: Linear probing 3D classification results. TAMM outperforms previous
methods by a large margin, _e.g_., $+11.2\%$ accuracy gain on the challenging
Objaverse-LVIS dataset.

###  4.2 Linear Probing 3D Classification

To further evaluate the 3D representation quality of our pre-trained models,
we design and conduct linear probing classification experiments. Here, we
first freeze our pre-trained Point-BERT model along with Image Alignment
Adapter (IAA) and Text Alignment Adapter (TAA), and then append a single
learnable linear layer to the model. Given a batch of point clouds, as
illustrated in Figure 2, we generate features from both the IAA and TAA and
concatenate them, and learn the appended linear classification layer on the
concatenated features. We evaluate the accuracy of the linear classifier on
three benchmarks: Objaverse-LVIS [10], ModelNet40 [49], and ScanObjectNN [45].
As introduced above, Objaverse-LVIS [10] is a long-tailed dataset containing
$1,156$ categories. To ensure evaluation with statistically meaningful number
of samples per class, we exclude categories with fewer than $10$ instances,
leaving us $1,046$ categories. Samples in each class are then divided into
training and testing sets at an $8:2$ ratio. For ModelNet40 and ScanObjectNN,
we just use the standard splits, following [48, 36].

The results are shown in Table 2. TAMM consistently outperforms all previous
methods by a large margin. For example, pre-trained on the Ensembled dataset,
TAMM improves over OpenShape by $+11.2\%$ and $+2.2\%$ overall accuracy on
Objaverse-LVIS and ModelNet40, respectively. This demonstrates that TAMM
effectively exploits knowledge from CLIP and learns generalizable 3D
representations that are _directly applicable to novel linear classification
tasks_. Notably, TAMM even surpasses the Point-BERT which is pre-trained
without multi-modality but allowed to be _fully fine-tuned_ on OBJ-BG
($87.4\%$) [53] by $+1.1\%$ accuracy. These strong results indicate that TAMM
has learned excellent transferable 3D representations, showing a great
potential in real-world applications.

###  4.3 Few-Shot Linear Probing 3D Classification

Following previous evaluations, we perform few-shot classification experiments
on the ModelNet40 [49] to assess TAMM in low-data scenarios. Similar to the
linear probing experiments described in Section 4.2, we extend our model with
an additional linear classification layer and only train this linear layer
instead of fine-tuning the entire model. Following previous work [53, 56], we
adopt the “$K$-way $N$-shot” configuration, wherein we select $K$ classes at
random and sample ($N+20$) instances per class. Training is conducted on a
“support set” of $K\times N$ samples, while evaluation is based on a “query
set” comprised of the remaining $20$ instances per class. We assess our model
under four distinct scenarios: “$5$-way $10$-shot”, “$5$-way $20$-shot”,
“$10$-way $10$-shot”, and “$10$-way $20$-shot”. For each scenario, we carry
out $10$ separate trials, and report both the mean performance and the
standard deviation across these trials.

Pre-Training | Method | 5-way | 10-way
---|---|---|---
Dataset | 10-shot | 20-shot | 10-shot | 20-shot
ShapeNet | ULIP [51] | $94.4_{\pm 3.7}$ | $93.2_{\pm 4.2}$ | $86.6_{\pm 5.3}$ | $90.6_{\pm 5.2}$
OpenShape [26] | $95.3_{\pm 2.6}$ | $97.9_{\pm 3.9}$ | $89.2_{\pm 5.1}$ | $92.9_{\pm 3.9}$
TAMM (Ours) | $97.8_{\pm 1.9}$ | $98.1_{\pm 1.4}$ | $95.3_{\pm 3.9}$ | $95.9_{\pm 2.6}$
Ensembled | OpenShape [26] | $96.1_{\pm 2.7}$ | $95.7_{\pm 2.5}$ | $89.1_{\pm 4.6}$ | $91.8_{\pm 3.7}$
TAMM (Ours) | $\bm{99.0_{\pm 1.3}}$ | $\bm{99.4_{\pm 0.7}}$ | $\bm{96.8_{\pm 2.9}}$ | $\bm{97.4_{\pm 2.2}}$

Table 3: Few-shot linear probing classification results on ModelNet40. We
report the average accuracy and standard deviation of 10 independent
experiments. Our TAMM consistently achieves both the best average accuracy and
the lowest variance in various few-shot settings.

Method | Avg. | Bed | Cab | Chair | Sofa | Tabl | Door | Wind | Bksf | Pic | Cntr | Desk | Curt | Fridg | Bath | Showr | Toil | Sink
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP2Point [21] | 24.9 | 20.8 | 0.0 | 85.1 | 43.3 | 26.5 | 69.9 | 0.0 | 20.9 | 1.7 | 31.7 | 27.0 | 0.0 | 1.6 | 46.5 | 0.0 | 22.4 | 25.6
PointCLIP w/ TP. [61] | 26.1 | 0.0 | 55.7 | 72.8 | 5.0 | 5.1 | 1.7 | 0.0 | 77.2 | 0.0 | 0.0 | 51.7 | 0.3 | 0.0 | 0.0 | 40.3 | 85.3 | 49.2
CLIP2Point w/ TP. [21] | 35.2 | 11.8 | 3.0 | 45.1 | 27.6 | 10.5 | 61.5 | 2.6 | 71.9 | 0.3 | 33.6 | 29.9 | 4.7 | 11.5 | 72.2 | 92.4 | 86.1 | 34.0
CLIP2 [55] | 38.5 | 32.6 | 67.2 | 69.3 | 42.3 | 18.3 | 19.1 | 4.0 | 62.6 | 1.4 | 12.7 | 52.8 | 40.1 | 9.1 | 59.7 | 41.0 | 71.0 | 45.5
OpenShape$\dagger$ [26] | 45.6 | 66.7 | 3.2 | 75.8 | 83.5 | 37.7 | 49.2 | 47.5 | 64.9 | 48.2 | 1.9 | 66.1 | 70.2 | 1.8 | 50.0 | 57.1 | 45.2 | 7.1
TAMM (Ours)$\dagger$ | 49.4 | 66.7 | 4.8 | 83.6 | 84.5 | 48.9 | 57.9 | 48.2 | 80.5 | 61.3 | 1.9 | 60.6 | 83.6 | 7.0 | 41.4 | 56.1 | 48.4 | 3.6

w/ TP. denotes training with the real-world data provided by CLIP2. $\dagger$
Results using Point-BERT [53] as 3D encoder, pre-trained on the Ensembled
dataset.

Table 4: Zero-shot classification results on the real-world ScanNet dataset.
Avg.: the mean average Top-1 accuracy of all classes. TAMM achieves the best
results.

CIA | IAA | TAA |  Img-Txt Acc | Objaverse-LVIS | ModelNet-40
---|---|---|---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
| ✓ | ✓ | 40.1 | 13.5 | 23.9 | 29.1 | 72.8 | 88.2 | 91.7
✓ |  |  | 60.9 | 12.5 | 22.4 | 27.3 | 71.4 | 86.0 | 89.8
✓ |  | ✓ | 60.9 | 12.9 | 22.9 | 27.7 | 73.8 | 88.2 | 92.5
✓ | ✓ |  | 60.9 | 13.0 | 23.0 | 28.2 | 72.9 | 87.8 | 90.4
✓ | ✓ | ✓ | 60.9 | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9

(a) Adapters in pre-training. Pre-training with the CLIP Image Adapter (CIA),
Image Alignment Adapter (IAA) and Text Alignment Adapter (TAA) is the most
effective.

Stages |  Img-Txt Acc | Objaverse-LVIS | ModelNet-40
---|---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
One stage | 55.3 | 12.2 | 21.4 | 27.0 | 71.4 | 86.4 | 90.7
Two stages | 60.9 | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9

(b) Pre-training stages. Learning the CLIP Image Adapter first and then pre-
train the 3D encoder with the Dual Adapters is better than training modules
all together.

Image | Text | Objaverse-LVIS | ModelNet-40
---|---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
✓ |  | 11.9 | 20.8 | 25.9 | 67.6 | 87.3 | 92.5
| ✓ | 10.5 | 19.2 | 23.7 | 70.2 | 85.0 | 88.0
✓ | ✓ | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9

(c) Pre-training modalities. Exploiting both image and text data provides the
most performance gain. They contribute almost equally in TAMM.

IAA | TAA | Objaverse-LVIS | ModelNet-40
---|---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
✓ |  | 13.0 | 23.1 | 28.3 | 68.1 | 86.5 | 91.4
| ✓ | 12.9 | 22.5 | 27.5 | 72.4 | 86.2 | 90.4
✓ | ✓ | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9

(d) Adapters for inference. The Dual Adapters are complementary, and their
combination more accurately classifies 3D shapes at test time.

Images | Objaverse-LVIS | ModelNet-40
---|---|---
Top-1 | Top-3 | Top-5 | Top-1 | Top-3 | Top-5
1 | 13.6 | 23.6 | 28.9 | 72.5 | 88.0 | 92.6
2 | 13.5 | 23.5 | 28.8 | 73.1 | 87.6 | 92.8
4 | 13.7 | 24.2 | 29.2 | 73.1 | 88.5 | 91.9
8 | 13.5 | 23.1 | 28.4 | 73.0 | 89.0 | 92.8

(e) Number of images. Aligning a 3D shape with 4 images of it is the best.

Table 5: Ablation study of components in TAMM. We pre-train Point-BERT models
on ShapeNet in various settings and evaluate theri zero-shot classification
performance on ModelNet40 and Objaverse-LVIS. The setting adopted by TAMM is
marked.

As illustrated in Table 3, our TAMM achieves the best results and sets a new
state of the art in the few-shot classification task. TAMM shows _both higher
overall accuracy and smaller deviations_ than other methods, showcasing the
generalizability and robustness of TAMM-learned 3D representations. For
example, when pre-trained on the Ensembled dataset, our TAMM surpasses
OpenShape [26] by $2.9\%$, $3.7\%$, $7.7\%$, $5.6\%$, respectively for all
four settings. These results indicate that our TAMM is able to learn 3D
representations that are more generalizable and can be readily adapted to
downstream tasks under _low-data regimes_.

###  4.4 Real-World Recognition

To evaluate the capability of TAMM in understanding _3D shapes from the real
world_ , we follow the previous work CLIP2 [55] and test TAMM on a real-world
recognition task, in which the model aims at correctly classifying each
instance from a complex scene in a _zero-shot_ manner. Specifically, we select
the real-world scene-level dataset ScanNet [9] and adopt same data splits as
[55], containing $17$ classes. We perform zero-shot classification (Section
4.1) on the point cloud of each object instance extracted from scenes, and
report both the Top-1 accuracy of each class and the mean accuracy of all $17$
classes. As shown in Table 4, TAMM significantly outperforms all prior
methods, showing improvements of $3.8\%$ and $10.9\%$ in average Top-1
accuracy compared with OpenShape and CLIP2, respectively. These results
underscore TAMM’s ability in recognizing and understanding 3D shapes captured
from real-world scenarios.

###  4.5 Ablation Study

In this section, we provide additional experimental results to further test
the performance gain from each design of TAMM. We pre-train Point-BERT [53] on
ShapeNet [4], and use two challenging and representative zero-shot
classification benchmarks ModelNet40 [49] and Objaverse-LVIS [10] for
evaluation.

Adapters in Pre-Training. As described in Section 3.1, our CLIP Image Adapter
(CIA) mitigates the domain gap between rendered images and natural images. We
first measure the contrastive accuracy, computed as the ratio of image-text
pairs where the text feature is more similar to the image from the same
triplet than any other text. The contrastive accuracy of CLIP without CIA is
only $40.1\%$, indicating that the original image-text feature space is
compromised by the domain gap. CIA brings the accuracy up to $60.9\%$, showing
the effectiveness of our CIA in resolving the shifted data domain.
Furthermore, as shown in Table 5(a), CIA increases the zero-shot accuracy on
Objaverse-LVIS dataset from $12.2\%$ to $13.7\%$, showing its effectiveness on
improving the pre-trained 3D representations.

TAMM adopts the Image Alignment Adapter (IAA) and the Text Alignment Adapter
(TAA) to decouple the feature space and enrich the learned 3D representations.
To further explore the performance gains introduced by our Dual Adapters (IAA
and TAA), we experiment tri-modal pre-training with or without IAA and TAA. As
shown in Table 5(a), both adapters bring performance gains, and their
combination achieves the best result, demonstrating the effectiveness of the
Dual Adapters.

Pre-Training Stages. We investigate the significance of our two-stage pre-
training design, which initially learns the CLIP Image Adapter (CIA) to re-
align image-text pairs and subsequently employs the Dual Adapters in decoupled
tri-modal pre-training. An alternative could be learning these modules all
together in one stage. As shown in Table 5(b), remarkably, the two-stage pre-
training approach achieves a better performance compared with one-stage pre-
training.

Pre-Training Modalities. We further explore the effectiveness of image and
language modalities in pre-training. Table 5(c) shows that integrating the 3D
modality with both the image and language modalities consistently yields
superior performance across various benchmarks, compared with aligning it with
either modality alone. Moreover, unlike prior methods which ineffectively
learn from images (Figure 1-left), the image and language modalities
contribute almost equally in TAMM, which also signifies that TAMM better
exploits the image modality and brings more gains by learning from both image
and text data.

Figure 3: Qualitative results. Top: CIA re-aligns the images rendered from 3D
shapes with the text descriptions. The rendered images are inaccurately
matched with texts when the image features are directly extracted by CLIP, and
CIA can correct the matching. Bottom: IAA and TAA decouple 3D features with
complementary visual and semantic focuses. Features from one single adapter
are matched with classes whose appearance or semantics resemble the true
class; using both adapters leads to the correct class.

Adapters for Inference. Our IAA and TAA decouple 3D features with visual and
semantic focuses, respectively, offering a more accurate and comprehensive
understanding for 3D shapes. At inference time, we combine the outputs from
the Dual Adapters for classification tasks. To investigate this design, we
separately utilize features derived from IAA or TAA in zero-shot
classification experiments. As shown in Table 5(d), the combined features
achieve a more accurate classification, demonstrating that the learned
features from IAA and TAA are complementary.

Number of Images. In order to gain more comprehensive knowledge from the image
modality, TAMM simultaneously aligns the 3D feature with multi-view 2D images,
each projected from a different perspective. We evaluate this design by
aligning the 3D feature with a varied number of corresponding 2D images,
ranging from 1 to 8 images. As illustrated in Table 5(e), multi-view image
features offer a performance gain by fully exploiting the image modality.
Aligning a 3D shape with 4 images of it achieves the best result.

Qualitative Results. Finally, we provide some visualizations to intuitively
demonstrate the benefit of our proposed TAMM. As illustrated in Figure 3, CIA
successfully mitigates the domain gap introduced by rendered images, and leads
to the more accurate matching between images and text. IAA and TAA learn 3D
representations with subtly different but complementary focuses on vision and
semantics, respectively, and their combination brings more robust and
comprehensive 3D representations. More visualized results are included in the
supplementary material.

##  5 Conclusion

In this work, we examine the sub-optimal utilization of 2D images in existing
multi-modal pre-training methods for 3D shape understanding, and propose
TriAdapter Multi-Modal Learning (TAMM), a novel two-stage representation
learning approach built on three synergetic adapter modules. Extensive
experiments verify that TAMM consistently learns improved 3D features in
various settings.

Limitations. Due to the limited computation resources, we are not able to
perform pre-training on very large-scale 3D backbones with billions of
parameters.The quality of learned 3D representation could be further improved
if the 3D backbones are scaled to a larger size.

## References

  * Agarap [2018] Abien Fred Agarap.  Deep learning using rectified linear units (ReLU).  _arXiv preprint arXiv:1803.08375_ , 2018.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _NeurIPS_ , 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chang et al. [2015] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An information-rich 3D model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _ICML_ , 2020.
  * Cherti et al. [2023] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _CVPR_ , 2023.
  * Choy et al. [2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4D spatio-temporal ConvNets: Minkowski convolutional neural networks.  In _CVPR_ , 2019.
  * Collins et al. [2022] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik.  ABO: Dataset and benchmarks for real-world 3D object understanding.  In _CVPR_ , 2022.
  * Dai et al. [2017] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D reconstructions of indoor scenes.  In _CVPR_ , 2017.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3D objects.  In _CVPR_ , 2023.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2020.
  * Fu et al. [2021] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.  3D-FUTURE: 3D furniture shape with texture.  _IJCV_ , 129:3313–3337, 2021.
  * Gao et al. [2023] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  CLIP-Adapter: Better vision-language models with feature adapters.  _IJCV_ , 2023.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  LVIS: A dataset for large vocabulary instance segmentation.  In _CVPR_ , 2019.
  * Ha and Song [2022] Huy Ha and Shuran Song.  Semantic abstraction: Open-world 3D scene understanding from 2D vision-language models.  In _CoRL_ , 2022.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , 2020.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (GELUs).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hong et al. [2022] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  AvatarCLIP: Zero-shot text-driven generation and animation of 3D avatars.  _ACM Transactions on Graphics_ , 41(4):1–19, 2022.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _ICML_ , 2019.
  * Huang et al. [2023] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo.  CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training.  In _ICCV_ , 2023.
  * Jain et al. [2022] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In _CVPR_ , 2022.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , 2022a.
  * Li et al. [2022b] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In _CVPR_ , 2022b.
  * Liu et al. [2023] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su.  OpenShape: Scaling up 3D shape representation towards open-world understanding.  In _NeurIPS_ , 2023.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Ma et al. [2022] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu.  Rethinking network design and local geometry in point cloud: A simple residual MLP framework.  In _ICLR_ , 2022.
  * Michel et al. [2022] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2Mesh: Text-driven neural stylization for meshes.  In _CVPR_ , 2022.
  * Mo et al. [2019] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su.  PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding.  In _CVPR_ , 2019.
  * Mohammad Khalid et al. [2022] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  CLIP-Mesh: Generating textured meshes from text using pretrained image-text models.  In _SIGGRAPH Asia_ , 2022.
  * Pang et al. [2022] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In _ECCV_ , 2022.
  * Peng et al. [2023] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser.  OpenScene: 3D scene understanding with open vocabularies.  In _CVPR_ , 2023.
  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  PointNet: Deep learning on point sets for 3D classification and segmentation.  In _CVPR_ , 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  PointNet++: Deep hierarchical feature learning on point sets in a metric space.  In _NeurIPS_ , 2017b.
  * Qi et al. [2023] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi.  Contrast with reconstruct: Contrastive 3D representation learning guided by generative pretraining.  In _ICML_ , 2023.
  * Qian et al. [2022] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem.  PointNeXt: Revisiting PointNet++ with improved training and scaling strategies.  In _NeurIPS_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Roberts et al. [2021] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind.  Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding.  In _ICCV_ , 2021.
  * Rozenberszki et al. [2022] David Rozenberszki, Or Litany, and Angela Dai.  Language-grounded indoor 3D semantic segmentation in the wild.  In _ECCV_ , 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _NeurIPS Datasets and Benchmarks Track_ , 2022.
  * Shi et al. [2020] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  PV-RCNN: Point-voxel feature set abstraction for 3D object detection.  In _CVPR_ , 2020.
  * Tevet et al. [2022] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  MotionCLIP: Exposing human motion generation to CLIP space.  In _ECCV_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Uy et al. [2019] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung.  Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.  In _ICCV_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Vu et al. [2022] Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, and Chang D. Yoo.  SoftGroup for 3D instance segmentation on 3D point clouds.  In _CVPR_ , 2022.
  * Wu et al. [2022] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao.  Point Transformer V2: Grouped vector attention and partition-based pooling.  In _NeurIPS_ , 2022.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3D ShapeNets: A deep representation for volumetric shapes.  In _CVPR_ , 2015.
  * Xu et al. [2023] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao.  Dream3D: Zero-shot text-to-3D synthesis using 3D shape prior and text-to-image diffusion models.  In _CVPR_ , 2023.
  * Xue et al. [2023] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding.  In _CVPR_ , 2023.
  * Yang et al. [2023] Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi.  RegionPLC: Regional point-language contrastive learning for open-world 3D scene understanding.  _arXiv preprint arXiv:2304.00962_ , 2023.
  * Yu et al. [2022] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-BERT: Pre-training 3D point cloud transformers with masked point modeling.  In _CVPR_ , 2022.
  * Zareian et al. [2021] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In _CVPR_ , 2021.
  * Zeng et al. [2023] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu.  CLIP2: Contrastive language-image-point pretraining from real-world point cloud data.  In _CVPR_ , 2023.
  * Zhang et al. [2022a] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.  Point-M2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training.  In _NeurIPS_ , 2022a.
  * Zhang et al. [2022b] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  PointCLIP: Point cloud understanding by CLIP.  In _CVPR_ , 2022b.
  * Zhang et al. [2021] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra.  Self-supervised pretraining of 3D features on any point-cloud.  In _ICCV_ , 2021.
  * Zhong et al. [2022] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao.  RegionCLIP: Region-based language-image pretraining.  In _CVPR_ , 2022.
  * Zhou et al. [2022] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _ECCV_ , 2022.
  * Zhu et al. [2023] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao.  PointCLIP V2: Prompting CLIP and GPT for powerful 3D open-world learning.  In _ICCV_ , 2023.

Supplementary Material

##  Appendix A Implementation Details

For a fair comparison with previous methods, we adopt two representative 3D
encoders for our study: Point-BERT [53] (Transformer-based) and SparseConv [7]
(convolution-based). We employ OpenCLIP-ViT-G/14 [6] as the pre-trained CLIP
model. TAMM is pre-trained for $200$ epochs using the AdamW optimizer [23, 27]
and a cosine learning rate scheduler and a two-epoch warm-up, with a base
learning rate of $5\times 10^{-4}$. Regarding the CLIP Image Adapter and Dual
Adapters, we set $\alpha$ at $0.2$ in Equation 3 and employ ReLU [1] and GELU
[18] activation functions, respectively, following [14, 51].

##  Appendix B Additional Results on Complex Scene Recognition

To further assess TAMM’s capability in understanding 3D shapes from scene
data, we conduct experiments using the Hypersim dataset [39], a photorealistic
synthetic dataset designed for comprehensive indoor scene understanding. In
this experiment, we extract the point clouds of object instances from
segmentation annotations and focus on $17$ classes to evaluate TAMM’s zero-
shot recognition ability. Some classes are excluded due to their amorphous
shapes (_e.g_., “floor,” “ceiling”) or because they are not well-defined for
classification (_e.g_., “otherfurniture,” “otherstructure”). The results are
detailed in Table 7, which demonstrate that TAMM surpasses OpenShape in terms
of both overall accuracy and average of per-class accuracy, with respective
improvements of $+5.9\%$ and $+1.8\%$. Significantly, TAMM also outperforms
OpenShape in $11$ out of the $17$ evaluated classes. This evaluation on
Hypersim underscores TAMM’s robustness in recognizing and understanding 3D
shapes derived from various scene contexts.

##  Appendix C Additional Results on Instance Segmentation

To delve deeper into TAMM’s proficiency in 3D scene understanding, we test
whether the 3D backbone pre-trained by TAMM can further enhance SoftGroup++
[47], the current state-of-the-art 3D instance segmentation method. More
specifically, we integrate the pre-trained Point-BERT model into the feature
extractor module in the top-down refinement stage of SoftGroup++, and
subsequently fine-tune the classification branch. The 3D instance segmentation
results on ScanNet are illustrated in Table 6. These results reveal that TAMM
can indeed improve the overall performance of SoftGroup++. Notably, TAMM
attains an $AP/AP_{50}$ score of $46.1/68.0(\%)$, marking an enhancement of
$0.6/1.0(\%)$ over SoftGroup++. Furthermore, as an improved pre-training
approach, TAMM exceeds OpenShape by $0.4\%~{}AP$ and $0.6\%~{}AP_{50}$. The
results on real-world instance segmentation underscores TAMM’s significant
potential in the tasks of scene-level 3D understanding.

Method | $AP$ | $AP_{50}$ | $AP_{25}$
---|---|---|---
SoftGroup++$\dagger$ [47] | 45.5 | 67.0 | 78.7
SoftGroup++ & OpenShape [26] | 45.7 | 67.4 | 78.7
SoftGroup++ & TAMM (Ours) | 46.1 | 68.0 | 79.0

$\dagger$ Reproduced using the original implementation [47].

Table 6: 3D instance segmentation results on ScanNet v2. SoftGroup++ & TAMM
improve the $AP$ performance from $45.5\%$ to $46.1\%$, achieving the best
results.

Method | OAvg. | Avg. | Cabi | Bed | Chair | Sofa | Tabl | Door | Bksh | Shlv | Curt | Pill | Clth | TV | Papr | Twl | Nght | Sink | Lamp
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
OpenShape$\dagger$ [26] | 56.7 | 48.8 | 13.0 | 40.0 | 71.2 | 70.7 | 73.3 | 81.4 | 20.5 | 54.6 | 66.7 | 60.0 | 23.9 | 43.1 | 36.5 | 24.0 | 38.1 | 62.0 | 51.0
TAMM (Ours)$\dagger$ | 62.6 | 50.6 | 20.6 | 38.2 | 75.3 | 76.2 | 72.6 | 88.1 | 8.2 | 59.1 | 61.9 | 62.8 | 41.0 | 26.2 | 57.3 | 24.0 | 31.0 | 63.0 | 55.0

$\dagger$ Results using Point-BERT [53] as 3D encoder, pre-trained on the
Ensembled dataset.

Table 7: Zero-shot classification results on the Hypersim dataset. OAvg.:
overall Top-1 accuracy of all shapes, Avg.: the mean average Top-1 accuracy of
all classes. TAMM achieves the best results. Figure 4: Qualitative results of
text-to-point-cloud retrieval. We use TAMM to acquire the features of the
given query texts and retrieve the point clouds with the most similar
features. The shown examples demonstrate TAMM’s strong multi-modal
comprehension. Figure 5: Qualitative results of image-to-point-cloud
retrieval. We use TAMM to acquire the features of the given query images and
retrieve the point clouds with the most similar features. The shown examples
demonstrate TAMM’s strong multi-modal comprehension. Figure 6: Qualitative
results of CLIP Image Adapter (CIA). CIA re-aligns the images rendered from 3D
shapes with the text descriptions. The rendered images are inaccurately
matched with texts when the image features are directly extracted by CLIP, and
CIA can correct the matching.  Figure 7: Qualitative results of Image
Alignment Adapter (IAA) and Text Alignment Adapter (TAA). IAA and TAA decouple
3D features with complementary visual and semantic focuses. Features from one
single adapter are matched with classes whose appearance or semantics resemble
the true class; using both adapters leads to the correct class.

##  Appendix D Additional Qualitative Results

In this section, we provide additional qualitative results to supplement the
visualizations presented in the main body of the paper. Figure 4 showcases
examples of cross-modal retrieval from text to 3D point clouds. Figure 5
showcases examples of cross-modal retrieval from 2D images to 3D point clouds.
More specifically, we extract the adapted features of the query text or image
and employ the TAMM-learned 3D backbone to find the point clouds with the most
similar features. The retrieved point clouds highly resemble objects in the
query texts or images, reflecting that the representations learned by TAMM are
cross-modal and unified. Figure 6 demonstrates how our CLIP Image Adapter
(CIA) effectively bridges the domain gap caused by rendered images, resulting
in more accurate image-text matching. Additionally, Figure 7 illustrates the
distinctive yet synergistic roles of the Image Alignment Adapter (IAA) and
Text Alignment Adapter (TAA). These adapters learn 3D representations with
focuses on vision and semantics, respectively. Their integration yields more
robust and comprehensive 3D representations, highlighting the effectiveness of
our approach.