# 3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

###### Abstract

The remarkable potential of multi-modal large language models (MLLMs) in
comprehending both vision and language information has been widely
acknowledged. However, the scarcity of 3D scenes-language pairs in comparison
to their 2D counterparts, coupled with the inadequacy of existing approaches
in understanding of 3D scenes by LLMs, poses a significant challenge. In
response, we collect and construct an extensive dataset comprising 75K
instruction-response pairs tailored for 3D scenes. This dataset addresses
tasks related to 3D VQA, 3D grounding, and 3D conversation. To further enhance
the integration of 3D spatial information into LLMs, we introduce a novel and
efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the
alignment stage between 3D scenes and language and extends the instruction
prompt with the 3D modality information including the entire scene and
segmented objects. We evaluate the effectiveness of our method across diverse
tasks in the 3D scene domain and find that our approach serves as a strategic
means to enrich LLMs’ comprehension of the 3D world. Our code is available at
https://github.com/staymylove/3DMIT.

Index Terms—  Multi-modal, 3D-LLMs, 3D Scene Understanding

##  1 Introduction

In recent years, Large Language Models (LLMs) such as LLaMA [1], Vicuna [2]
and MiniGPT [3] have emerged as powerful tools in natural language processing,
demonstrating exceptional capabilities in tasks such as language
understanding, generation, and translation. LLMs operate on the principle of
pre-training followed by fine-tuning on specific tasks. This pre-training
phase involves exposure to vast amounts of diverse text data, allowing the
model to learn the intricacies of language. Fine-tuning can tailors the
model’s knowledge to specific domains or tasks, enhancing its performance in
targeted applications.

In response to the remarkable capabilities demonstrated by LLMs, researchers
have extended their scope to develop Multi-modal Language Models (MLLMs).
These MLLMs capitalize on the prowess of existing LLMs to address multi-modal
applications, which involve the fusion of information from diverse sources,
such as images and 3D point clouds. This integration of vision and language
not only expands the utility of LLMs but also elevates them into potent tools
for a range of tasks, covering visual captioning, visual question answering
(VQA), and visual grounding. In recent studies [4, 5, 6], LLMs have been
employed to comprehend the real 3D world. By leveraging 3D scene features and
language prompts, researchers have successfully trained 3D multi-modal
Language Models (3D-LLMs) for scene understanding. Traditional methods which
excel in specific tasks, but these models often lack adaptability to other
tasks. In contrast, 3D-LLMs utilize an unified architecture to perform diverse
tasks, such as 3D VQA and 3D grounding, owing to the reasoning and
understanding capabilities inherent in LLMs. The development of 3D-LLMs thus
holds great promise. However, despite the optimistic outlook for 3D-LLMs,
there are three challenges: 1) there exists a scarcity of high-quality 3D
scene-language data when compared to the abundance of textual data associated
with image and 3D object pairs. 2) the stage of alignment language with the
features of 3D objects and scenes is a time-consuming and laborious process.
It is inefficient to train 3D-LLMs in multiple stages. 3) if training with
object-centric 3D instruction and only objects or only the global scene, the
3D-LLMs may not simultaneously grasp global scene information and fine-grained
object information.

In this paper, we construct a comprehensive 3D scene-language instruction
dataset designed for multi-task applications and we propose 3DMIT, an
efficient 3D multi-modal instructions tuning method to train LLMs [2] and
MLLMs [7] for multi-task scene understanding by leveraging our presented
dataset. Our instruction dataset builds upon existing datasets, including
Scannet [8] and ScanRefer [9]. We utilize the original data and employ the
GPT-API to generate high-quality instructions and responses for both object-
centric and global scene-related tasks. Our 3D instruction dataset encompasses
75K 3D scene-language pairs derived from 3D scene dataset: Scannet. Tasks
addressed in the dataset include 3D VQA, 3D Captioning, 3D Grounding, and 3D
Conversations.

To capture the visual information of the scene, we initially employ a pre-
trained scene encoder [10] $E_{S}$ to extract features $f_{S}$ from the point
cloud of the scene. While this provides global scene features, there is a
potential loss of fine-grained object visual information. To address this
limitation, we subsequently segment the scene and extract the point cloud of
each object. Utilizing a pre-trained 3D encoder [11, 12] $E_{O}$ , we extract
features $f_{O}$ of objects within the scene and fuse with the attributes of
the objects, complementing the global scene features. Notably, different from
Chat-3D [6] and 3D-LLM [4], we skip the stage that employs visual features
aligned with text embeddings, and directly concatenate visual features and
embeddings of text prompts as our 3D multi-modal prompts. With instructions of
each task, we can fine-tune 3D multi-modal LLMs efficiently.

We also evaluate our method on traditional 3D-language downstream datasets,
such as ScanQA [13], ScanRefer [9] and 3D Multi-choice [5]. Our model
outperforms previous 3D-LLMs baselines along with more efficiency.

Our contributions can be summarized as follows:

  * •

We construct a comprehensive 3D scene-language instruction dataset with
GPT4-API, which encompasses 75K 3D scene-language pairs for 3D VQA, 3D Scene
Captioning, 3D Visual Grounding, and 3D Conversation task.

  * •

We propose 3DMIT, an efficient 3D multi-modal instructions tuning method to
train LLMs. Distinguishing itself from other 3D-LLMs, 3DMIT eliminates the
alignment stage between 3D scenes and language, prioritizing efficiency.

  * •

We evaluate our method on traditional 3D-language downstream tasks, ScanQA for
VQA, ScanRefer for 3D grounding and 3D Multi-choice. Our method outperforms
our baselines along with more efficiency.

##  2 Related work

###  2.1 3D Scene Understanding

The 3D scene understanding is similar to existing visual question answering.
This concept is termed as ”3D-language scene understanding” and contains
various downstream tasks: 3D Question Answering [13], requires models to
answer questions based on 3D scenes; 3D Visual Grounding [9] employs models to
localize target objects in 3D scenes according to language queries. While
traditional methods which excel in specific tasks, but these models often lack
adaptability to other tasks. In contrast, 3D Multi-modal Large language models
(3D-LLMs) can utilize an unified architecture to perform 3D scene
understanding covering 3D VQA and 3D grounding, owing to the reasoning and
understanding capabilities inherent in LLMs.

###  2.2 3D Multi-modal Large Language Models

Inspired by remarkable capabilities of LLMs, researchers develop Multi-modal
LLMs (MLLMs) by leveraging these models for multi-modal tasks. 3D-LLM [4] can
take 3D points with features and language prompts as input, and perform a
variety of 3D-related tasks. ImageBind-LLM [14] align 3D point clouds with
other modalities within a joint embedding space. Chat-3D [6] introduce three-
stage training scheme to enable the model to transition from learning
individual object attributes. However, it is inefficient to train 3D-LLMs with
multiple stages and 3D-LLMs may not simultaneously grasp global scene
information and fine-grained object information. Our 3DMIT removes the
alignment stage of 3D and text, which can make training 3D-LLMs is efficient
and effective, with better integration of vision-text information.

##  3 Dataset

Due to the scarcity of high-quality 3D scene-language data, we collect and
construct a comprehensive 3D scene-language instruction dataset with GPT4-API.
The dataset covers 1514 3D scenes of Scannet and encompasses 75K 3D scene-
language pairs for 3D VQA, 3D Scene Captioning, 3D Grounding, and 3D
Conversation task. Table 1 presents a detailed breakdown of the dataset,
delineating the quantity of data for each respective task.

Table 1: The dataset we bulid to train our LLM

Tasks | ScanNet
---|---
VQA | 25563
Scene-level Captioning | 562
Conversations | -
Grounding | 36665
Multiple-choice | 11895
All | 75K

###  3.1 3D VQA, Captioning and Conversations

Based on ScanQA [13] and ScanRefer [9], we collect and construct $Instruction-
Answer$ pairs with GPT-API. We utilize the caption of each object and generate
corresponding questions. We use GPT to generate choices for multiple-choice
question answering problem on 3D scene dataset. Furthermore, sequential
questions and answers for individual scenes were generated to facilitate 3D
conversations task.

Fig. 1: Overview Architecture of our method. The 3D input covers 3D scene and
segmented objects. Our 3D perceiver contains frozen 3D scene encoder and 3D
object encoder, trainable attributes encoder and projectors. We construct 3D
vision-language prompt with system message, 3D features and task instructions.
Leveraging the 3D prompt, we can fine-tune LLM efficiently by Lora to solve 3D
tasks.

###  3.2 3D Grounding

We construct our grounding dataset with ground truth bounding box of each
object and its description in ScanRefer [9]. We use GPT to generate
corresponding instructions with the prompt of “$\ object-description$ and
locate its position with the coordinate of center x, y, z and its length,
width and height.”, answers with “$\ object-id$ and its bounding box”.

##  4 Method

###  4.1 Overview

Considering the time-consuming alignment stage of 3D-text in Chat-3D [6] and
3D-LLM [4], we explore an efficient instruction tuning method to enhance the
perception and comprehension of 3D scenes by LLM. Without alignment stage, we
propose an end-to-end fine-tuning to inject 3D information into LLM. Our
overview architecture is illustrated in Figure 1.

###  4.2 3D Perceiver

The architecture comprises a 3D perceiver designed to extract essential 3D
visual features for LLMs understanding the 3D scenes. To capture both global
scene information and objects detail, the 3D perceiver is consists of several
key components: frozen 3D Scene Encoder $Es$ and 3D Object Encoder $Eo$,
trainable Attributes Encoder $Ea$ and Projectors for each part. We use EPCL
[10] as our 3D Scene Encoder $Es$ to extract the scene features $fs$
representing global scene information. For fine-grained object information, we
segment the point clouds of each object in one scene and get [coordinate,
size, color] as attributes of each object point cloud. We use the pre-trained
3D encoder [11, 12] as our 3D Object Encoder $Eo$ to extract object features
$fo$. And we follow LLaMA [1] to use position embedding layers.We utilize some
Linear Layers as our Attributes Encoder $Ea$ to encode the attributes of each
object as $fa$. In detail, while position embedding layers encoding the
coordinate as $fp$, and linear layers encoding the size $fq$ and color $fc$,
we have $fa$ = concentrate[$fp$+$fq$+$fc$]. And we use $Ps$ to project $fs$ as
$fs^{\prime}$, $Po$ to project the features of each object $fo$ as
$fo^{\prime}$. $fs^{\prime}$ and $fo^{\prime}$ are normalized and have the
same dimension. In addition, the scene features $fs^{\prime}$ and the list of
all the objects features
[$fo^{\prime}_{1},fo^{\prime}_{2},...,fo^{\prime}_{n}$] are projected to the
dimension that LLM can receive by LLM Projector $P_{L}$ respectively. And we
represent them as $Fs$ and [$Fo_{1},Fo_{2},...,Fo_{n}$]

####  4.2.1 3D Scene Encoder

We use EPCL [10] as our pre-trained 3D Scene Encoder which directly leverage
the frozen CLIP[15] transformer as the encoder for point cloud tasks. The EPCL
builds a bridge between 3D modality and 2D modality by using CLIP model which
is trained on billions of image-text pairs. We consider that EPCL can extract
3D scene features, which contains the alignment between the image and language
domain.

####  4.2.2 3D Object Encoder

We use Ulip2 [11] and Uni3D [12] respectively as our 3D object encoder. Ulip2
is a pre-trained 3D object encoder trained with 800K 3D point clouds and
millions of image-text pairs. Uni3D is an end-to-end pre-trained scalable 3D
object encoder which aligns the 3D point cloud features with the image-text
aligned features. Because of our method without the alignment stage , we can
get the 3D features which latently align with text by using Ulip2 and Uni3D.

Table 2: Evaluation results of 3D VQA on ScanQA validation dataset. Method | Data | Alignment | EM | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr
---|---|---|---|---|---|---|---|---
VoteNet [16]+MCAN [17] | - | - | 17.3 | 28.0 | 6.2 | 11.4 | 29.8 | 54.7
ScanRefer [9]+MCAN [17] | - | - | 18.6 | 26.9 | 7.9 | 11.5 | 30.0 | 55.4
Chat-3D [6] | - | $\checkmark$ | - | 29.1 | 6.4 | 11.9 | 28.5 | 53.2
3D-LLM (flamingo) | - | $\checkmark$ | 20.4 | 30.3 | 7.2 | 12.2 | 32.3 | 59.2
3D-LLM (BLIP2-flant5) | - | $\checkmark$ | 20.5 | 39.3 | 12.0 | 14.5 | 35.7 | 69.4
LLaVA [18](zero-shot) | - | $\usym{2717}$ | 0.0 | 7.1 | 0.3 | 10.5 | 12.3 | 5.7
LAMM [5] | 25k | $\usym{2717}$ | 9.82 | 26.77 | 5.78 | 9.95 | 23.64 | 42.41
3DMIT(Vicuna-7b) | 25k | $\usym{2717}$ | 10.10 | 27.86 | 6.44 | 10.40 | 24.64 | 44.38
3DMIT(LLaVA1.5)+IMG | 25k | $\usym{2717}$ | 9.62 | 26.93 | 5.98 | 10.64 | 24.46 | 46.42
3DMIT(Vicuna-7b) | 75K | $\usym{2717}$ | 13.04 | 27.63 | 5.24 | 10.70 | 26.22 | 48.03

###  4.3 Instruction Tuning

####  4.3.1 Prompt

After we get the 3D features contains 3D scene features $Fs$ and the list of
all the objects features [$Fo_{1},Fo_{2},...,Fo_{n}$], we combine 3D features
and text as our vision-language prompt.

For different tasks, we utilize different system messages with 3D features. We
complete the prompt with the instructions of different tasks. Here is an
example and the intact and various prompts are in the supplement.

> $\\#\\#\\#<$System message$>$: You are an AI visual assistant can analyze
> point clouds of the whole scene and objects in scene…$\\#\\#\\#$ The whole
> scene information: $Fs$. The information of all the objects in the scene:
> [$Fo_{1},Fo_{2},...,Fo_{n}$]. $<Question>$: …

####  4.3.2 Pre-trained LLM and Fine-tune

We use Vicuna-7b [2] and LLaVA1.5-7b [7] as our basic LLM and MLLM
respectively. Vicuna-7b is a LLM, which is fine-tuned from LLaMA [1].
LLaVA1.5-7b is a MLLM which is fine-tuned with instructional vision-language
data.

We use Lora [19] to fine-tune our basic LLM and MLLM with 3D scene-language
instruction pairs.

##  5 Experiments

###  5.1 Implementation Details

We leverage pre-trained 3D scene encoder EPCL based on the ViT-L/14 of CLIP.
We use pre-trained 3D object encoder Ulip2 based on the ViT-B/16 of CLIP and
Uni3D based ViT-L/14 of CLIP. For encoding object attributes, we employ a
position embedding layer inspired by LLaMA for coordinating point clouds,
while linear layers encode the size and color attributes of point clouds. The
architectural components of our model include linear layers for both the scene
projector and LLM projector, 3D object projector comprises a normalization
layer and a three-layer MLP. To optimize the training process, we adopt the
bp16 Mixed-precision Training Strategy facilitated by DeepSpeed, conducting
training for 2 epochs. We set train batch size as 256, train micro batch size
per gpu as 1, and gradient accumulation steps as 32. We set lora_r as 32,
lora_alpha as 32, lora_dropout as 0.1 and learning rate as 5e-4. The entire
fine-tuning process requires approximately 7 hours on 8 NVIDIA A100 GPUs.

###  5.2 Evaluation on 3D VQA

The training datasets covers 75K scene-language instruction pairs contraining
25K 3D VQA data. We choose Vicuna-7b and LLaVA1.5-7b as our basic LLM and
MLLM. We evaluate our method on ScanQA [13] validation dataset, which employs
VoteNet [16] to generate object proposals, integrating them with text
embeddings.

####  5.2.1 Baselines

The compared baselines contain 3 categories: supervised expert models, general
3D-LLMs with alignment and 3D-LLMs without alignment.

  * •

Supervised Expert Models: ScanRefer [9]+MCAN [17] and VoteNet [16]+MCAN[17]
leverage MCAN to detect 3D objects and incorporate them into a standard VQA
model.

  * •

General 3D-LLMs with alignment: 3DLLM [4] algins 3D scene, 2D images with text
features, and employs 2D VLMs as backbone, projecting 3D features into the
VLM’s input space. Chat-3D [6] aligns 3D objects features with text but is
limited to object-centric tasks.

  * •

3D-LLMs without alignment: LLaVA [18] is a MLLLM that connects a vision
encoder and LLM for general purpose visual and language understanding. LAMM,
based on Vicuna-7b, introduces the LAMM-Dataset and framework for scene
understanding without the alignment stage of 3D point cloud and text.

####  5.2.2 Evaluation Metrics

We report ROUGE-L, METEOR, CIDEr for robust answer matching. We also use exact
match (EM) metric, BLEU-1 and BLEU-4 metrics.

####  5.2.3 Result Analysis

As the results shown in Table 2, we compare our method with expert models,
general 3D-LLMs with alignment and 3D-LLMs without alignment. Notably, when
utilizing a limited training dataset of only 25k data, our method based on
Vicuna-7b, surpasses 3D-LLMs without alignment, such as LAMM and LLaVA (zero
shot), across all evaluated metrics. As the training dataset expands to 75K
instances encompassing various tasks, our method 3DMIT(Vicuna-7b) outperforms
3DMIT(Vicuna-7b) with 25K data on EM, METEOR, ROUGE and CIDEr metrics, which
suggests that our method is more diverse and creative with more data. The
performance on BLEU-1 and BLEU-4 metrics shows a marginal decrease, indicating
that our method remains competitive and robust.

Furthermore, our method achieves results comparable to expert models and
3D-LLMs with alignment, such as Chat-3D and 3D-LLM (flamingo). Notably, it
outperforms VoteNet+MCAN on BLEU-4, as well as ScanRefer+MCAN and Chat-3D on
BLEU-1. We also find that the results of 3DMIT (LLaVA1.5) with multi-view
image tokens closed to the performance of 3DMIT (Vicuna-7b), which proves that
our method has promising transferability across different LLMs and MLLMs. In
addition, the ablation study of introducing multi-view image tokens will be
discussed in Section 5.4. We also evaluate our method on the validation
dataset of multiple-choice in Supplement.

In summary, our results consistently outperform 3D-LLMs without alignment and
closely approach the performance of 3D-LLMs with alignment. This substantiates
the effectiveness of our proposed method in achieving competitive results
across diverse evaluation metrics.

Table 3: Evaluation results of 3D grounding on Scanrefer validation dataset.

Method | Alignment |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
ScanRefer | - | 37.3 | 24.3
3D-LLM (flamingo) | $\checkmark$ | 21.2 | -
3D-LLM (BLIP2) | $\checkmark$ | 30.3 | -
LLM-Grounder [20] | $\usym{2717}$ | 17.1 | 5.3
LAMM | $\usym{2717}$ | - | 3.38
3DMIT(Vicuna-7b) | $\usym{2717}$ | 10.2 | 7.07
3DMIT(LLaVA1.5) | $\usym{2717}$ | 10.7 | 7.2

Table 4: Ablation study of the scene of the multi-view image tokens. Method | Model | BLEU-1 | CIDEr
---|---|---|---
w/o IMG | 3DMIT(Vicuna-7b) | 27.86 | 44.38
w IMG | 3DMIT(Vicuna-7b) | 24.78 | 40.23
w/o IMG | 3DMIT(LLaVA1.5) | 24.08 | 37.45
w IMG | 3DMIT(LLaVA1.5) | 26.93 | 46.42

###  5.3 Evaluation on 3D Visual Grounding

In the evaluation based on the ScanRefer validation dataset, the model is
given the object description. The model should understand the location and
object description. Subsequently, it is required to provide the object
identifier along with its corresponding 3D bounding box.

####  5.3.1 Baselines

  * •

ScanRefer [9] employs a pretrained VoteNet backbone integrated with a GRU to
select the matching bounding box, thereby demonstrating expertise in 3D scene
understanding.

  * •

General 3D-LLMs with alignment: 3D-LLM [4] predicts bounding boxes as location
tokens added to the LLM’s vocabularies. These bounding boxes are learned from
scratch, showcasing an alignment strategy to facilitate enhanced
understanding.

  * •

3D-LLMs without alignment: LLM-Grounder [20] achieves zero-shot 3D visual
grounding by utilizing an open-vocabulary visual grounding tool and using GPT4
as the language assistant. LAMM [5] presents the LAMM-Dataset and framework
for scene understanding without alignment stage of 3D point cloud and text.

####  5.3.2 Valuation Metrics

We measure 3D grounding capability by<EMAIL_ADDRESS>and @0.5, which are
accuracies of bounding box predictions whose Intersection-over-Union (IoU)
w.r.t. ground-truth box exceeds 0.25 and 0.5, respectively.

####  5.3.3 Result Analysis

As the results shown in Table 3, our comparative analysis contains our method
against expert models, general 3D-LLMs with the alignment of 3D modality and
text, and 3D-LLMs without alignment.

An insightful observation is that our method, based on Vicuna-7b and LLaVA1.5,
outperforms 3D-LLMs without alignment, such as LLM-Grounder and LAMM in terms
of<EMAIL_ADDRESS>It is proved the effectiveness of our method with answering the
object identifiers and its bounding box. And we find that LLMs face challenges
in numerical and computational tasks. Considering expert models and 3D-LLMs
with alignment outperform our method, our method has limitations in
comprehending spatial location details without alignment.

However, through our experiment, we can inspire researchers to consider how to
train 3D-LLMs understand location information easily and effectively.

Table 5: Ablation study of different pre-trained 3D object encoders. Method | Model | ROUGE | CIDEr
---|---|---|---
3DMIT(Vicuna-7b) | Ulip2 | 27.63 | 48.03
3DMIT(Vicuna-7b) | Uni3D | 25.38 | 47.04
3DMIT(LLaVA1.5) | Ulip2 | 21.23 | 37.45
3DMIT(LLaVA1.5) | Uni3D | 22.38 | 36.68

###  5.4 Ablation Study

We conduct an ablation study to discuss the effectiveness of multi-view image
tokens of the scene. To evaluate the effectiveness of 3DMIT with the multi-
view image of the scene, we train 3DMIT with multi-view tokens based on
Vicuna-7b and LLaVA1.5-7b. The results are presented in Table 4. We find that
if introducing image tokens to 3DMIT (Vicuna-7b), the performance decreases
because Vicuna-7b is language model which can not understand the 3D
information and 2D image simultaneously. By contrast, LLaVA1.5 connects the
visual encoder of CLIP with LLaMA, and is fine-tuned end-to-end on
instructional vision-language data. Therefore, the performance increases by
introducing image tokens to 3DMIT based on LLaVA1.5-7b. We conclude that if
based on MLLM such as LLaVA, introducing multi-view can improve the
performance of 3DMIT.

Furthermore, we assess the effectiveness of the pre-trained 3D object encoder
chosen for our model. The result is shown in Table 5. We compare our method
utilizing Ulip2 or Uni3D as our 3D object encoder, the results are closed.
Because both Ulip2 and Uni3D are based on CLIP for 3D-2D-text alignment,
differing only in the dimensions of extracted features. Consequently, the pre-
trained 3D object encoder used in Table 2 and Table 3 is Ulip2 without
specific declaration.

##  6 Conclusion

This paper introduces 3DMIT, an innovative and efficient 3D multi-modal
instruction tuning framework designed to train LLMs in comprehending the
intricacies of 3D world. Distinguishing itself from other 3D-LLMs, 3DMIT
eliminates the alignment stage between 3D scenes and language, prioritizing
efficiency. The framework leverages both global scene information and fine-
grained object details to enhance the overall scene understanding capabilities
of LLMs. We construct a comprehensive 3D scene-language instruction dataset
with GPT-API, which encompasses 75K 3D scene-language pairs for 3D tasks. We
evaluate our method on ScanQA for 3D VQA and ScanRefer for visual grounding.
Our experimental results demonstrate the effectiveness of the proposed method.
We believe our insights and methodologies will inspire researchers to further
explore and refine the integration of multi-modal information into LLMs
directly and efficiently.

## References

  * [1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [2] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez _et al._ , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * [3] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [4] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan, “3d-llm: Injecting the 3d world into large language models,” _arXiv preprint arXiv:2307.12981_ , 2023.
  * [5] Z. Yin, J. Wang, J. Cao, Z. Shi, D. Liu, M. Li, L. Sheng, L. Bai, X. Huang, Z. Wang _et al._ , “Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark,” _arXiv preprint arXiv:2306.06687_ , 2023.
  * [6] Z. Wang, H. Huang, Y. Zhao, Z. Zhang, and Z. Zhao, “Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes,” _arXiv preprint arXiv:2308.08769_ , 2023.
  * [7] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” _arXiv preprint arXiv:2310.03744_ , 2023.
  * [8] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5828–5839.
  * [9] D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object localization in rgb-d scans using natural language,” in _European conference on computer vision_. Springer, 2020, pp. 202–221.
  * [10] X. Huang, S. Li, W. Qu, T. He, Y. Zuo, and W. Ouyang, “Frozen clip model is efficient point cloud backbone,” _arXiv preprint arXiv:2212.04098_ , 2022.
  * [11] L. Xue, N. Yu, S. Zhang, J. Li, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip-2: Towards scalable multimodal pre-training for 3d understanding,” _arXiv preprint arXiv:2305.08275_ , 2023.
  * [12] J. Zhou, J. Wang, B. Ma, Y.-S. Liu, T. Huang, and X. Wang, “Uni3d: Exploring unified 3d representation at scale,” _arXiv preprint arXiv:2310.06773_ , 2023.
  * [13] D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe, “Scanqa: 3d question answering for spatial scene understanding,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 19 129–19 139.
  * [14] J. Han, R. Zhang, W. Shao, P. Gao, P. Xu, H. Xiao, K. Zhang, C. Liu, S. Wen, Z. Guo _et al._ , “Imagebind-llm: Multi-modality instruction tuning,” _arXiv preprint arXiv:2309.03905_ , 2023.
  * [15] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [16] Z. Ding, X. Han, and M. Niethammer, “Votenet: A deep learning label fusion method for multi-atlas segmentation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III 22_. Springer, 2019, pp. 202–210.
  * [17] Z. Yu, J. Yu, Y. Cui, D. Tao, and Q. Tian, “Deep modular co-attention networks for visual question answering,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6281–6290.
  * [18] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [20] J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai, “Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,” _arXiv preprint arXiv:2309.12311_ , 2023.

Supplemental Material

In this supplemental material, we report more experiments about our 3DMIT,
including case study of our method, results of more experiments and more
visual prompt examples.

##  Appendix A Case Study of our method

Figure 2 shows the case study of 3D conversation and 3D caption task. Figure 3
shows the case study of 3D conversation and 3D VQA task.

Fig. 2: The case study of 3D conversation and 3D caption task. Fig. 3: The
case study of 3D conversation and 3D VQA task.

##  Appendix B Evaluation results of 3D Multiple-choice

We construct the training dataset and validation dataset based on ScanQA
dataset, which employs VoteNet to generate object proposals, integrating them
with text embeddings. We use GPT to generate choices for multiple-choice
question answering problem on 3D scene dataset. The result is presented in
Table 6. We utilize the accuracy of multiple-choice to evaluate our results,
which is represented as Acc. We can find that the Acc of all of our method is
over 90%. It can prove that the effectiveness of our method in 3D VQA and its
corresponding multiple-choice format Question-Answer task. Furthermore, with
the results and conclusions in the paper, we can prove our method 3DMIT
provide a promising ability of understanding of the 3D world.

Table 6: Evaluation results of 3D Multiple-choice on ScanQA validation dataset Method | Alignment | Object Encoder | Acc
---|---|---|---
3DMIT(Vicuna-7b) | $\usym{2717}$ | Ulip2 | 91.17
3DMIT(Vicuna-7b) | $\usym{2717}$ | Uni3D | 90.68
3DMIT(LLaVA1.5) | $\usym{2717}$ | Uni3D | 93.53

##  Appendix C visual prompt examples

For different 3D tasks, we construct different 3D prompt with various system
message and task instructions. Figure 4 shows the detailed 3D vision-language
prompt for VQA task, and Figure 5 shows the detailed 3D vision-language prompt
for 3D visual grounding task.

Fig. 4: The 3D vision-language prompt for VQA with system message, 3D features
and task instructions.  Fig. 5: The 3D vision-language prompt for Visual
grounding with system message, 3D features and task instructions.