# Trustworthiness of Children Stories Generated by Large Language Models

Prabin Bhandari
Department of Computer Science
George Mason University
<EMAIL_ADDRESS>
&Hannah Marie Brennan
Department of English, Linguistics Program
George Mason University
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) have shown a tremendous capacity for generating
literary text. However, their effectiveness in generating children’s stories
has yet to be thoroughly examined. In this study, we evaluate the
trustworthiness of children’s stories generated by LLMs using various
measures, and we compare and contrast our results with both old and new
children’s stories to better assess their significance. Our findings suggest
that LLMs still struggle to generate children’s stories at the level of
quality and nuance found in actual stories.111Code and dataset are publicly
available: https://github.com/prabin525/trustworthiness-of-children-stories-
generated-by-LLMs

##  1 Introduction

Advancements in pretrained large language models (LLMs) like GPT-3 Brown et
al. (2020) and LLaMA Touvron et al. (2023), have made it easier to generate
natural language text for a variety of downstream tasks, including generating
narrative text like children’s stories. The ability to generate natural text
using LLMs has seen substantial improvement with the innovation of
instruction-following models like InstructGPT Ouyang et al. (2022) and Alpaca
Taori et al. (2023), resulting in a better alignment with user intentions.

These systems are being used as a general-purpose chat-bots by the general
public. As these models are integrated more into everyday applications, it is
crucial to continuously evaluate LLMs’ performance to ensure that they are
indeed trustworthy and accurate.

Trustworthiness in the case of LLMs is a broad term that refers to reliability
and confidence in the generated text outputs along with their suitability for
a specific downstream task. A trustworthy LLM minimizes errors, biases, and
potentially harmful content while consistently producing clear and
contextually suitable text. With the advancing capabilities of LLMs, concerns
regarding their trustworthiness have arisen. Notably, they are being used more
frequently to support creative writing Clark et al. (2018), raising concerns
about the generation of inappropriate or offensive text Price (2016) and
biased content Lucy and Bamman (2021a). One domain in which trustworthiness is
of particular importance is text generation intended for children. This paper
seeks to evaluate the trustworthiness of children’s stories generated by LLMs
including generative LLMs and instruction following models. In the case of
text generation geared towards children, LLMs’ ability to generate age-
appropriate materials to target audiences also becomes a vital aspect of
overall trustworthiness.

To assess the trustworthiness of LLMs in generating children’s stories, we use
two open-source foundation language models, OPT Zhang et al. (2022) and LLaMA
Touvron et al. (2023), along with an instruction-following model Alpaca Taori
et al. (2023) to generate children’s stories. Then, we compare these generated
stories against actual children’s stories, old and modern. Our assessment
takes into account a number of aspects, including statistics derived from the
text like the Flesch reading ease score Flesch (1948), toxicity present in the
text, the most influential topics present in the text, and the sentence
structure of these texts.

Our findings reveal that LLMs lack a high level of trustworthiness when tasked
with generating children’s stories. While the generated children’s stories do
share similarities in topics and patterns with the actual stories (mostly
modern ones), they are also susceptible to generating toxic content. Moreover,
LLMs struggle to capture the intricacies and nuances of children’s literature,
evident from the disparity in sentence structure between the generated and
actual stories.

##  2 Related Work

###  2.1 Story Generation

Recently, LLMs have been increasingly used to supplement creative writing
efforts for entertainment and social media. Applications include work related
to narrative generations Sun et al. (2023); Simon and Muise (2022);
Razumovskaia et al. (2022); Xiang et al. (2018). Yuan et al. (2022) tested
Wordcraft, a tool created to assist writers with story generation using LLMs.
In their study, writers who were tasked with working with the AI agent noted
that Wordcraft lacked content awareness and would create grammatical stories
with nonsensical topics or plots.

###  2.2 Children and AI

AI and LLMs have also been applied to contexts involving children. Researchers
at MIT had children work with social robots to evaluate how much the children
could learn through activities involving robots Williams (2019). There is much
discussion on how to integrate AI into early childhood education Yang (2022);
Kasneci et al. (2023). With the increasing use of AI by and around children,
there is an urgent need for more thorough evaluations of LLMs and the
appropriateness of generated content for vulnerable audiences.

###  2.3 Trustworthiness Testing

Chiang and Lee (2023) investigated whether LLMs can replace humans in
evaluating texts. Specifically, they looked at open-ended story generation and
adversarial attacks. They found that there were similar ratings between LLMs
and human evaluators. Venkit et al. (2023) found that unbalanced sources of
training data result in biased generations in GPT-2, and proposed strategies
to reduce bias using adversarial triggers. Tang et al. (2022) presented
EtriCA, a neural generation model which aims to remedy issues of relevance and
coherence of generated texts. Lucy and Bamman (2021b) studied the bias
existing in GPT-3’s generated stories. Guo et al. (2023) have proposed a
similar study specifically testing how similar text generated by ChatGPT is to
text produced by human writers.

##  3 Methodology

To investigate the trustworthiness of children’s stories generated by LLMs, we
compare them with actual old and modern children’s stories. We collect a
diverse set of stories from different sources, including both older stories
such as folktales, and more recent children’s stories. We use both LLMs and
instruction-following models to generate stories with different prompt lengths
and instruction templates. As story generation is an open-ended problem with
no reference text, we rely on other metrics instead of any automatic measure
of evaluation like BARTScore Yuan et al. (2021) or BERTScore Zhang et al.
(2019). We use various metrics to compare the generated stories with actual
stories, including in-text statistics such as sentence length and a measure of
toxicity in the text, as well as an evaluation of topics covered in these
stories. Furthermore, we analyze and compare the grammatical structures of the
stories using dependency structures extracted from both the original and the
artificially generated stories.

In the following section, we describe the experimental setup, including
details on the collected data, the story generation process, and the
evaluation metrics used for comparison. Subsequently, we present the results
obtained from our experimentation.

##  4 Experiments

###  4.1 Data

Our data consists of 132 original children’s stories collected from various
online sources and categorized into two categories: old and modern. The old
stories generally include traditional children’s stories like folktales and
fairy tales, whereas the modern stories include more recent children’s
literature published after the year 2000. Both sets of original children’s
stories are comprised of English texts aimed at children between the ages of
three and thirteen, with both data sets representing the full range of these
target ages. Overall, 122 are classified as old stories, and the remaining 10
as modern stories. Specifically, the older stories were obtained via Project
Gutenberg,222https://www.gutenberg.org/ and the modern stories from various
online platforms.333https://www.freechildrenstories.com/,
https://monkeypen.com/ We use the old stories as a reference for the story
generation task and compare the generated stories against both old and modern
stories.

###  4.2 Story generation

We generate stories using language models and an instruction-following model.

#### Language Models

Our story generation task using LLMs uses two foundational language models:
OPT Zhang et al. (2022) and LLaMA Touvron et al. (2023), with model sizes of
6.7 billion and 7 billion parameters, respectively. To generate stories, we
provide a portion of each old story as context for the LLMs. Specifically, we
use the first sentence, the first 256 tokens, and the first 512 tokens of each
old story as a prompt. We use top-$k$ sampling-based decoding with $k$ set to
100 and generate five samples for each prompt, resulting in a total of 3660
generated stories. The breakdown of the generated stories along with the
length of the prompt is given in Table 1.

Model | Prompt Length | Count
---|---|---
OPT | First Sentence (OPT-Line) | 610
| First 256-tokens (OPT-256) | 610
| First 512-tokens (OPT-512) | 610
LLaMA | First Sentence (LLaMA-Line) | 610
| First 256-tokens (LLaMA-256) | 610
| First 512-tokens (LLaMA-512) | 610
| Total | 3660
Table 1: Breakdown of the stories generated using LLMs.

#### Instruction-following Models

For instruction-following story generation, we use Alpaca Taori et al. (2023),
which is an instruction-following model that is based on the LLaMA
architecture and is fine-tuned using self-instruct Wang et al. (2022). We use
the Alpaca model based on the 7B variant of the LLaMA model. We use four
different instruction templates to generate stories, two of which require a
story title as input and two of which do not. For the templates that require a
story title, we use the title of old stories as input. The templates are
provided in Table 2. To generate stories, we use top-$k$ sampling-based
decoding with $k$ set to 100 and generate five samples for each template,
resulting in a total of 2440 generated stories with 610 stories per template.

S.N. | Template
---|---
T1 | Below is an instruction that describes a task, paired with an input that provides further context.
| Write a response that appropriately completes the request.
| ### Instruction:
| Write a short children’s story given the title.
| ### Input:
| TITLE
| ### Response:
T2 | Below is an instruction that describes a task. Write a response that appropriately completes the request.
| ### Instruction:
| Write a short children’s story.
| ### Response:
T3 | Below is an instruction that describes a task, paired with an input that provides further context.
| Write a response that appropriately completes the request.
| ### Instruction:
| Write a children’s story given the title.
| ### Input:
| TITLE
| ### Response:
T4 | Below is an instruction that describes a task. Write a response that appropriately completes the request.
| ### Instruction:
| Write a children’s story.
| ### Response:
Table 2: Templates used by Alpaca for story generation.

###  4.3 In-text statistics

We compare various statistics derived from the text of the generated stories
against those of actual stories. Specifically, we use two metrics: sentence
length and Flesch reading ease score Flesch (1948).

#### Flesch Reading Ease Score

The Flesch reading ease score (FRES) measures the readability of a text and is
based on two factors: average sentence length and the average number of
syllables per word. It provides a score between 0 and 100, with higher scores
indicating easier readability. A Flesch reading ease score above 60 for a text
indicates that it can easily be read by children up to the age of 15. The
formula for calculating the FRES of a text is shown in Equation 1.

$FRES=206.835-1.015\left(\frac{total\;words}{total\;sentence}\right)\\\
-84.6\left(\frac{total\;syllables}{total\;words}\right)$ (1)

###  4.4 Toxicity of text

Gehman et al. 2020 found that the LLMs can generate ‘toxic’ text from a very
innocuous prompt and attribute this to a significant amount of offensive,
factually unreliable, and otherwise toxic content in the training data of
these models. We want to investigate the level of toxicity in our generated
children’s stories. Ideally, generated children’s stories should be free of
any toxic text.

We use Detoxify Hanu and Unitary team (2020), a BERT Devlin et al. (2019)
based toxic text detector, to identify the presence of toxic text in the
generated children’s stories. Detoxify generates score labels in the range of
0 to 1, assessing the toxicity of the text based on categories such as toxic,
severely toxic, obscene, threat, insult, and identity hate. Specifically, we
use detoxify for each sentence of our actual and generated stories to get
toxicity measures across the six categories.

###  4.5 Topic Modeling

We also analyze the data for topic modeling using pyLDAvis Tran (2022). We
compare the topics found in the data set of older stories with the LLM-
generated stories. The older stories and the modern stories are also compared
to assess whether there has been a shift in topics over time that would
potentially influence topic properties in the LLM-generated stories. A
probable diachronic shift in topics of stories geared towards young audiences
also highlights the need to test the toxicity of generated stories, as seen in
the previous section.

To avoid uninformative topics, the data is preprocessed to remove stopwords
and names. All texts are categorized for specific topics using word clustering
for a set of documents. Modeling is performed automatically without a
predefined list of labels. The visualizations using pyLDAvis break down the
topics based on the 122 older stories, the 10 modern stories, and the
generated stories from OPT, LLaMA, and Alpaca.

###  4.6 Sentence structure

The structure of the sentences within a text can reveal the type or genre of
the text. To analyze sentence structures, we construct a dependency tree for
each sentence in both the original and generated children’s stories. The
dependency tree depicts the syntactic dependencies between the words in a
sentence, effectively capturing the grammatical structure of the sentence. We
then convert these dependencies into unlabeled directed graphs, preserving
sentence structure while removing specific words. We then generate the
Weisfeiler Lehman graph hash Shervashidze et al. (2011) for each graph. The
Weisfeiler Lehman hashes are identical for isomorphic graphs and strongly
guarantee that non-isomorphic graphs will get different hashes. We compare the
frequency of hashes to evaluate the similarity between the sentence structure
of the generated stories and the actual stories.

##  5 Generated stories follow modern trends but struggle with nuances

Figure 1 shows the box plot of sentence lengths for old and modern original
stories, as well as for the generated stories. Being literary texts,
children’s stories do not strictly confine to formal English conventions and
many contain sentences with higher word counts; so for clarity, we removed all
the outliers from the plot. One interesting observation is that modern
children’s stories generally have shorter sentence lengths than older
children’s stories, which adheres to previous research that shows a trend of
decreasing sentence length in print Haussamen (1994). The generated stories
from OPT and LLaMA show an increase in sentence length as the prompt length
increases. We hypothesize that these models learn the pattern of larger
sentence length from the older stories used as context, which is then
reflected in the generated text. However, stories generated using the
instruction-following model Alpaca, have sentence lengths similar to modern
actual stories, indicating that language models may have been trained mostly
on the newer text, and tend to generalize modern trends when instructed to
generate text of a specific type.

Figure 1: Comparison of sentence length in generated children’s stories and
actual children’s stories. The generated children’s stories exhibit shorter
sentence lengths compared to the older original stories but are similar in
sentence length to modern stories. Language models prompted with older stories
tend to generate longer sentences following the patterns of the context that
had been provided.

The Flesch reading ease score is a statistical measure of the readability of a
text and was optimized to be general enough at the time of its formulation, as
can be seen with the constant values in equation 1. That is why, we may find
FRES values not within the range of 0 to 100 as seen in Figure 2(a). We also
removed the outliers from the box plot in Figure 2(a). Since we are not
interested in exact values but in the general trend these values represent, we
use the FRES values in the range of 0-100 and show their box plot in Figure
2(b).

Our results from the Flesch reading ease score reveal several interesting
observations. Firstly, we see that modern children’s stories have a higher
FRES than older stories, meaning that the modern ones are easier to read. This
can be attributed to the fact that sentences are getting shorter and might
have to do with simpler word selection. Secondly, we see that LLMs prompted
with older stories tend to follow the pattern of the context and generate
stories that are more difficult to read, as the context length increases.
Finally, we see that the instruction-following model Alpaca generates stories
that are easier to read compared to older original children’s stories but are
not as readable as modern children’s stories. We posit that this observation
can be attributed to the fact that LLMs used in our study are generic models,
and the instruction following model is also only fine-tuned for general
instructions rather than instructions specific to children’s story generation.

((a)) FRES on all data (over 100 is undefined).

((b)) FRES limited to well-defined 0-100 range.

Figure 2: Comparison of FRES in generated children’s stories and actual
children’s stories : (a) FRES with all data and (b) FRES only in the range of
0 and 100. The generated children’s stories are easier to read compared to
older actual stories but are not as easy as modern original stories. Language
models prompted with older stories tend to generate text that is more
difficult to read, likely because they follow the patterns in the prompts.

((a)) toxic

((b)) severe toxic

((c)) obscene

((d)) threat

((e)) insult

((f)) identity hate

Figure 3: Various toxicity measures for the actual and generated stories. Each
cell in a subplot represents the percentage of sentences rated on a toxicity
scale, with x-axis values indicating the toxicity level. Values for ratings in
the range of 0-0.1 have been omitted from the plots for clarity.

Overall, we see that modern children’s stories are easier to read than older
children’s stories. As most of the training data for LLMs comes from newer
text, the model tends to follow the trend of modern children’s stories in
their generated text for sentence length and word selection. However, it
should be noted that these models are not fine-tuned for children’s stories
generation, and therefore may not capture the nuances of children’s stories
resulting in stories that might be difficult to read for intended readers.

##  6 Generated stories may contain toxic text

Our analysis of toxicity in actual and generated stories reveals several
noteworthy findings. We present the toxicity measures for both actual and
generated stories in Figure 3. Notably, we find that older stories tend to be
more toxic than modern stories across all toxicity measures. This trend is not
solely due to the smaller sample size of modern actual stories, as we have
normalized the toxicity ratings to ensure an accurate comparison. Rather, it
suggests that writers are becoming more mindful of the language they use in
children’s literature. Although modern stories are less toxic compared to
older stories, we still observe some level of toxicity in them. This toxicity
in modern actual stories is often related to the narrative of the story. For
example, threats and insults might be needed for some stories, but identity
hate is not appropriate for children’s stories. It is noteworthy that modern
stories do not have toxic text related to identity hate but older stories do.

Similar to our previous observation, we see that LLMs tend to learn patterns
from the context they are provided with. As evident from the stories generated
by OPT and LLaMA, we see that the toxicity aligns with older stories and
gradually increases with an increase in the length of the context. The stories
generated using the instruction-following model Alpaca tend to be less toxic
and mostly resemble modern stories. However, stories generated using the T1
and T3 templates have a lot of obscene text compared to stories generated
using T2 and T4, which have none. As shown in Table 2, T1 and T3 take the
title as input whereas T2 and T4 do not. It is possible that the model
remembered the story title and generalized the patterns of the story or
generalized to some other text in the template, leading to the generation of
obscene text. This finding is consistent with Gehman et al. 2020, who suggest
that children’s stories generated by LLMs can contain highly toxic text
despite an innocuous prompt.

Our analysis of toxicity in original and generated stories reveals that older
stories tend to be more toxic than modern ones, that LLMs can learn toxic
patterns from context leading to the generation of toxic text, and that LLMs
can even generate toxic text from a very innocuous prompt. These findings
suggest that further work is needed to make LLMs useful as tools for
generating age-appropriate children’s literature.

##  7 Generated stories share main topics with original stories

After preprocessing the data, the original stories were found to have four
major topics. All of the topics tended to share the existence of some small
character. The first topic mentions elements such as time, goodness, and
greatness, and the presence of words like head, round, night, and water likely
indicate specific scenes or settings within the narrative. The second topic
contained new elements like a prince, the color white, a girl, and eyes. These
additional keywords suggest different perspectives within the overarching
narrative. The third topic introduces elements like a house and a heart. Like
the previous topics, it shares mentions of a little character, time, goodness,
and a prince. The difference between ’house’ and ’heart’ could indicate a
change in the setting or moral of the narrative. The last topic introduces new
elements of wolf, people, eyes, and a mother. These keywords might suggest
narratives that introduce new characters and themes. Overall, these topics
provide insight into the underlying themes present in the older 122 stories in
the data set. The topics revolve around narratives involving a small
character, time, goodness, and various other elements such as princes, nights,
water, girls, and wolves.

Comparatively, the topics of the generated stories obtained from OPT, LLaMA,
and Alpaca show minor differences. The first topic suggests a narrative that
involves characters like kings, mothers, princes, and princesses. It also
mentions elements of time, goodness, greatness, and shadow. The prince,
princess, and shadow hint at the fairy tale or fantasy theme. The second topic
shares similarities with the previous topic, with a focus on little, prince,
time, goodness, and greatness, but it also introduces new elements like eyes,
houses, heads, and the color white. These additions suggest different scenes,
perhaps removed from the monarchy or castle theme, and suggest a different
narrative. The third topic seems to center around family dynamics, with
mentions of mothers, fathers, and children. It also includes keywords relating
to time, goodness, night, and poverty. This suggests a change in the narrative
away from the fantasy-focused topic. The last topic includes keywords like
little, time, and goodness. It includes elements of fathers, eyes, and houses.
The presence of ’long’ and ’night’ suggests a different tone or atmosphere
within the narrative. These general results show remarkable similarity with
the data set on which the LLMs were trained. The topics revolve around
narratives involving characters such as kings, princes, mothers, fathers, and
children. The topics also touched upon topics of time, goodness, greatness,
poverty, and setting elements of houses, nights, and the color white.

Figure 4: Comparison of topics in generated children’s stories and actual
children’s stories. The plot shows that the most shared topics (x:2, y:2)
include ’white’, ’world’, ’great’, ’water’, ’black’, ’house’, ’little’,
’king’, ’called’, and ’good’. The least shared topics (x:1, y:1) include
’heart’, ’head’, ’poor’, ’house’, ’looking’, ’children’, ’good’, ’young’,
’lady’, and ’night’.

As with toxic content testing, we ran topic modeling for a small number (10)
of modern stories in order to compare the general topics that are currently
aimed at children. The first topic includes keywords related to spatial
orientation (right, inside, door, left), objects (ream, head, frog), time, and
actions (started). The keyword ‘eyes’ may suggest a focus on visual perception
or observation. The second topic emphasizes time, objects (ream, door),
spacial orientation (right, inside), a frog, a head, fairies, and ‘need’. The
presence of fairies introduces a fantastical or imaginative element to the
topic. The third topic revolves around time, spacial orientation (right,
inside, door), physical attributes (head, eyes, long, hand), and a frog. The
inclusion of ‘long’ might suggest a temporal or duration-related aspect. The
last topic highlights time, spatial orientation (right, inside, door), objects
(ream, frog), physical attributes (head, eyes, small), and the action of
starting something. The modern stories’ topic modeling results suggest a
recurring theme involving concepts such as time, spatial orientation, objects,
and actions. Each topic emphasizes different aspects and introduces additional
elements like fairies or physical attributes. Figure 4 represents the level of
similarity and difference between the real stories and the LLM-generated
stories. There are greater similarities between these stories than there
appear to be differences in the main topics.

As expected, the results of the topic modeling showed similarities between the
original 122 stories in the training corpus and the stories generated by the
LLMs. These stories shared fairy tale and fantasy elements as well as topics
of goodness, greatness, time, and setting elements of night, houses, and the
color white. Once we compare this with the modern stories, we see that the
focus of the small data set we have is similarly focused on time and fairies,
but has more topics relating to spatial orientation. We are likely seeing a
change in the content of stories written for children. With only ten modern
stories, we cannot reliably generalize over all stories, but we noticed
tendencies such as that the modern story set did tend to involve more overtly
educational elements aimed at younger age groups when compared to the older
stories.

##  8 Generated stories do not have similar sentence structure to original
stories

Table 3 shows the percentage of overlapping Weisfeiler Lehman hashes between
the dependency tree graphs of sentences generated by various models and those
actual children’s stories, both old and modern. We also got an overlap of
35.57 percentage between old and modern actual stories, which is greater than
all the values in Table 3. This shows that the structure of sentences in
children’s literature has changed over time, which supports our earlier
findings that children’s literature has undergone noticeable changes over
time.

Model | Percentage overlap with
---|---
Old stories | Modern stories
OPT-Line | 34.82 | 34.21
OPT-256 | 31.37 | 28.88
OPT-512 | 32.49 | 29.89
LLaMA-Line | 34.23 | 33.64
LLaMA-256 | 32.14 | 29.82
LLaMA-512 | 32.27 | 30.73
Alpaca: T-1 | 17.31 | 20.37
Alpaca: T-2 | 14.67 | 17.52
Alpaca: T-3 | 15.20 | 16.92
Alpaca: T-4 | 15.41 | 17.84
Table 3: Overlap of the hashes of the dependency tree graph of the sentences
in generated stories against old and modern actual stories.

Additionally, we observe a higher percentage of overlap between old original
stories and the stories generated by OPT and LLaMA, which again aligns with
our earlier findings that LLMs learn from their context. Furthermore, for the
stories generated by OPT and LLaMA, we see an average overlap of 30% with
modern stories, which can be attributed to the fact that these models were
trained on a dataset consisting of recent text.

The stories generated by Alpaca have a slightly higher overlap with modern
stories compared to old stories, but the percentage overlap in sentence
structures is still relatively low ( $\leq$ 20%). Given that the old and
modern actual stories share around 35% of the same sentence structures, we
expected Alpaca’s generated stories to overlap more with modern stories. But
since Alpaca is a generic model fine-tuned for instruction-following and not
solely trained or fine-tuned on children’s literature, it seems plausible that
it would not be capable of fully generalizing over sentence or grammatical
structures observable in children’s literature.

##  9 Conclusion and Future Work

Our study examines the trustworthiness of children’s stories generated by
large language models. While these generated stories may share similar topics
and patterns with actual stories, they fail to capture all the nuances present
in children’s literature, and may even contain toxic material that is
inappropriate for children. Based on our findings, we conclude that LLMs are
not yet appropriate for generating high-quality children’s literature. Moving
forward, we plan to extend our work by implementing reinforcement learning
with both automatic and human feedback to improve the quality of LLM-generated
children’s stories.

## Acknowledgments

This work was supported by resources provided by the Office of Research
Computing, George Mason University (URL: https://orc.gmu.edu) and by the
National Science Foundation (Awards Number 1625039 and 2018631). Additionally,
Prabin Bhandari has been partially supported by the National Science
Foundation Grant No. IIS-2127901. Our sincere appreciation goes to Antonios
Anastasopoulos and Géraldine Walther for their valuable suggestions and
unwavering support throughout the course of this research.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can Large Language Models Be an Alternative to Human Evaluations? ArXiv:2305.01937 [cs].
  * Clark et al. (2018) Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018.  Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories.  In _23rd International Conference on Intelligent User Interfaces_ , pages 329–340, Tokyo Japan. ACM.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Flesch (1948) Rudolph Flesch. 1948.  A new readability yardstick.  _Journal of applied psychology_ , 32(3):221.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.  _arXiv.org_.  Place: Ithaca Publisher: Cornell University Library, arXiv.org.
  * Hanu and Unitary team (2020) Laura Hanu and Unitary team. 2020.  Detoxify.  Github. https://github.com/unitaryai/detoxify.
  * Haussamen (1994) Brock Haussamen. 1994.  The future of the english sentence.  _Visible language._ , 28(1).
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ , 103:102274.
  * Lucy and Bamman (2021a) Li Lucy and David Bamman. 2021a.  Gender and representation bias in GPT-3 generated stories.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 48–55, Virtual. Association for Computational Linguistics.
  * Lucy and Bamman (2021b) Li Lucy and David Bamman. 2021b.  Gender and Representation Bias in GPT-3 Generated Stories.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 48–55, Virtual. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Price (2016) Rob Price. 2016.  Microsoft deletes racist, genocidal tweets from AI chatbot Tay - Business Insider.
  * Razumovskaia et al. (2022) Evgeniia Razumovskaia, Joshua Maynez, Annie Louis, Mirella Lapata, and Shashi Narayan. 2022.  Little red riding hood goes around the globe:crosslingual story planning and generation with large language models.
  * Shervashidze et al. (2011) Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. 2011.  Weisfeiler-lehman graph kernels.  _Journal of Machine Learning Research_ , 12(9).
  * Simon and Muise (2022) Nisha Simon and Christian Muise. 2022.  TattleTale - Storytelling with Planning and Large Language Models.
  * Sun et al. (2023) Yuqian Sun, Ying Xu, Chenhang Cheng, Yihua Li, Chang Hee Lee, and Ali Asadipour. 2023.  Explore the future earth with wander 2.0: Ai chatbot driven by knowledge-base story generation and text-to-image model.  In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI EA ’23, New York, NY, USA. Association for Computing Machinery.
  * Tang et al. (2022) Chen Tang, Chenghua Lin, Henglin Huang, Frank Guerin, and Zhihao Zhang. 2022.  EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention.  ArXiv:2210.12463 [cs].
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.
  * Tran (2022) Khuyen Tran. 2022.  pyLDAvis: Topic Modelling Exploration Tool That Every NLP Data Scientist Should Know.
  * Venkit et al. (2023) Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao ’Kenneth’ Huang, and Shomir Wilson. 2023.  Nationality bias in text generation.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language model with self generated instructions.
  * Williams (2019) Kyle Williams. 2019.  Neural lexicons for slot tagging in spoken language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers)_ , pages 83–89, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Xiang et al. (2018) Jingwen Xiang, Zoie Zhao, Mackie Zhou, Megan McKenzie, Alexis Kilayko, Jamie C Macbeth, Smith Edu, Smith Edu, Smith Edu, Smith Edu, Smith Edu, and Smith Edu. 2018.  Interleaving a Symbolic Story Generator with a Neural Network-Based Large Language Model.
  * Yang (2022) Weipeng Yang. 2022.  Artificial Intelligence education for young children: Why, what, and how in curriculum design and implementation.  _Computers and Education: Artificial Intelligence_ , 3:100061.
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022.  Wordcraft: Story Writing With Large Language Models.  In _27th International Conference on Intelligent User Interfaces_ , IUI ’22, pages 841–852, New York, NY, USA. Association for Computing Machinery.  Event-place: Helsinki, Finland.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.


# Duoduo CLIP: Efficient 3D Understanding
with Multi-View Images

Han-Hung Lee1,* Yiming Zhang1,* Angel X. Chang1,2
1Simon Fraser University, 2Canada CIFAR AI Chair, Amii
{hla300, yza440<EMAIL_ADDRESS>
https://github.com/3dlg-hcvc/DuoduoCLIP

###### Abstract

We introduce Duoduo CLIP, a model for 3D representation learning that learns
shape encodings from multi-view images instead of point-clouds. The choice of
multi-view images allows us to leverage 2D priors from off-the-shelf CLIP
models to facilitate fine-tuning with 3D data. Our approach not only shows
better generalization compared to existing point cloud methods, but also
reduces GPU requirements and training time. In addition, we modify the model
with cross-view attention to leverage information across multiple frames of
the object which further boosts performance. Compared to the current SOTA
point cloud method that requires 480 A100 hours to train 1 billion model
parameters we only require 57 A5000 hours and 87 million parameters. Multi-
view images also provide more flexibility in use cases compared to point
clouds. This includes being able to encode objects with a variable number of
images, with better performance when more views are used. This is in contrast
to point cloud based methods, where an entire scan or model of an object is
required. We showcase this flexibility with object retrieval from images of
real-world objects. Our model also achieves better performance in more fine-
grained text to shape retrieval, demonstrating better text-and-shape alignment
than point cloud based models.

##  1 Introduction

Learning representations for 3D shapes is increasingly useful with growing
applications in object generation, shape understanding, grounding and
robotics. In recent years, there has also been growing interest in 3D
representations that are semantically aligned with text. By having a 3D
representation aligned to language, these representations can easily be used
for text-to-3D shape retrieval [36, 41, 45] and generation [4], or as a basis
for 3D feature maps that can be queried with text [16, 25, 19].

Initial work in this domain [4, 36, 41] attempted to train such aligned models
on limited data. More recent work [49, 25] leveraged large pre-trained vision-
language models (e.g. CLIP [35]) that already align the text and image
embeddings. ULIP [49] used frozen CLIP text-image encoders, and trained only
the point-cloud encoder but was still limited due to the limited number of 3D
objects as well as paired text captions. With larger 3D datasets like
Objaverse [7] becoming available in conjunction with automated captioning
methods like BLIP-2 [23], recent works [50, 25, 33, 60] trained on such
datasets have showcased better performance and stronger generalizability.
Despite this, more and more GPU resources are required for training such
methods. These recent works [50, 25, 33] also predominantly choose point
clouds as their 3D representation. The point clouds are typically limited in
resolution and have a domain gap with real-world images, and thus these point
cloud based methods can not effectively leverage the 2D priors from CLIP.
Uni3D [60] proposed a training regime for incorporating 2D priors with point
cloud as input. However, their model requires significant scaling (1B
parameters) for the best performance. In this work, we instead choose multi-
view images as our shape representation. This not only allows for an easier
extension and a more natural way to adapt off-the-shelf 2D representation
models. Our Duoduo CLIP also results in more efficient training. As shown in
Figure 1(b), our model not only has much stronger generalization on shapes
that were not seen during training with a modest model size, but also
significantly lowers the amount of compute resources required for training
(see Table 1). While some recent work [39, 24] also explore using multi-images
with CLIP, their methods do not investigate the use of contrastive learning
and only focus on zero/few-shot performance on small-scale datasets.

(a) We use multi-view images, instead of point-clouds, to represent the shape.

(b) Accuracy on Objaverse-LVIS with all models trained on Ensembled (no LVIS).
Our Duoduo CLIP is fast to train, relatively small while also achieves the
best performance.

Figure 1: We propose Duoduo CLIP, a simple model for 3D shape understanding
that represents the shape with a multi-view images encoder instead of a point-
cloud (PC) encoder. Compared to recent PC-based methods (e.g. OpenShape [25],
Uni3D [60]), our model is both accurate and efficient. Table 1: Comparison of
training time and GPU used for our method and recent point cloud based
methods.

Method | GPU | Time
---|---|---
OpenShape [25] | 1$\times$A100 (80GB) | 300 hr
Uni3D [60] | 24$\times$A100 (40GB) | 20 hr
RECON++ [33] | 8$\times$A800 (80GB) | 1 day
Ours (Full) | 4$\times$A40 (48GB) | 14.3 hr
Ours (6 layers) | 4$\times$A5000 (24GB) | 14.3 hr

Using multi-view images also offers greater flexibility in many scenarios.
Point cloud based methods require a fairly complete scan of the object, which
can be challenging to obtain. In contrast, our model can encode objects from a
variable number of frames (1 to M), with more frames capturing more details.
Note that we use frames and views interchangeably. This makes our model plug-
and-play for any method using image features for encoding 3D objects,
enhancing performance with multiple frames. Additionally, previous methods
that only train on synthetic objects have limited performance on real-world
objects. We further train both our model point cloud based methods on real
photos and densely reconstructed objects. Due to artifacts in the point cloud
reconstruction pipeline, our model outperforms on real-world objects.
Performance is boosted with additional multi-view images where point clouds
aren’t available, demonstrating our model’s potential for scaling as more data
becomes available.

In summary, our Duoduo CLIP model provides the following: Simple model. We
average embeddings across views of an object to obtain 3D representations,
while modifying the attention layers to span multiple frames of the shape to
enable information flow across frames in the model. Better efficiency. We
explore several strategies for efficiently fine-tuning off-the-shelf CLIP
models, including training only the attention layers and selectively choosing
the number of layers to train without hurting performance. More flexible
representation. We showcase benefits of choosing multi-view images as a more
flexible 3D representation over point clouds on real-world objects.
Furthermore, we also demonstrate better text to shape retrieval with more
fine-grained descriptions.

##  2 Related Work

There is a large literature of work on 3D representation learning from 3D data
either using supervised training [31, 34] or using self-supervised learning
(with reconstruction loss [5], contrastive learning [48], masked prediction
[54], and others). Here we focus on work that learns aligned representation
spaces between text and either 2D or 3D.

2D-text joint representation. Modeling multiple modalities through a shared
representation has been explored through early works [10, 11, 21, 47] by
aligning images and text embeddings. With the introduction of the InfoNCE loss
[29], ConVIRT [57] proposed using contrastive learning with InfoNCE style loss
to learn joint text and image embeddings. This was popularized by follow-up
work such as CLIP [35] and ALIGN [20], which demonstrated that by scaling up
training on a large-scale dataset, the learned embeddings can be useful for
many downstream tasks such as classification, retrieval, grounding and
generation. Followup work [20, 51, 28, 55, 52] has since been proposed to
improve upon the formulation or training. There have also been attempts such
as MERU [8] that use a hyperbolic embedding space to better model the
hierarchical relations between text and images.

3D-text joint representation. Text2Shape [4] was one of the first works that
introduced a captioned text and 3D dataset and used metric learning to learn
an aligned space between text and 3D shapes. Follow-up work investigated how
to improve the aligned representation for text-to-shape retrieval with triplet
loss [41] and contrastive losses [36, 49, 45]. Prior work also investigated
the use of CLIP to align the embeddings [43, 36, 49]. ULIP [49] used frozen
CLIP text-image encoders, and trained only the point-cloud encoder. However,
the generalizability of these works were limited by the available dataset size
at the time. With the introduction of larger scale 3D datasets such as
Objaverse [7], and the use of LVLMs for captioning 3D datasets [25, 27], more
recent work [25, 50] were able to train with larger datasets of paired text
and 3D shapes. OpenShape [25] and ULIP-2 [50] were among the first to use such
datasets to train a point cloud encoder aligned with text using contrastive
learning. TAMM [58] leverages additional adapters to better align with the
pre-trained CLIP models. Uni3D [60] exploits 2D priors within pre-trained
large-scale vision models and maps point cloud patches to image patches within
pre-trained ViTs to enable the scaling of even larger models. VIT-LENS [22]
follows a similar pipeline by training a point cloud embedding layer and
perceiver model to map to a frozen CLIP model. RECON++ [33] similar to RECON
[32] employs both contrastive and MAE [15] reconstruction losses with an
additional multi-view image matching cost. These recent works typically rely
on frozen text and image encoders, and all used point-cloud encoders. Our work
shows that taking embeddings of multi-view images is more efficient and works
better with real-world images.

Multi-view images for text-3D representations. Multi-view images have long
been used to represent 3D shapes [2, 37, 40, 46, 13], with recent work [39,
24] using multi-view images for shape representation by aggregating CLIP
features. While these works also use CLIP with multi-view images, our focus is
whether using a trainable multi-view shape encoder as a drop-in-replacement
for the point-cloud encoder in contrastive settings such as OpenShape will
result in better shape-representation for downstream tasks. These works do not
focus on improved representation learning, but on how to use prompting to
select better views or view aggregation and improve shape classification.
There is also a line of work [56, 61, 17] that uses multi-view depth maps from
point cloud to adapt CLIP models for 3D. However, the focus of these methods
is still on point clouds and the models show limited performance on large-
scale datasets [50, 25].

##  3 Method

Figure 2:  Our multi-view encoder takes a variable number $[1,M]$ of images as
input and outputs a single shape embedding. The first few layers of the ViT
model are frozen (blue) and attention operates on the individual frames in
parallel. For the latter layers, the attention layers are trainable (orange)
and modified to attend over all $M$ views. The embeddings for each frame are
averaged to get the final embedding. Note that only major components of the
model are depicted.

Our Duoduo CLIP learns a shape representation by using contrastive learning to
encode multi-view images into a pre-aligned text-image space. Our contrastive
learning framework is similar to previous works [49, 50, 25, 58, 60], with a
shape encoder $E^{S}$ and frozen image $E^{I}$ and text $E^{T}$ encoders from
a pre-trained CLIP model. The key difference of our model is that we replace
the point cloud based shape encoder with one that uses multi-view images.
Figure 1(a) shows a comparison of our model to prior work, and Figure 2 shows
the details of our multi-view shape encoder.

To perform training, the dataset is organized into triplets of
$(S_{i},I_{i},T_{i})$, where $S$ is the shape representation, $I$ are rendered
images of the 3D object and $T$ is the corresponding text description.
Embeddings are calculated for each modality $f^{S}_{i}=E^{S}(S_{i})$,
$f^{I}_{i}=E^{I}(I_{i})$ and $f^{T}_{i}=E^{T}(T_{i})$. The asymmetric
contrastive loss [57] between any two modalities is:

$l^{a\rightarrow b}_{i}=-\log\frac{\exp(\langle
f^{a}_{i},f^{b}_{i}\rangle)/\tau}{\Sigma_{k=1}^{N}\exp(\langle
f^{a}_{i},f^{b}_{k}\rangle)/\tau}$ (1)

where $a,b$ is any two modalities, $\tau$ is the temperature and $N$ is the
batch size. Then, the full loss is:

$L_{CON}=\frac{1}{4N}\Sigma_{i=1}^{N}(l^{S\rightarrow T}_{i}+l^{T\rightarrow
S}_{i}+l^{S\rightarrow I}_{i}+l^{I\rightarrow S}_{i})$ (2)

Multi-view shape encoder. For our multi-view shape encoder, we use rendered
images of the object. Note that the input shape and image representations are
the same, i.e. $S_{i}=I_{i}=\\{r_{1},...,r_{m}\\}$, where $r_{j}$ is a
rendered frame, and $m$ is the number of multi-views. $E^{I}$ and $E^{S}$
share the same network architecture, with the main difference between $E^{I}$
and $E^{S}$ is that $E^{I}$ is frozen, while we train $E^{S}$. Weights for
$E^{S}$ are initialized from $E^{I}$. Empirically, we found it was useful to
keep the contrastive losses $l^{S\rightarrow I}_{i}+l^{I\rightarrow S}_{i}$.
We suspect that by keeping $E^{I}$ frozen and the contrastive loss with the
original image embeddings, our $E^{S}$ can learn a good representation that is
close to the text embeddings, but does not drift too much away from the
initial image embeddings.

For both $E^{S}$ and $E^{I}$ we average the extracted embeddings for all $m$
views to produce a single latent representation for an object:
$f^{S}_{i}=E^{S}(S_{i})=\frac{1}{m}\Sigma_{j=1}^{m}\text{ViT}(S_{i})[j]$,
where ViT [9] is the backbone encoder. Here we abuse the notation a bit, since
our method allows interaction between views in the model, the index operation
gets the corresponding embedding from the $j$-th frame’s [CLS] token. We
initialize $E_{S}$ initially from $E^{I}$. During training, we randomly sample
$1$ to $M$ views for each step so that the model is robust to a variable
number of views.

Multi-view attention. We incorporate information flow between views by
modifying the self-attention layers to attend over tokens across multiple
views similar to recent multi-view aware methods like MVDream [38]. The
original self-attention formulation [44] is
$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}}V)$,
where $Q,K,V\in R^{L\times k}$, and $L$ is the token length for a single image
after patchify operations and $k$ is the channel dimension. For cross-view
attention we aggregate token features across $m$ multi-views such that
$Q^{\text{MV}},K^{\text{MV}},V^{\text{MV}}\in R^{(L\times m)\times k}$ (MV
stands for multi-view) and the
$\operatorname{Attention}(Q^{\text{MV}},K^{\text{MV}},V^{\text{MV}})$ can be
calculated without changing the parameters of the original model. After
attention, the views are separated and processed in parallel in other parts of
the model. This mechanism helps the model reason between different views of
the object and aggregate the most important information across all tokens.

Trainable layers. To reduce the memory needed for fine-tuning, we only train
the self-attention layers. We further cut down on memory usage by reducing the
number of trainable layers. Since the non-trainable self-attention layers have
only been trained to see patches of a single image, we only enable the multi-
view attention layers for the unfrozen attention layers (see Figure 2).

Figure 3:  Left. We compare the Top 1 accuracies on Objaverse-LVIS for our
model (Duoduo CLIP) and two pretrained CLIP models (zero shot) for different
number of frames. B-32 is the model we initialize from and train with and
BigG-14 is the model used by OpenShape. The dashed lines show the difference
between using 1 and 12 frames. Right. We show the Top1, Top3 and Top5
accuracies for our model for 1 to 12 frames (note the y-axis starts at 40).
Gray dashed lines shows the inference time(s) for 1 iteration and batch size
200 at different number of frames.

##  4 Experiments

To compare our Duoduo CLIP to prior work, we conduct experiments showing shape
classification on a collection of 3D assets (on Objaverse-LVIS in Section
4.2), real-world multi-view images (on MVImgNet in Section 4.3), and more
fine-grained text-to-shape retrieval (Section 4.4). We first describe the
experimental setup including implementation details and the datasets we use.

###  4.1 Experimental setup

Implementation details. All of our models are trained with 16-bit mixed
precision and a batch size of $1600$ for 80 epochs. We use a learning rate of
$5e-5$ with cosine annealing. At each training step we randomly sample $1$ to
$6$ multi-views for a batch of objects. The pretrained CLIP model used for
initialization as well as the contrastive target is the ViT-B/32 CLIP model
with checkpoint _laion2b_s34b_b79k_ from the open source implementation
OpenCLIP [18]. Additionally, we re-implemented OpenShape with 16-bit mixed
precision training without projection layers for faster training in some
baselines, indicated by OpenShape$\dagger$.

3D Datasets. We follow OpenShape [25] and train our model on a combination of
four datasets consisting of Objaverse [7], ABO [6], ShapeNet [3] and 3D-FUTURE
[12]. Text for the shapes consists of captions produced by image-captioning
models and filtered text descriptions from metadata of the 3D objects crawled
from the web (see Liu et al. [25] for details). In total, the combined dataset
contains $\sim$874k shapes, with 46k shapes within the LVIS subset of
Objaverse which are used for evaluation. As in OpenShape [25], we consider two
variations of the data: 1) Ensembled which contains all shapes, and 2)
Ensembled (no LVIS) which excludes shapes from the LVIS subset.

Renderings. We use images from Zero123 [26], which provides renderings of most
shapes from Objaverse from $12$ random views using spherical sampling. The
shapes not included in Zero123 (a few remaining objects in Objaverse and the
other 3 datasets) are rendered using their Blender 111https://www.blender.org/
script to match the shapes in OpenShape. We render $12$ views for all these
objects.

Real-world data. MVImgNet [53] consists of multi-view images for 220k real-
world objects over 238 categories. We first filter objects that has at least
$12$ multi-views to obtain $\sim$190k remaining objects, where $12$ views are
evenly sampled. BLIP-2 [23] is then used to produce captions for each of the
views. To compare with point cloud based methods, we use the MVPNet subset
containing $\sim$87k objects with densely reconstructed point clouds. After
the preprocessing and filtering above, we obtain from MVPNet, $\sim$66k
objects for the train split and $\sim$16k objects for the validation set,
which has 180 categories. Note that the objects in the training and validation
splits for MVPNet are disjoint but are both included in MVImgNet.

Table 2: Classification accuracy on Objaverse-LVIS trained on Ensembled (no
LVIS) and Ensembled. The zero shot CLIP (B-32 and BigG-14) models are not
trained on 3D data and serve as baselines. For each model, we indicate the
representation used for computing the embeddings, multi-view (MV) or point-
cloud (PC), and the shape encoder (Enc) architecture that is used. Our multi-
view based Duoduo CLIP is able to outperform point-cloud based approaches with
the exception of Uni3D is a large model with 1B parameters. $\dagger$
indicates our reimplementation of OpenShape which is faster and outperforms
the original OpenShape.

|  |  | Ensembled (no LVIS) | Ensembled
---|---|---|---|---
Method | Rep | Enc | Top1 | Top3 | Top5 | Top1 | Top3 | Top5
ZS B-32 (12F) | MV | Avg | 35.7 | 54.8 | 62.1 | 35.7 | 54.8 | 62.1
ZS BigG-14 (12F) | MV | Avg | 43.5 | 64.2 | 71.3 | 43.5 | 64.2 | 71.3
ULIP [49] | PC | PointBERT | 21.4 | 38.1 | 46.0 | 26.8 | 44.8 | 52.6
OpenShape [25] | PC | PointBERT | 39.1 | 60.8 | 68.9 | 46.8 | 69.1 | 77.0
OpenShape$\dagger$ | PC | PointBERT | – | – | – | 50.0 | 72.0 | 79.2
TAMM [58] | PC | PointBERT | 42.0 | 63.6 | 71.7 | 50.7 | 73.2 | 80.6
Uni3D [60] | PC | 3D VIT | 47.2 | 68.8 | 76.1 | 55.3 | 76.7 | 82.9
ShapeLLM [33] | PC | RECON++ | – | – | – | 53.7 | 75.8 | 82.0
VIT-LENS [22] | PC | VIT-LENSG | 50.1 | 71.3 | 78.1 | 52.0 | 73.3 | 79.9
Duoduo CLIP (5F) | MV | MVA | 51.3 | 73.1 | 79.9 | 54.1 | 76.0 | 82.4
Duoduo CLIP (12F) | MV | MVA | 52.7 | 74.5 | 81.3 | 55.2 | 77.1 | 83.4

###  4.2 Classification on Objaverse-LVIS

We conduct classification experiments on Objaverse-LVIS to investigate the
performance of different variants of Duoduo CLIP and compare our model to
prior work. As default settings for our experiments in this section, we use
$12$ multi-views to encode an object for evaluation. And only the multi-view
attention of the latter 6 layers of the model is trained (total 12 layers). We
run evaluations with 3 different seeds and average the results for
experiments. Unless otherwise stated models are trained with the Ensembled
dataset. For classification, we use do nearest neighbor search of the multi-
view shape embedding with class embeddings. As in OpenShape, for each class
label, we construct several sentences with the label and then take the average
of the sentence embeddings as the class embedding. The final class chosen is
the class with the highest average similarity between the class embedding and
the shape embedding.

We compare the performance of our model against two CLIP models for different
numbers of frames in Figure 3, and recent state-of-the-art models in Table 2.
For the CLIP models, we selected VIT with B-32 (the model that our Duoduo CLIP
is initialized with), and BigG-14 (the CLIP teacher model used by OpenShape).
Both models are run in a zero-shot manner without training on additional 3D
data (the weights are frozen, with embeddings from the image encoder
averaged).

How well does performance scale with number of views? Having a more complete
view of a 3D object is important for 3D representation learning as elements of
the object may be occluded for certain views. Thus we investigate how the
number of views affects the classification accuracy. From Figure 3 (left), we
see that our model performance scales better as the amount of information
(views) increases compared to the zero-shot CLIP models ($11.22$ for our
Duoduo CLIP, and $7.45$ for B-32, the model we used for initializing our
weights). This suggests that our training strategy of randomly sampling 1 to 6
views during training is effective in letting the model learn to leverage the
number of views given to it. Although, the accuracy increases as the number of
multi-views increases, the inference time also increases (Figure 3 right). As
the accuracy tapers off as we increase the number of views, to avoid
additional computational cost during inference, we use 12 views as the
default.

How many multi-views to beat a point cloud? Compared with using multi-views,
using a point cloud representation offers good coverage of the entire geometry
of the 3D object. Thus, we ask the question of how many views does it take to
match existing methods on the Objaverse-LVIS classification task. We compare
against ULIP [49], OpenShape [25], TAMM [58] that use point clouds with
PointBERT [54] as the 3D encoder, but different ways of fine-tuning the 3D
encoder. We also compare against more recent methods [60, 33, 22] that are the
current top-performing methods on Objaverse-LVIS. From Table 2, on the
Ensembled dataset, we are able to beat most previous methods with 5 multi-
views except for Uni3D which has a much larger model (1B parameters). When we
increase to 12 frames we are on par and even surpass Uni3D. This is also while
using much fewer resources as seen in Table 1.

Does multi-views generalize better compared to point cloud? From the Ensembled
(no LVIS) column of Table 2, we see that even the zero-shot BigG-14 model used
as the teacher model for OpenShape beats it. Suggesting that the CLIP models
already generalize well to unseen data, and point cloud methods potentially
having worse generalization on unseen shapes. With our model that is further
trained on Ensembled (no LVIS) we surpass point cloud methods by a large
margin. Showing that fine-tuning the CLIP model better preserves their
generalization ability.

Table 3: Multi-view attention (MVA) ablation of accuracy on Objaverse-LVIS.

Method | Frames | Top 1 | Top 3 | Top 5
---|---|---|---|---
-MVA | 6 | 54.22 | 76.13 | 82.74
+MVA | 6 | 54.59 | 76.46 | 82.89
-MVA | 12 | 55.02 | 76.92 | 83.49
+MVA | 12 | 55.32 | 77.08 | 83.49

Table 4: Number of layers ablation. Accuracy on Objaverse-LVIS. Method | Top 1 | Top 3 | Top 5
---|---|---|---
3 layers | 53.77 | 75.8 | 82.41
6 layers | 55.24 | 77.08 | 83.43
12 layers (full) | 55.32 | 77.08 | 83.49

Does multi-view attention help? We verify the effectiveness of our cross view
attention in Table 4. For both -MVA and +MVA, we train all 12 attention
layers, with the difference being +MVA has the cross-views attention enabled.
We see that using multi-view attention helps the performance most in boosting
Top1 accuracy for both 6-frame and 12-frame evaluations.

Do we need to train all the layers? We conduct an ablation to see if it is
necessary to train all 12 layers. In Table 4, we show the accuracy for
restricting the training to the last 3 and 6 MVA layers vs the full 12 MVA
layers. We see that the number of trainable layers can be cut by half with
minimal losses to accuracy, while further cuts start to hurt model
performance. Training only half the layers reduces the VRAM usage by roughly
half and allows us to train on GPUs with less memory. By restricting the
training to the last 6 MVA layers, we can shift from training on 4$\times$A40s
(48 GB) to 4$\times$A5000s (24 GB) (see Table 1).

Table 5: MVImgNet classification comparison.

Method | Top 1 | Top 3 | Top 5
---|---|---|---
ZeroShot B-32 (12F) | 52.68 | 70.99 | 77.22
OpenShape$\dagger$ | 3.55 | 8.94 | 12.70
OpenShape$\dagger$ (+MVPNet) | 54.35 | 71.62 | 77.81
Ours (12F) | 49.16 | 66.96 | 74.12
Ours (+MVPNet) (1F) | 59.23 | 76.12 | 81.74
Ours (+MVPNet) (6F) | 64.18 | 80.80 | 85.92
Ours (+MVPNet) (12F) | 64.44 | 81.11 | 85.97
Ours (+MVImgNet) (1F) | 61.75 | 79.07 | 84.18
Ours (+MVImgNet) (6F) | 65.70 | 82.44 | 87.08
Ours (+MVImgNet) (12F) | 66.06 | 82.72 | 87.21

Table 6: Text to shape retrieval comparison. Methods are trained on the
Text2Shape [4] dataset (T2S) or Ensembled.

Model | Data | RR@1 | RR@5 |
---|---|---|---|---
Text2Shape [4] | T2S | 0.40 | 2.37 |
Y2Seq2Seq [14] | T2S | 2.93 | 9.23 |
Parts2Words [41] | T2S | 12.72 | 32.98 |
TriCoLo [36] | T2S | 12.22 | 32.23 |
MXM-CLR [45] | T2S | 16.83 | 39.06 |
ZeroShot B-32 | Images | 11.90 | 27.85 |
OpenShape$\dagger$ | Ens. | 10.53 | 25.94 |
Uni3D-G | Ens. | 10.78 | 26.37 |
Duoduo CLIP | Ens. | 15.90 | 34.08 |

###  4.3 Classification on real-world data

To investigate how well our Duoduo CLIP works on real-world images, we
evaluate on multi-view images from MVImageNet. For these experiments, we take
our Duoduo CLIP model trained on the Ensembled dataset and add additional
training data from MVImgNet. We consider two sets of data from MVImgNet, the
MVPNet training split of objects or the entire MVImgNet (after our
preprocessing and filter). For comparison with point cloud based methods, we
evaluate on the validation set of MVPNet. As a representative of point cloud
based methods, we choose OpenShape as it is relatively resource-light compared
to other point cloud methods.

Classification results. We report accuracy results on MVPNet in Table 6. For
OpenShape, the top-1 accuracy is extremely low (only $3.55\%$ top-1) when
evaluating on MVPNet directly with the model trained on Ensembled only. We
suspect this is due to noisy reconstructed point clouds and inconsistent
orientations compared to the training data. After training on MVPNet data,
OpenShape’s performance improves significantly. For our Duoduo CLIP, even
without training on MVImgNet or MVPNet data, our model achieves a top-1
accuracy of $49.16\%$. However, this is lower than the zero-shot CLIP B-32
model that our model is initialized from, suggesting possible overfitting to
synthetic data. When we include training with multi-view images from MVPNet,
our method surpasses both OpenShape and the zero-shot CLIP B-32 model even
when just using one 1 view. Performance is further boosted when trained on the
entire MVImgNet, highlighting the advantages of our Duoduo CLIP which uses
multi-view images over OpenShape’s point clouds. By using multi-view images,
we have better generalization on unseen data due to the priors from
initializing with CLIP B-32. We also can train on more data, as multi-view
images are more readily available compared to point clouds.

Figure 4: Retrieval results for OpenShape and Duoduo CLIP given input images
from MVPNet. We perform retrieval with 12 input images, on the left most
column we show one of the frames of the query object. For both methods we show
the top 3 retrieval results using Objaverse as a retrieval library. Overall,
Duoduo CLIP retrieves shapes that are more similar to the query image, while
OpenShape sometimes retrieves broken geometry (row 3).

Retrieval examples. In Figure 4, we present example retrieval results for
querying Objaverse objects using images from the MVPNet validation set. We
compare retrievals using OpenShape (+MVPNet) and Duoduo CLIP (+MVPNet). For
OpenShape, we use 12 query images for each object, encode them with the frozen
BigG-14 CLIP model, and average the image embeddings to obtain the query
embedding.222For the largest BigG-14 CLIP model, OpenShape does not use any
linear projection layer $g^{I}$. This embedding is then matched against a
database of point cloud embeddings of Objaverse objects to retrieve the top 3
results. For our method, Duoduo CLIP encodes both the query and Objaverse
objects using 12 multi-view images.

From Figure 4, we see that for basic objects like benches, both methods are
able to retrieve plausible objects. For the armchair (row 2), while OpenShape
retrieved objects of the same semantic class, our Duoduo CLIP retrieved
armchairs that has better matching textures. This is because multi-view images
can represent textures better than point clouds. Note that some retrieved
objects look similar due to duplicate objects in Objaverse. For the jacket
(row 3), the objects returned by OpenShape have broken geometry and are
incoherent. We hypothesize this may happen when the sampled points produce
embeddings that are by chance similar to the query image. This issue is also
present last example (a red and black soccer ball) where the first object
retrieved by OpenShape has broken geometry. However, our method also fails in
this case, as our model retrieves ladybugs with red wings and black dots that
resemble the soccer ball. Interestingly, OpenShape is able to retrieve a
soccer ball and a red ball. This can indicate that point cloud based methods
may be more robust to geometric properties of the object compared to the
rendered images. However, we find that overall the retrieved objects from our
method is more consistent.

###  4.4 Fine grained text-to-shape retrieval

Previous evaluations focused on classification or image-based retrieval and do
not assess how well the methods align fine-grained text descriptions with 3D
models. To evaluate this aspect, we use the test split of Text2Shape [4],
which contains $\sim$1.4k tables and chairs from ShapeNet and $\sim$7.4k
human-captioned descriptions ($\sim$5 per shape). We report the shape
retrieval rates at 1 and 5.

Retrieval performance. We compare the performance of our model to prior work
in Table 6. We also include previous state-of-the-art methods that were
trained on the Text2Shape dataset. Overall, compared to other methods trained
on the Ensembled dataset, our method demonstrates superior performance. The
zero-shot performance alone is impressive, surpassing point cloud methods.
Notably, we observed a significant discrepancy between the zero-shot CLIP
model reported by Ruan et al. [36] and our findings, where the LAION
checkpoint trained by OpenCLIP significantly outperforms the original OpenAI
checkpoint. Lastly, our model approaches the performance of the SOTA method
trained directly on Text2Shape, indicating that there is still much to explore
regarding more fine-grained text alignment in large-scale text-to-3D models.

Figure 5: Retrieval using fine-grained text from the Text2Shape dataset. On
the left is the input text description and we show the top 3 retrieval results
for Uni3D and Duoduo CLIP with the ground truth shape highlighted in green.

Retrieval examples. We present example retrieval results compared to Uni3D in
Figure 5. Generally, for text descriptions emphasizing complex textures, such
as the ‘man image’ in the first row, our model successfully retrieves the
corresponding ground truth 3D model. However, for descriptions focusing on
geometric properties, like ‘round back’ in the second row, point cloud-based
methods tend to retrieve shapes that better match the specified criteria.

##  5 Limitations and Future Work

We introduced Duoduo CLIP, an efficient joint text and 3D representation
method that leverages multi-view images and surpasses current point cloud
based state-of-the-art methods across almost all quantitative metrics while
using significantly fewer resources on both synthetic and real datasets.

However, our method sometimes fails in cases requiring geometric
understanding, as seen in the qualitative retrieval results from images or
text. This is an inherent advantage of point clouds, as each point contains
three-dimensional coordinates. To address this, we could explore additional
inputs to the model such as depth or normal images of the objects, which
provide more geometric information, especially with recent depth estimation
methods becoming better. Another potential way to improve performance is with
LLMs, as demonstrated by Tang et al. [42], where representations from ULIP-2
were used by an LLM to boost performance and enable question-answering
applications. Using our model might further increase performance, offering the
flexibility of requiring only a single or a sparse number of images as input.
Banani et al. [1] also point out the relative weakness of CLIP pretraining on
both 3D structure and consistency benchmarks compared to newer SSL methods
[59, 30], which can potentially replace the current pretraining framework used
by most works.

Overall, we believe that our method will provide better resource scaling
compared to point cloud based methods as more 3D data becomes available in the
future.

Acknowledgements. This work was funded by a CIFAR AI Chair, a NSERC Discovery
grant, and a CFI/BCKDF JELF grant.

## References

  * Banani et al. [2024] Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani.  Probing the 3D awareness of visual foundation models.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.  URL https://doi.org/10.48550/arXiv.2404.08636.
  * Bradski and Grossberg [1994] Gary Bradski and Stephen Grossberg.  Recognition of 3-D objects from multiple 2-D views by a self-organizing neural architecture.  In _From Statistics to Neural Networks: Theory and Pattern Recognition Applications_ , pages 349–375. Springer, 1994.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  ShapeNet: An information-rich 3D model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.  URL https://doi.org/10.48550/arXiv.1512.03012.
  * Chen et al. [2019] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese.  Text2shape: Generating shapes from natural language by learning joint embeddings.  In _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , pages 100–116. Springer, 2019.  URL https://doi.org/10.48550/arXiv.1803.08495.
  * Chen and Zhang [2019] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5939–5948, 2019.  URL https://doi.org/10.48550/arXiv.1812.02822.
  * Collins et al. [2022] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al.  ABO: Dataset and benchmarks for real-world 3D object understanding.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 21126–21136, 2022.  URL https://doi.org/10.48550/arXiv.2110.06199.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3D objects.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.  URL https://doi.org/10.48550/arXiv.2212.08051.
  * Desai et al. [2023] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam.  Hyperbolic image-text representations.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 7694–7731. PMLR, 2023.  URL https://doi.org/10.48550/arXiv.2304.09172.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proc. of the International Conference on Learning Representations (ICLR)_ , 2021.  URL https://doi.org/10.48550/arXiv.2010.11929.
  * Faghri et al. [2018] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler.  VSE++: Improving visual-semantic embeddings with hard negatives.  In _Proc. of the British Machine Vision Conference (BMVC)_ , 2018.  URL https://doi.org/10.48550/arXiv.1707.05612.
  * Frome et al. [2013] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  _Advances in neural information processing systems_ , 26, 2013.  URL https://papers.nips.cc/paper_files/paper/2013/file/7cce53cf90577442771720a370c3c723-Paper.pdf.
  * Fu et al. [2021] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.  3D-FUTURE: 3D furniture shape with texture.  _International Journal of Computer Vision (IJCV)_ , 129:3313–3337, 2021.  URL https://doi.org/10.48550/arXiv.2009.09633.
  * Hamdi et al. [2021] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem.  MVTN: Multi-view transformation network for 3D shape recognition.  In _Proc. of the International Conference on Computer Vision (ICCV)_ , 2021.  URL https://doi.org/10.48550/arXiv.2011.13244.
  * Han et al. [2019] Zhizhong Han, Mingyang Shang, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker.  Y2Seq2Seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences.  In _AAAI Conference on Artificial Intelligence_ , volume 33, pages 126–133, 2019.  URL https://doi.org/10.48550/arXiv.1811.02745.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16000–16009, 2022.  URL https://doi.org/10.48550/arXiv.2111.06377.
  * Huang et al. [2023a] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Visual language maps for robot navigation.  In _International Conference on Robotics and Automation (ICRA)_ , pages 10608–10615. IEEE, 2023a.  URL https://doi.org/10.48550/arXiv.2210.05714.
  * Huang et al. [2023b] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo.  CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training.  In _Proc. of the International Conference on Computer Vision (ICCV)_ , pages 22157–22167, 2023b.  URL https://doi.org/10.48550/arXiv.2210.01055.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  OpenCLIP, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Jatavallabhula et al. [2023] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba.  ConceptFusion: Open-set multimodal 3D mapping.  In _Robotics: Science and Systems (RSS)_ , 2023.  URL https://doi.org/10.48550/arXiv.2302.07241.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 4904–4916. PMLR, 2021.  URL https://doi.org/10.48550/arXiv.2102.05918.
  * Kiros et al. [2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  _arXiv preprint arXiv:1411.2539_ , 2014.  URL https://doi.org/10.48550/arXiv.1411.2539.
  * Lei et al. [2024] Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dylan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou.  VIT-lens: Towards omni-modal representations.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.  URL https://doi.org/10.48550/arXiv.2311.16081.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 19730–19742. PMLR, 2023.  URL https://doi.org/10.48550/arXiv.2301.12597.
  * Lin et al. [2024] Dongyun Lin, Yi Cheng, Shangbo Mao, Aiyuan Guo, and Yiqun Li.  PEVA-net: Prompt-enhanced view aggregation network for zero/few-shot multi-view 3D shape recognition.  _arXiv preprint arXiv:2404.19168_ , 2024.  URL https://doi.org/10.48550/arXiv.2404.19168.
  * Liu et al. [2023a] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su.  OpenShape: Scaling up 3D shape representation towards open-world understanding.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 44860–44879. Curran Associates, Inc., 2023a.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/8c7304e77c832ddc70075dfee081ca6c-Paper-Conference.pdf.
  * Liu et al. [2023b] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3D object.  In _Proc. of the International Conference on Computer Vision (ICCV)_ , pages 9298–9309, 2023b.  URL https://doi.org/10.48550/arXiv.2303.11328.
  * Luo et al. [2024] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.  Scalable 3D captioning with pretrained models.  _Advances in Neural Information Processing Systems_ , 36, 2024.  URL https://doi.org/10.48550/arXiv.2306.07279.
  * Mu et al. [2022] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  SLIP: Self-supervision meets language-image pre-training.  In _Proc. of the European Conference on Computer Vision (ECCV)_ , pages 529–544. Springer, 2022.  URL https://doi.org/10.48550/arXiv.2112.12750.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.  URL https://doi.org/10.48550/arXiv.1807.03748.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  DINOv2: Learning robust visual features without supervision.  _Transactions on Machine Learning Research (TMLR)_ , 2023.  URL https://doi.org/10.48550/arXiv.2304.07193.
  * Qi et al. [2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  PointNet: Deep learning on point sets for 3D classification and segmentation.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 652–660, 2017.  URL https://doi.org/10.48550/arXiv.1612.00593.
  * Qi et al. [2023] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi.  Contrast with reconstruct: Contrastive 3D representation learning guided by generative pretraining.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 28223–28243. PMLR, 2023.  URL https://doi.org/10.48550/arXiv.2302.02318.
  * Qi et al. [2024] Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma.  ShapeLLM: Universal 3D object understanding for embodied interaction.  _arXiv preprint arXiv:2402.17766_ , 2024.  URL https://doi.org/10.48550/arXiv.2402.17766.
  * Qian et al. [2022] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem.  PointNeXt: Revisiting PointNet++ with improved training and scaling strategies.  _Advances in Neural Information Processing Systems_ , 35:23192–23204, 2022.  URL https://doi.org/10.48550/arXiv.2206.04670.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 8748–8763. PMLR, 2021.  URL https://doi.org/10.48550/arXiv.2103.00020.
  * Ruan et al. [2024] Yue Ruan, Han-Hung Lee, Yiming Zhang, Ke Zhang, and Angel X Chang.  TriCoLo: Trimodal contrastive loss for text to shape retrieval.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 5815–5825, 2024.  URL https://doi.org/10.48550/arXiv.2201.07366.
  * Shen et al. [2003] Yu-Te Shen, Ding-Yun Chen, Xiao-Pei Tian, and Ming Ouhyoung.  3D model search engine based on lightfield descriptors.  In _Eurographics (Posters)_ , 2003.
  * Shi et al. [2023] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  MVDream: Multi-view diffusion for 3D generation.  In _Proc. of the International Conference on Learning Representations (ICLR)_ , 2023.  URL https://doi.org/10.48550/arXiv.2308.16512.
  * Song et al. [2023] Dan Song, Xinwei Fu, Weizhi Nie, Wenhui Li, and Anan Liu.  MV-CLIP: Multi-view CLIP for zero-shot 3D shape recognition.  _arXiv preprint arXiv:2311.18402_ , 2023.  URL https://doi.org/10.48550/arXiv.2311.18402.
  * Su et al. [2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller.  Multi-view convolutional neural networks for 3D shape recognition.  In _Proc. of the International Conference on Computer Vision (ICCV)_ , pages 945–953, 2015.  URL https://doi.org/10.48550/arXiv.1505.00880.
  * Tang et al. [2023] Chuan Tang, Xi Yang, Bojian Wu, Zhizhong Han, and Yi Chang.  Parts2words: Learning joint embedding of point clouds and texts by bidirectional matching between parts and words.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6884–6893, 2023.  URL https://doi.org/10.48550/arXiv.2107.01872.
  * Tang et al. [2024] Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen.  MiniGPT-3D: Efficiently aligning 3D point clouds with large language models using 2D priors.  _arXiv preprint arXiv:2405.01413_ , 2024.  URL https://doi.org/10.48550/arXiv.2405.01413.
  * Thomason et al. [2022] Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer.  Language grounding with 3D objects.  In _Conference on Robot Learning_ , pages 1691–1701. PMLR, 2022.  URL https://doi.org/10.48550/arXiv.2107.12514.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.  URL https://doi.org/10.48550/arXiv.1706.03762.
  * Wang et al. [2023] Ye Wang, Bowei Jiang, Changqing Zou, and Rui Ma.  MXM-CLR: A unified framework for contrastive learning of multifold cross-modal representations.  _arXiv preprint arXiv:2303.10839_ , 2023.  URL https://doi.org/10.48550/arXiv.2303.10839.
  * Wei et al. [2020] Xin Wei, Ruixuan Yu, and Jian Sun.  View-GCN: View-based graph convolutional network for 3D shape analysis.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1850–1859, 2020.  URL https://doi.org/10.1109/CVPR42600.2020.00192.
  * Weston et al. [2011] Jason Weston, Samy Bengio, and Nicolas Usunier.  WSABIE: Scaling up to large vocabulary image annotation.  In _International Joint Conference on Artificial Intelligence_ , 2011.
  * Xie et al. [2020] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  PointContrast: Unsupervised pre-training for 3D point cloud understanding.  In _Proc. of the European Conference on Computer Vision (ECCV)_ , pages 574–591. Springer, 2020.  URL https://doi.org/10.48550/arXiv.2007.10985.
  * Xue et al. [2023] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1179–1189, 2023.  URL https://doi.org/10.48550/arXiv.2212.05171.
  * Xue et al. [2024] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  ULIP-2: Towards scalable multimodal pre-training for 3D understanding.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.  URL https://doi.org/10.48550/arXiv.2305.08275.
  * Yang et al. [2022] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19163–19173, 2022.  URL https://doi.org/10.48550/arXiv.2204.03610.
  * Yu et al. [2022a] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  CoCa: Contrastive captioners are image-text foundation models.  _Transactions on Machine Learning Research (TMLR)_ , 2022a.  URL https://doi.org/10.48550/arXiv.2205.01917.
  * Yu et al. [2023] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al.  MVImgNet: A large-scale dataset of multi-view images.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9150–9161, 2023.  URL https://doi.org/10.48550/arXiv.2303.06042.
  * Yu et al. [2022b] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-BERT: Pre-training 3D point cloud transformers with masked point modeling.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19313–19322, 2022b.  URL https://doi.org/10.48550/arXiv.2111.14819.
  * Zhai et al. [2022] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  LIT: Zero-shot transfer with locked-image text tuning.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18123–18133, 2022.  URL https://doi.org/10.48550/arXiv.2111.07991.
  * Zhang et al. [2022a] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  PointCLIP: Point cloud understanding by CLIP.  In _Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8552–8562, 2022a.  URL https://doi.org/10.48550/arXiv.2112.02413.
  * Zhang et al. [2022b] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , pages 2–25. PMLR, 2022b.  URL https://doi.org/10.48550/arXiv.2010.00747.
  * Zhang et al. [2024] Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang.  TAMM: Triadapter multi-modal learning for 3D shape understanding.  In _arXiv preprint arXiv:2402.18490_ , 2024.  URL https://doi.org/10.48550/arXiv.2402.18490.
  * Zhou et al. [2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  iBOT: Image BERT pre-training with online tokenizer.  In _Proc. of the International Conference on Learning Representations (ICLR)_ , 2022.  URL https://doi.org/10.48550/arXiv.2111.07832.
  * Zhou et al. [2024] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang.  Uni3D: Exploring unified 3D representation at scale.  In _Proc. of the International Conference on Learning Representations (ICLR)_ , 2024.  URL https://doi.org/10.48550/arXiv.2310.06773.
  * Zhu et al. [2023] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao.  PointCLIP v2: Prompting CLIP and GPT for powerful 3D open-world learning.  In _Proc. of the International Conference on Computer Vision (ICCV)_ , pages 2639–2650, 2023.  URL https://doi.org/10.48550/arXiv.2211.11682.