# Unified Map Handling for Robotic Systems: Enhancing Interoperability and
Efficiency Across Diverse Environments

James R. Heselden1,2 and Gautham P. Das1,3 *This work was not supported by any
organization1Lincoln Institute of Agri-Food Technology, University of Lincoln,
<EMAIL_ADDRESS><EMAIL_ADDRESS>{0000-0001-5351-9533}

###### Abstract

Mapping is a time-consuming process for deploying robotic systems to new
environments. The handling of maps is also risk-adverse when not managed
effectively. We propose here, a standardised approach to handling such maps in
a manner which focuses on the information contained wherein such as global
location, object positions, topology, and occupancy. As part of this approach,
associated management scripts are able to assist with generation of maps both
through direct and indirect information restructuring, and with template and
procedural generation of missing data. These approaches are able to, when
combined, improve the handling of maps to enable more efficient deployments
and higher interoperability between platforms. Alongside this, a collection of
sample datasets of fully-mapped environments are included covering areas such
as agriculture, urban roadways, and indoor environments.

##  I Introduction

A map can be described as a diagrammatic representation of the ontology of a
space [1]. This encodes the spatial arrangement of physical features. There
are many forms this can take such as: grid maps showing occupancy, to
topological maps showing traversability, and world maps showing explicit
object positions. Maps traditionally show a variety of physical features,
however, modern map formats are used to encode more virtual and meta
information.

For robotics systems, maps are a necessity as such systems are unable to sense
and perceive all the information about a space in real-time. Constructed maps
are used here to represent a subset of information in a manner accessible for
the robot to read and respond to in real-time. Depending on the purpose of the
map, different information can be encoded and stored. In deployments to a new
location, significant time may be taken to create all the necessary maps to
allow it to function.

Many outdoor services can rely on GNSS-based navigation such as used in [2]
and indoor navigation using topologies-based navigation such as in [3]. For
agricultural environments, much of the environment can be viewed from
satellite imagery, so generating maps can be a trivial task by hand. However,
for more complex maps, necessary for more sophisticated activities by
precision agricultural machinery, work must be completed to extract the
information necessary for mapping [4, 5]. In contrast, for indoor environments
such as warehouses, office buildings, and shopping centres, occupancy mapping
is most often completed with SLAM approaches [6].

Consistency between environmental representation is a necessity in the
management and coordination of robotic fleets. Guaranteeing maps are both up-
to-date and consistent across robots is key to ensuring interoperability and
efficiency in dynamic environments. Given a map is a representation of an
environment, if the environment is changed with new static obstacles in place
or new routes opened up, all robots should be aware of these changes, and have
a shared common knowledge. Through the sharing of this knowledge, robots can
be more adaptive to changes, increasing responsiveness and reducing downtime.

Currently, standard practices for handling maps in single-robot deployments is
often ill-defined. This leads to developers storing map files within the same
file-structure as the rest of the codebase. In contrast, with larger fleet
deployments the maps are often stored in a centralised database, on dedicated
infrastructure such as an edge-server. In these instances, the maps are
distributed and updated live. When maps are stored within other code used for
running robot-specific tasks, this can cause problems in consistency as
mentioned above. When spaces are being used by multiple independent robotic
systems, this type of map handling can lead to multiple maps being updated,
with different understandings of the environment. This can lead to problems
such as miscommunication between robots where locations are registered under
different names, or collisions with obstacles when maps are not up to date
with the latest environmental changes.

The main contributions of this work are the open, flexible standardisation of
environment representation; and the demonstration of how such an approach can
both enhance interoperability and improve deployment efficiency through
enabling pipeline development for automating cascaded deployments.
Additionally, the paper contributes an open dataset of sample environments in
a variety of environment structures.

The remainder of this document is as follows. Section II details the types of
information contained within map files, and how various commonly-used maps are
structured. Section III gives an overview of the types of conversion
approaches which have been developed under this work, along with an
introduction into procedural generation approaches. Section IV walks through
the template structure used for the work, and the public datasets being
provided. Finally Section V summarises the findings and contributions of this
work.

##  II Environmental Information

###  II-A Information Types

In this section, we overview some types of information available from an
environment and the manners in which this information is stored and used.

####  II-A1 Location

Location information refers to the location of the space on a wider scale, in
particular, this is the type of information used to transform the location of
a robot through GNSS and apply them to a location within the space’s metric
positioning.

####  II-A2 Objects

Objects are the items which exist within the space, their location, properties
and interactions within the space. This type of information is used to give an
understanding of objects which can be interacted with such as the location of
a door in a room.

####  II-A3 Occupancy

Occupancy is the information about object permanence, that being how the world
is filled by obstacles and where those obstacles couple against the potential
movements of the robot. The position of walls in a 2d space or manner in which
a table occupies a room and the space in which the table does not occupy in a
3D space.

####  II-A4 Topology

The topology of a space detail the connection of regions within the space, how
rooms connect to one another, and how an agent is able to pass through such
regions.

###  II-B Information Encoding

Below are a variety of file types which encode different types of information,
both directly and indirectly. Although this is not an exhaustive list, most of
the variety of information required for autonomous robot operations is
available in these. Table I summarises the types of information stored in
various map types, showing how some information types can be extracted
directly from the files, and how some can be generated directly from the same
files. Notable, is how often, topology can be extracted indirectly, while in
contrast location cannot be extracted without this information being supplied.

TABLE I: Information stored within maps formats, information available in raw form (shown with $-$) and indirect information available without use of complementary files (shown with $+$). Map Type | Location | Objects | Occupancy | Topology | Ref
---|---|---|---|---|---
Datum | $-$ |  |  |  | [1]
OSM Graph | $-$ | $+$ |  | $-$ | [2]
KML/KMZ | $-$ | $+$ |  | $-$ | [2]
OpenRMF |  | $-$ | $+$ | $-$ | [3]
Satellite |  | $-$ | $+$ | $+$ | [4]
Gazebo |  | $-$ | $-$ | $+$ | [5]
Grid Map |  |  | $-$ | $+$ | [6]
Octo Map |  |  | $-$ | $+$ | [7]
NavGraph |  |  |  | $-$ | [8]
TopoMap2 |  |  |  | $-$ | [9]

####  II-B1 KML File

Keyhole Markup Language (KML) is an international standard for use in
rendering geographic information in an Earth-browser [7]. This file format
contains three levels of detail, Points, Polylines and Polygons. Points are
single locations with just a latitude, longitude and elevation, while
Polylines are a series of coordinates, and Polygons are Polylines which
connect at the end. Zipped KMZ files can optionally contain, alongside the KML
file, complementary files such as icons, images, overlays, and COLLADA 3D
models for further detailing in rendering information.

####  II-B2 Datum File

Datum data used for localisation of world coordinates to global coordinates
[8]. This file is formatted in YAML. It contains standard properties such as a
latitude, longitude of the centre coordinate in the local coordinate space
exists within the global coordinate frame. Additionally, it includes a GNSS
fence for defining the boundaries of the region of interest, gmapping map
size, mapviz origin points, and navsat transform points, all for use with
standard ROS systems for utilisation of GNSS-based localisation and mapping.

####  II-B3 Satellite Image

Image file captured from a satellite or drone, or birds-eye view of an outdoor
space, or conversely, a floor plan of an indoor space. Such images can contain
top-down object information and topological data, however such formats require
decoding for extraction.

####  II-B4 Occupancy Grid

Occupancy file used to manage occupancy in 2D space, used with the ROS
navigation stack [9, 10]. Information is stored in two files, an image file in
PGM or PNG and a properties file stored as YAML. The image file shows a grey-
scale or binary image in which occupation is encoded for each cell. In this
file, a scale is used to determine occupancy with white showing free space,
and black showing occupied space. The accompanying properties file is used to
reference (i) the map image file, (ii) the relative position of the image to
the world, (iii) the size which each cell represents in the real world, (iv)
the point on the scale at which a cell should be regarded as occupied, and (v)
the position on the scale at which a cell should be regarded as free.

####  II-B5 Octo Map

Occupancy file used to render occupancy efficiently in three-dimensional space
[11]. Information is stored in binary Octree format which stores space
occupation under a tree where each level of the tree stores the region
subdivided into a 2x2x2 grid in which each sub-division is either fully empty,
fully occupied, or partially occupied containing another sub-tree. Each voxel
can optionally encode RGB colour information. Limiting query-depth can enable
higher performance by reducing the level of detail.

####  II-B6 Open-Street Map

Open-Street Maps content is used for a variety of systems in government and
commercial operations [12]. Information is stored in XML. This map type
details geographic data such as buildings, roads, and city boundaries,
including properties such as names, tower types, speed limits, road surfaces,
population estimates, and a large assortment of properties for different types
of markings. The types of entries are categorised into (i) Nodes which are
single points such as buildings, (ii) Ways which are collections of nodes
connected in a sequence, and (iii) Relations, which are collections of Ways
with a shared relationship.

####  II-B7 NavGraph

Topological file used for basic detailing of networks of permissible routes
within an environment under the FawkesRobotics simulator [13]. Data is stored
in YAML format with information structured in three sections. In (i), default
properties detail the tolerances of how closely the robot should follow a
path. Nodes (ii) define the list of nodes which each have a position, name,
and any special typing which require special behaviours. Connections (iii)
include a list of node name pairs to show paths between nodes.

####  II-B8 Topological Map

Topological file used to enable high-level navigation through environments.
Data is stored in YAML format and detailed in [14]. Information contained
within includes local-space positions of nodes and connections to neighbouring
nodes. Each edge detail restrictions of which robots are permitted awareness
of the edge, and each node details a boundary and tolerance of which the agent
may use to localise.

####  II-B9 OpenRMF Map

Simulation world file for use in the OpenRMF ROS2 Traffic Simulator [15]. Data
is stored in YAML format and contained information is broken down into Crowd
Sim which details dynamic agents such as robots and humans to include into the
simulations. Under Levels, it also details: (i) static obstacles on each floor
of the building, such as models, floors, and walls, (ii) interactive objects
such as doors, and (iii) lanes which robots may pass along, and human lanes
for use by humans.

####  II-B10 Gazebo World

Simulation world file for use in the Gazebo Simulator [16]. Data is in SDF, an
XML-flavour format specifically designed for simulation definitions. Data
includes: (i) static object positions like walls, shelves, and tables, (ii)
dynamic obstacles such as animals or people, (iii) intractable objects such as
doors and elevators, and (iv) environmental properties such as lighting,
weather, and gravity.

(a) Satellite view of region

(b) Datum-based GNSS fence

(c) Occupancy grid

(d) Topological map

Figure 1: Collection of maps rendered using Google Earth using KML.

##  III Information Reformatting

###  III-A Applications of Conversions

With understanding the type of information stored within such map files, we
can find approaches to extract this information reuse in other formats. For
example, by extracting the path information from an OpenRMF file, we can
generate a topological map. We can further supplement available information
with the use of API calls, i.e. we can request a satellite image from an API
when given a datum fence. A satellite image can in turn, be used to extract
object information for use in generating a simulation. Allowing the
construction of pipelines for cascading the generation of environments.

Fig.2 shows the conversion approaches which have been developed thus far as
part of the open-source converters stack in our $environment\\_common$
111https://github.com/LCAS/environment_common structured as a ROS2 package.
Each colour details a different converter, with multiple lines ending in the
same colour indicating a combination of those files to construct the
destination file type. Due to the limited sources of information on location
data, the datum files have been used extensively for localising to the world
space, a necessity for KML generation.

Figure 2: Chord Diagram detailing examples of conversions between different
standard map types. Each line indicates a conversion from the black end of the
line to the destination file. Each conversion process is indicated by a
different destination colour with multiple lines being used together.

Detailed below, are some of the approaches considered for extracting
information from existing files for then combining them into new formats.

In its fundamental form, a topology is a collection of points and a list of
how these points connect together. This information is used to represent the
connections in a space and how a robot can move through the space. This type
of information, as detailed in Table.I, can be generated implicitly through
occupancy information. As shown in [17], this can be done through
morphological skeletonisation in which permissible space in an occupancy grid
is repeatedly thinned to leave only single lines and connecting points.

Occupancy, showing how things within a region reserve space, can be determined
in several ways. In [4], occupancy is implied through the projection of trees
onto an occupancy grid. While in [5], occupancy is determined from satellite
image segmentation. For the generation of occupancy maps from object-based
maps, approaches such as [18] have been developed to slice through simulated
worlds at a given height. In [19], the authors develop an approach to segment
satellite imagery, with effective results in classifying paths and roads. This
type of segmentation could be effective to further extract topology
information with approaches such as in [17].

In the identification of objects, many works have been developed for effective
image segmentation [20]. These, when combined with effective labelling
provided by such works as YOLOv7 [21], and deployed on a drone, can be an
efficient way to collect object information from the real world. Approaches
such as YOLOv7 [21] could also be applied to satellite images to gather this
type of information, albeit in lower quality, from satellite images prior to
visiting the field.

Some information on objects can be obtained from occupancy information,
especially when minor context is given such as what objects could possibly be
within a given space. For instance, if it is known that the deployment is in a
forest, and an occupancy map is offered, then it could be reasonable to assume
obstacles which appear to have a circular occupancy may in fact be trees, and
in an indoor office, that flat edges to the occupancy could be walls.

###  III-B Types of Procedural Generation

When restructuring environmental data for new purposes, not all information
can be obtained through conversions of existing files and use of API calls.
Some pieces of information may be missing, as they were never collected in the
first place. To fill in these pieces of missing data, and to fabricate entire
environments from scratch, procedural generation can be used.

For instance, the conversion of a NavGraph map, to an Topological Map, there
is information about node positions and connections which is retained, however
information on which navigation approach to use for each connection simply
does not exist. This information could be supplemented by hand with generation
of new files for integration, however with the use of template data to auto-
fill the missing fields, this can be skipped.

When template data is not enough, and more realistic and reasonable data must
be generated, procedural generation can be used instead. Detailed below are a
few examples of the types of procedural generation which can be applied to
auto-fill missing data while keeping environments realistic.

####  III-B1 Wave-Function Collapse

Inspired by quantum theory, wave-function collapse works by creating and
collapsing states based on compatibility with a template structure. In
practice, a template is provided describing a likely sample of a full
generation, and the system iteratively generates parts of the environment and
destroys parts which are not in the template sample.

####  III-B2 Cellular Automata

Approaches in Cellular Automata work to generate complex patterns from simple
rules. Specifically, they work by simulating rule-based interactions of cells
on a grid where the state of each cell is dependent on the state changes of
neighbouring cells.

####  III-B3 Gradient Noise

Gradient Noise is often used for generating texture maps. It works by applying
an algorithm over a grid to construct natural-looking patterns. This approach
is used commonly in the forms of Perlin Noise which uses a square-grid
structure, and Simplex Noise which uses a triangle-based grid structure.

####  III-B4 Procedural Modelling

Procedural Modelling is used for constructing types of models with defined
architectural traits. L-Systems are a well-known approach of this which is
primarily used for procedural generation of plants and foliage. It works by
utilising structural rules about how individual components relate to one
another in order to build up simulated architectural models. ArcGIS CityEngine
by Esri is another well-known approach which is used for urban planning and
design, which generates city models using GIS data, or through rule-based
generation.

##  IV Environment Template

The $environment\\_template$ 222https://github.com/LCAS/environment_template
uses a fixed file structure in which files are categorised into their primary
data category. This ensures proper organisation and a tidier workspace. At the
root of this hierarchy is a source file named $environment.sh$. Each file
included within the file structure should also have a standard export included
into this source file. This is to enable flexible file referencing wherein
regardless of the specific file locations within the hierarchy, users can
always identify the file they need through sourcing the statically positioned
source file.

The environment template hierarchy is contained within a ROS2 package format.
This is to enable simple utility, within ROS2 infrastructure projects, and to
enable compatibility with the map conversion scripts which utilise ROS2 for
tidier file management and execution. It is important to stress that the
environment template is not dependent on ROS2 for execution, the template of
choice can be downloaded to any location on the device so long as the source
file is referenced appropriately.

Example datasets of environments have been made public, created under this
template. These are available on the environment template repository, and are
detailed further on the repository wiki. A short breakdown of each is detailed
below:

###  IV-A Agricultural Dataset

The agricultural environments dataset is generated over Riseholme Park Farm
(Fig. 3). The dataset, consists of a collection of fully and partially mapped
regions, covering horticultural fields, pastoral fields, grower plots,
research buildings, parkland, woodlands livestock buildings, along with
connecting roads and footpaths. Further mapping is ongoing, with new maps
being generated alongside new conversion processes.

(a) Full map of Riseholme Park Farm

(b) Central region of Riseholme Park Farm zoomed in

Figure 3: Regions included with the agricultural environment dataset, rendered
with Google Earth. Colours of polygons indicate the type of space, with yellow
showing horticultural fields, green indicating pastoral field, dark green
showing forest, orange showing parkland, blue showing office buildings, cyan
showing animal housing, and purple showing grower plots for specialist crops.
White regions indicating connecting paths and roads across the farm.

###  IV-B Urban Roadway Dataset

The urban roadway environments dataset consists of a collection of
environments generated from the open-street map dataset. It includes a
collection of roadway maps from cities in Asia (Fig. 4(a) \- e.g. Tokyo,
Soeul, Singapore), Europe (Fig. 4(b) \- e.g. London, Madrid, Berlin) and North
America (Fig. 4(d) \- e.g. New York, Los Angeles). Road network in Ixtapa,
converted from OSM to Topological map format and embedded on Satellite imagery
in KML is shown in Fig. 4(c).

(a) Locations across Asia used for urban roadway environments.

(b) Locations across Europe used for urban roadway environments.

(c) Rendering in KML, of OSM data over Ixtapa converted into a topological
map.

(d) Locations across North America used for urban roadway environments.

Figure 4: Locations across the globe in the Urban Roadway Environment dataset.

###  IV-C Indoor Dataset

The indoor environment dataset consists of a collection of indoor environments
some converted from existing sources, and some generated procedurally using
wave-function collapse. It includes environments such as offices, warehouses,
airports and hospitals (Fig. 5).

(a) $10\text{\,}{\mathrm{m}}^{2}$ space

(b) $15\text{\,}{\mathrm{m}}^{2}$ space

Figure 5: Examples warehouse environments generated procedurally using wave-
function collapse. Both examples had been supplied with a
$5\text{\,}{\mathrm{m}}^{2}$ sample template detailing an example of an how an
arrangement of warehouse shelves and free space may appear on a 2D grid. (a)
shows the result of a $10\text{\,}{\mathrm{m}}^{2}$ generated regions, and (b)
shows the result of a $15\text{\,}{\mathrm{m}}^{2}$ generated region.

##  V Conclusion

Mapping is still a major hurdle for the rapid deployment of technologies into
new environments. This challenge is multi-fold when considering the long-term
deployment of heterogeneous robotic fleets that may use different map formats
for their navigation planning and execution.

In this work, we have shown the potential benefits that employing map-handling
standards can have for improving this rapid deployment. Alongside this, we
have detailed the necessity of sharing maps to ensure consistency and
interoperability between robots running different code-bases. Collating all
information related to an environment in one independent package would also
alleviate the challenges of scattered environment information and updates,
especially in large-scale deployments. We have also highlighted the potential
of environment generation pipelines, with direct and indirect information
extraction, complemented with the use of API calls and procedural generation
for determining missing data.

However, this work is not complete without wider usage in the robotics
community and identifying much more efficient ways to ensure interoperability
and information completion. Towards this, we have opened a large dataset of
environments covering agricultural spaces, urban road networks, and
warehouses, which are highly effective for comparative experimental analysis.
We have also open-sourced the standards, with the intent for this dataset to
be expanded further through community development.

## Acknowledgements

This work was supported by AgriFoRwArdS CDT, under the Engineering and
Physical Sciences Research Council [EP/S023917/1]. This work was completed in
association with the Agri-OpenCore project under Innovate UK grant 10041179.

## References

  * [1] Oxford English Dictionary  “Definition of map – English Dictionary”, https://www.oed.com/dictionary/map_n1?tab=meaning_and_use#38012621, 2023
  * [2] Clearpath Robtics  “Web UI Autonomous Mode - OutdoorNav Software”, https://clearpathrobotics.com/assets/manuals/outdoornav/archived/0.6.0/ui_autonomous_mode.html, 2022
  * [3] ANYbotics  “Install Inspection Robot - ANYbotics”, https://www.anybotics.com/install-inspection-robot/#accurate-navigation, 2023
  * [4] Dimitrios Katikaridis et al.  “UAV-supported route planning for UGVs in semi-deterministic agricultural environments”  In _Agron._ 12.8 MDPI, 2022, pp. 1937
  * [5] Luís Carlos Santos et al.  “Occupancy grid and topological maps extraction from satellite images for path planning in agricultural robots”  In _Rob._ 9.4 MDPI, 2020, pp. 77
  * [6] Steve Macenski and Ivona Jambrecic  “SLAM Toolbox: SLAM for the dynamic world”  In _J. of Open Source Software_ 6.61, 2021, pp. 2783
  * [7] Open Geospatial Consortium  “KML - Open Geospatial Consortium”, https://www.ogc.org/standard/kml/, 2024
  * [8] Tom Moore  “navsat_transform_node — robot_localization 2.3.1 documentation”, https://docs.ros.org/en/jade/api/robot_localization/html/navsat_transform_node.html, 2016
  * [9] Eitan Marder-Eppstein  “move_base - ROS Wiki”, https://wiki.ros.org/move_base, 2023
  * [10] ROS Navigation Working Group  “Nav2 Documentation”, https://navigation.ros.org/, 2024
  * [11] Armin Hornung et al.  “OctoMap: An efficient probabilistic 3D mapping framework based on octrees”  In _Auton. Rob._ 34 Springer, 2013, pp. 189–206
  * [12] OpenStreetMap-ROS  “route_network - ROS wiki”, http://wiki.ros.org/route_network, 2021
  * [13] Fawkes Robotics  “fawkesrobotics/fawkes_msgs: ROS messages for Fawkes integration”, https://github.com/fawkesrobotics/fawkes_msgs, 2018
  * [14] Lincoln Centre for Autonomous Systems (LCAS)  “Topological Navigation”, 2023  URL: https://www.github.com/LCAS/topological_navigation
  * [15] open-rmf  “ROS Package: rmf_traffic_editor”, https://index.ros.org/p/rmf_traffic_editor, 2023
  * [16] Open Robotics  “Gazebo”, https://gazebosim.org/home, 2024
  * [17] Luís Santos et al.  “Path planning approach with the extraction of topological maps from occupancy grid maps in steep slope vineyards”  In _2019 IEEE Int. Conf. on Auton. Rob. Sys. and competitions (ICARSC)_ , 2019, pp. 1–7  IEEE
  * [18] Marina Kollmitz  “gazebo_ros_2Dmap_plugin”  In _GitHub repository_ GitHub, https://github.com/marinaKollmitz/gazebo_ros_2Dmap_plugin, 2023
  * [19] N Bharatha Devi, A Celine Kavida and R Murugan  “Feature extraction and object detection using fast-convolutional neural network for remote sensing satellite image”  In _J. Indian Soc. Remote Sens._ 50.6 Springer, 2022, pp. 961–973
  * [20] Alexander Kirillov et al.  “Segment anything”  In _Proc. IEEE/CVF Int. Conf. on Comput. Vision_ , 2023, pp. 4015–4026
  * [21] Chien-Yao Wang, Alexey Bochkovskiy and Hong-Yuan Mark Liao  “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors”  In _Proc. IEEE/CVF Conf. on Comput. vision and pattern Recognit._ , 2023, pp. 7464–7475


Figure 1. Left: a gallery of 3D shapes generated by our approach. Right: a
style-consistent indoor scene produced by our approach from a text
description.

# EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape
Generation

Zhengzhe Liu, Jingyu Hu, Ka-Hei Hui  The Chinese University of Hong KongHong
Kong<EMAIL_ADDRESS>,  Xiaojuan Qi† The University of Hong
KongHong Kong<EMAIL_ADDRESS>,  Daniel Cohen-Or  Tel-Aviv UniversityIsrael
<EMAIL_ADDRESS>and  Chi-Wing Fu† The Chinese University of Hong KongHong
Kong<EMAIL_ADDRESS>
###### Abstract.

†††: Corresponding authors

This paper presents a new text-guided technique for generating 3D shapes. The
technique leverages a hybrid 3D shape representation, namely EXIM, combining
the strengths of explicit and implicit representations. Specifically, the
explicit stage controls the topology of the generated 3D shapes and enables
local modifications, whereas the implicit stage refines the shape and paints
it with plausible colors. Also, the hybrid approach separates the shape and
color and generates color conditioned on shape to ensure shape-color
consistency. Unlike the existing state-of-the-art methods, we achieve high-
fidelity shape generation from natural-language descriptions without the need
for time-consuming per-shape optimization or reliance on human-annotated texts
during training or test-time optimization. Further, we demonstrate the
applicability of our approach to generate indoor scenes with consistent styles
using text-induced 3D shapes. Through extensive experiments, we demonstrate
the compelling quality of our results and the high coherency of our generated
shapes with the input texts, surpassing the performance of existing methods by
a significant margin. Codes and models are released at
https://github.com/liuzhengzhe/EXIM.

3D shape generation, text-guided

††submissionid: 158††ccs: Computing methodologies Shape analysis††ccs:
Computing methodologies Neural networks††ccs: Computing methodologies Mesh
models††copyright: acmlicensed††journal: TOG††journalyear:
2023††journalvolume: 42††journalnumber: 6††article: 228††publicationmonth:
12††price: 15.00††doi: 10.1145/3618312

##  1\. Introduction

3D shape generation has a wide range of applications, including CAD, extended
reality, games, and movies. To create 3D shapes, a highly natural and
intuitive approach is to use natural language or texts. Text-guided 3D shape
generation can improve the efficiency of 3D modeling, allowing non-experts to
efficiently create shapes by providing a text description of the desired
shape.

To enable text-guided 3D shape generation, existing works (Jain et al., 2022;
Mohammad Khalid et al., 2022; Poole et al., 2023; Lin et al., 2023) propose to
leverage large-scale pre-trained vision-language models such as CLIP (Radford
et al., 2021) and text-to-image generation models (Ramesh et al., 2022;
Saharia et al., 2022; Rombach et al., 2022) to optimize 3D shapes at test
time. Yet, the optimization is time-consuming, _e.g._ , 72 min. for Dream
Fields (Jain et al., 2022) and 90 min. for Dream Fusion (Poole et al., 2023)
to generate a single 3D shape. Also, due to the lack of awareness of the 3D
prior, their generative fidelity is still not satisfactory,  _e.g._ , with
unrealistic structures; see Figure 6 in “Stable-Dreamfusion” (Poole et al.,
2023).

Another line of works (Chen et al., 2018; Liu et al., 2022; Sanghi et al.,
2022; Fu et al., 2022; Wei et al., 2023; Cheng et al., 2023; Jun and Nichol,
2023) aims to generate shapes without test-time optimization. By learning the
3D prior from a shape collection,  _e.g._ , ShapeNet (Chang et al., 2015),
shapes can be generated with great efficiency in a feed-forward manner. So,
these approaches particularly suit generating shapes in specific classes, such
as room assets.

However, they still face several challenges. One major challenge arises from
the need for appropriate 3D representations for text-guided 3D generation.
Some works adopt explicit representations,  _e.g._ , voxel grids (Chen et al.,
2018; Mittal et al., 2022) and point clouds (Nichol et al., 2022b); yet, they
cannot represent fine details due to their representation capabilities (See
Figure 6: Point-E (Nichol et al., 2022b)). Others utilize implicit neural
representations (Liu et al., 2022; Sanghi et al., 2022; Fu et al., 2022) to
first learn a latent feature vector/grid of the shape, then train an MLP to
predict the occupancy of the query point. However, their generative quality is
still far from satisfactory,  _e.g._ , with incomplete structures (see Figure
6: TITG3SG (Liu et al., 2022), CLIP-Forge (Sanghi et al., 2022), and
ShapeCrafter (Fu et al., 2022)). Moreover, the implicit representations used
in these approaches encode global (non-local) features during the encoding
process. As a result, achieving spatial disentanglement becomes challenging,
making it difficult to perform shape manipulation while leaving other regions
unaffected. Further details will be presented in Section 4.

Color is an important element in text-to-shape generation; yet, simply
adopting existing 3D shape representations like voxel grid (Mittal et al.,
2022) and implicit fields (Sanghi et al., 2022; Fu et al., 2022) cannot handle
color generation. An early work (Chen et al., 2018) attempts to represent
colors with voxels; its coarse results demonstrate that explicit
representation like voxels is not a good choice for color representation. In
more recent developments, researchers have explored the utilization of a joint
shape-color implicit representation (Liu et al., 2022) or adopt models for
unconditional 3D shape generation like GET3D (Gao et al., 2022). However,
their generative qualities are still not satisfying; see Figure 6 “TITG3SG”
and “TAPS3D”. More importantly, the color of 3D shapes should be conditioned
on the shape to ensure shape-color consistency; yet, in their methods, shape
and color are predicted in parallel, so it is challenging to manipulate one
while keeping the other consistent.

In this work, we present a new representation called EXIM, a hybrid explicit-
implicit representation for high-fidelity text-guided 3D shape generation,
enabling a certain level of local modification. It is a two-stage coarse-to-
fine pipeline. First, we adopt the explicit compact voxel representation
inspired by (Hui et al., 2022) to create a coarse shape by predicting TSDF
values, then locally condition on the stage-1 output to enhance the shape
details and also to produce colors based on the text descriptions using neural
implicit representations. As Figure 1 shows, our design is capable of
generating high-fidelity 3D shapes from textual descriptions.

Further, EXIM is flexible. First, our explicit representation is spatially
disentangled, allowing the operations to focus on the local regions of
interest, while avoiding the other regions. Second, EXIM decomposes shape and
color generation, in which the color prediction is conditioned on the shape.
This design encourages shape-color consistency when modifying the shape and
color, while enabling independent modification on shape and color.

Lastly, to enrich the existing text-to-shape datasets, we present a study
utilizing pre-trained vision-language foundation models, such as BLIP (Li et
al., 2023b), to generate textual descriptions for 3D shapes based on rendered
images. This enables us to effortlessly expand the existing dataset,
incorporating new categories without the need for extensive human annotations.

To evaluate our approach, we conduct experiments on multiple categories of
ShapeNet (Chang et al., 2015). Both qualitative and quantitative results
demonstrate that EXIM outperforms existing works by a large margin in terms of
generative fidelity and modification capability. Further, our approach can
create style-consistent and plausible indoor scenes through generating shapes
using text descriptions, as demonstrated in Figure 1.

##  2\. Related Work

##### 3D Shape Representation and Generation

There are many ways to represent a 3D shape, such as point clouds (Li et al.,
2021; Hui et al., 2020; Vahdat et al., 2022), voxels (Wu et al., 2016; Smith
and Meger, 2017), meshes (Tang et al., 2019), and neural implicit functions
(Mescheder et al., 2019; Chen and Zhang, 2019; Shue et al., 2022; Zhang et
al., 2023). Among the representations, the voxel-based explicit representation
effectively captures the shape topology, but high-resolution voxels require
substantial memory. To address this issue, (Hui et al., 2022) proposes
compressing the SDF volumes into a compact wavelet representation to retain
more details by representing shapes in a compressed domain. Neural implicit
functions offer a highly compact representation and excel at capturing surface
details (Chibane et al., 2020; Mittal et al., 2022; Yan et al., 2022; Zhang et
al., 2022) and colors (Oechsle et al., 2019; Niemeyer et al., 2020). In this
work, we propose EXIM, a novel hybrid explicit-implicit representation for
text-guided 3D shape generation, unifying the strengths of explicit and
implicit representations. Our new 3D shape representation achieves high
generative fidelity and supports a certain level of local shape and color
modification, as demonstrated in the various results shown in Section 4.

##### Text-Guided 3D Shape Generation

Existing works on text-guided 3D shape generation can be broadly categorized
into two branches: optimizing the shape at test time (Jain et al., 2022;
Michel et al., 2022; Chen et al., 2022; Mohammad Khalid et al., 2022; Liu et
al., 2023; Poole et al., 2023; Lin et al., 2023; Chen et al., 2023) and
generating in a feed-forward manner (Chen et al., 2018; Mittal et al., 2022;
Liu et al., 2022; Sanghi et al., 2022; Fu et al., 2022; Sanghi et al., 2023;
Li et al., 2023a; Wei et al., 2023; Cheng et al., 2023; Jun and Nichol, 2023).
Although the former branch covers a relatively wide range of generative
categories, the latter branch generally offers better generative fidelity,
efficiency, and stability. Consequently, we choose to explore this direction
to develop our method. As discussed in Section 1, existing works in the latter
branch still exhibit various limitations, such as inferior generative fidelity
and limited capability of modifying the generated shapes. Besides, some
existing works focus on text-to-NeRF (Neural Radiance Fields) (Jiang et al.,
2023; Wang et al., 2022a; Li et al., 2022) and avatar generation (Hong et al.,
2022). Unlike the objective of this work, they primarily concentrate on the
image rendering qualities using NeRF rather than shape generation quality.

##### Text-Guided 3D Shape Modification

Compared to text-guided 3D shape generation, text-guided 3D shape modification
has been relatively less explored. (Liu et al., 2022) proposes a two-way
cyclic consistency loss for editing shape or color. Yet, this method struggles
to achieve precise local modification, even with a complex framework, as the
color and shape are predicted in parallel, leading to potential color
mismatches with the 3D shapes. More recently, ShapeCrafter (Fu et al., 2022)
introduces an auto-regressive approach to progressively modify the shape by
successively adding text. Nevertheless, they may not accurately perform
region-aware modifications,  _i.e._ , unrelated regions can be significantly
affected by their editing operations, as shown in Figure 8. Also, the fidelity
and text consistency of both works are not satisfactory; see the results in
Section 4. Beyond existing works, our EXIM is a new hybrid explicit-implicit
representation, offering explicit shape representations and shape-color
disentanglement for a certain level of local modification, as well as
independently modifying the color and geometry.

##  3\. Methodology

Figure 2. Overview of our approach. (a) The Explicit Diffusion Stage (inspired
by (Hui et al., 2022)) transforms the truncated signed distance field (TSDF)
of the training shape $S$ into compact wavelet volumes $C$ and utilizes a 3D
diffusion model with the U-Net architecture to learn the coarse shape
$S_{\text{coarse}}$ conditioned on the text description $D$ with cross-
attention modules. (b) The Implicit Refinement Stage refines the coarse shape
$S_{\text{coarse}}$ to produce the fine shape $S_{\text{fine}}$ and generates
colors on the surface to create the final result $S$. It employs a 3D
convolutional encoder to extract features from multi-level feature maps and
predicts the occupancy and RGB values at each query location $p$. The color
prediction is also conditioned on the text $D$ using cross-attention modules.

###  3.1. Overview

Given a text description $D$, our goal is to generate 3D shapes $S$ with
colors, following $D$. We design a new hybrid explicit-implicit shape
representation, namely EXIM, by incorporating a two-stage generation pipeline.
The key in our approach is to combine the strengths of (i) the voxel-based
explicit 3D representation for topology generation and flexible modification
and (ii) the neural implicit representation for detail enhancement and color
generation.

Figure 2 presents our framework. In the first stage, the explicit stage, we
employ a 3D diffusion model in a compact wavelet domain to generate coarse
shapes in the voxel space $S_{\text{coarse}}$ according to $D$. This process
roughly outlines the 3D shape and enables subsequent shape modification; see
Figure 2 “Explicit Diffusion Stage”. In the second stage,  _i.e._ , the
implicit stage, based on $S_{\text{coarse}}$ and text description $D$, we
design an encoder network to extract multi-scale text-aware shape features. We
then exploit the neural implicit function to predict the details and dress up
the generated shape $S_{\text{fine}}$ with colors, yielding the final output
$S$; see Figure 2 “Implicit Refinement Stage”. The two stages complement each
other, enabling high-fidelity text-guided 3D shape generation. Further, our
hybrid representation offers an explicit voxel representation and a
disentangled shape and color space, allowing for a higher degree of localized
and independent modification of shape and color within a region of interest;
see Figure 4. Lastly, we leverage a large vision-language foundation model to
extend text-to-shape generation to handle new categories of shapes without
requiring human annotations, enabling a novel application of composing
generated shapes for style-consistent room-level scene generation; see Figure
11.

###  3.2. Text-Guided 3D Shape Generation

In the following, we elaborate on our two-stage explicit-implicit model
designed for text-guided 3D shape generation.

####  3.2.1. Explicit Diffusion Stage

In accordance with the text description $D$, the explicit stage generates
coarse 3D shapes $S_{\text{coarse}}$ using dense voxel grids of $256^{3}$ by
predicting the TSDF value for each voxel grid. To accomplish this, we leverage
the powerful diffusion model for text-guided 3D shape generation due to its
exceptional generative capabilities (Ramesh et al., 2022; Vahdat et al.,
2022). Instead of employing the original shape space $S$, which is memory and
computation-intensive, we adopt a compact wavelet space to perform the
diffusion process, as suggested in (Hui et al., 2022).

By utilizing biorthogonal wavelet filters (Cohen, 1992), the TSDF volume $S$,
which has a size of $256^{3}$, is transformed into a compact, low-frequency
wavelet volume $C_{0}$ with a size of $46^{3}$. By applying an inverse
transform, $C_{0}$ can be used to reconstruct the topology of $S$ while
omitting high-frequency details. Subsequently, a diffusion model is employed
to generate $C_{0}$, which can then be converted back to $S$. To accomplish
this objective, we propose a transformer-based architecture that conditions
the model on the text description $D$, extending the approach in (Hui et al.,
2022). In the following sections, we will first provide preliminaries of
diffusion models and then discuss our proposed model architecture.

Given a wavelet volume denoted as $C_{0}$ for a sample, the forward diffusion
process incrementally adds Gaussian noise over $T$ steps, resulting in the
sequence $\\{C_{0},C_{1},...,C_{T}\\}$, where $C_{T}$ eventually becomes pure
Gaussian noise following a normal distribution. Specifically, the noise data
sample $C_{t}$ can be obtained from the following conditional probability
density function:

(1)
$q\left(\mathbf{C}_{t}\mid\mathbf{C}_{0}\right)=\mathcal{N}\left(\mathbf{C}_{t};.\sqrt{\bar{\alpha}_{t}}\mathbf{C}_{0},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right),\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{t}),$

where $t$ is the time step and $\beta_{t}$ is a pre-defined noise parameter
that controls the amount of noise added at step $t$; $\beta_{t}$ increases as
$t$ increases (with $\beta_{T}$ = 1). A diffusion model is then trained to
reverse this process and recover $C_{0}$. It learns a deep denoising network
$\epsilon_{\theta}$, which predicts the noise added at each step according to
the corresponding text description $D$. Specifically, at each step $t$, The
network is trained to predict the noise $\epsilon$ using an $L_{2}$ loss:

(2)
$\mathcal{L}_{\text{coarse}}=E_{t,C_{0},\epsilon}[\left\|\epsilon-\epsilon_{\theta}(C_{t},t,D\right\|^{2}],\epsilon\sim\mathcal{N}(0,\mathbf{I}),$

where $t$ is the time index, $D$ is the text description, and
$\epsilon_{\theta}$ is a 3D U-Net designed for denoising. After training,
during the inference stage, given text description $D$, we randomly sample
noise $C_{T}$ and apply $\epsilon_{\theta}$ for $T$ iterations to
progressively denoise the input and generate sample $C_{0}$, following the
approach in (Song et al., 2021). $C_{0}$ is further transformed back to the
shape space to yield $S_{\text{coarse}}$.

In the following, we elaborate on $\epsilon_{\theta}(C_{t},t,D)$, which takes
$C_{t}$, time step $t$, and a text description $D$ as inputs and produces the
noise for denoising $C_{t}$ into $C_{t-1}$ as shown in Figure 2 (a). Given the
text description $D$, we employ text encoder $E$ based on the CLIP-text
encoder architecture (Radford et al., 2021) to map it into word-level features
$f_{D}\in\mathbb{R}^{L\times d}$, where $L$ is the number of words in $D$ and
$d$ is the feature dimension. Next, given $C_{t}\in\mathbb{R}^{46\times
46\times 46}$ and $f_{D}$ at each time step $t$, a 3D U-net with three
downsampling layers and three upsampling layers is designed to produce the
output. The core of the 3D U-net is a fusion module that combines the
intermediate features from the U-net and the text feature $f_{D}$ to encourage
the generation of text-aligned 3D shapes. The fusion module adopts a cross-
attention mechanism (Vaswani et al., 2017; Rombach et al., 2022) to learn the
correlation between each word and each spatial location in the feature map and
embeds the text features into spatial features as follows:

(3)
$f^{\prime}=\text{softmax}(\frac{W_{Q}(f)W_{K}(f_{D})^{T}}{\sqrt{d}})W_{V}(f_{D}),$

where $f$ represents an intermediate feature map from the U-net with a spatial
size of $H\times W$ and a feature dimension $d$. The matrices $W_{Q}$,
$W_{K}$, and $W_{V}$ embed the input into query, key, and value, respectively.
By doing so, the text information is locally embedded at spatial locations
with high correlation, resulting in the text-aware local feature $f^{\prime}$
that facilitates local modification while ensuring text-shape consistency. The
feature $f^{\prime}$ is further fed into the next layer in the U-net. In our
model, the fusion module is incorporated at the second and third downsampling
layers as well as the first and second upsampling layers of our backbone
architecture.

####  3.2.2. Implicit Refinement Stage

Figure 3. The Implicit Refinement Stage (green) improves the stage-1 output
(brown), particularly on the details. For a more clear comparison, color
prediction is disabled.

Although the Explicit Diffusion Stage can generate plausible 3D shapes
$S_{\text{coarse}}$, the quality of the generated details is not entirely
satisfactory, as high-frequency components are discarded, limiting the model’s
upper-bound performance; see Figure 3. To address this issue, we propose an
implicit network that compensates for the Explicit Diffusion Stage by
producing high-frequency details and enabling color generation.

Specifically, given the TSDF volume of the generated 3D shape
$S_{\text{coarse}}$ from the explicit diffusion stage, we design a 3D
convolutional encoder (Chibane et al., 2020) for shape refinement, which
extracts multi-scale hierarchical feature maps
$\\{f_{s_{1}},\cdots,f_{s_{4}}\\}$ from $S_{\text{coarse}}$ using several
convolution layers and four downsampling layers. Then, for a given query point
$p_{i}$, we obtain its associated feature vector from different scales of
feature maps and concatenate them into a feature vector
$f_{s}(p_{i})=[f_{s_{1}}(p_{i}),\cdots,f_{s_{4}}(p_{i})]$. Following this,
$f_{s}(p_{i})$ is processed through three fully connected layers, with a
sigmoid activation function at the final layer to predict the probability of
the occupancy value being $1$: $o(p_{i})\in(0,1)$. To train the network, we
encourage the predicted occupancy values to match the ground-truth values
using the sigmoid cross-entropy loss:

(4)
$\mathcal{L}_{\text{fine}}=-\big{(}\sum_{i}\hat{o}(p_{i})\log(o(p_{i}))+(1-\hat{o}(p_{i}))\log(1-o(p_{i}))\big{)},$

where $\hat{o}(p_{i})$ indicates ground-truth occupancy value of $p_{i}$.

For the color generation branch, we employ a 3D U-net to extract multi-scale
color feature maps and incorporate the cross-attention module proposed in the
explicit stage to inject word-level text features into the generated
intermediate feature maps. This results in text-aware feature maps
$\\{f_{c_{1}},\cdots,f_{c_{4}}\\}$, promoting color consistency with the text
description $D$. Then, for a given query point $p_{i}$, we concatenate
$\\{f_{c_{1}}(p_{i}),\cdots,f_{c_{4}}(p_{i})\\}$, the global text feature
$f_{D}$ from the text encoder to enhance the color and text consistency, and a
random noise vector $z\sim\mathcal{N}(0,\mathbf{I})$ to encourage the
generation of diverse outputs. The concatenated feature vector $f_{c}(p_{i})$
is further fed into three fully-connected layers to produce the RGB values at
query point $p_{i}$. To train the color generation branch, we randomly sample
a camera pose $[R|T]$ and render images $I$ and $\hat{I}$ based on the
predicted colored shape and the ground-truth colored shape, respectively,
using a differentiable renderer (Niemeyer et al., 2020). To train the model,
we employ the regression loss $L_{c}$ and the adversarial loss
$L_{\text{adv}}$ as follows:

(5)
$\mathcal{L}_{c}=\sum_{i}|I(p_{i})-\hat{I}(p_{i})|,\quad\text{and}\quad\mathcal{L}_{\text{adv}}=\mathbb{E}_{I}[\log(1-D_{\phi}(I))],$

where $D_{\phi}$ represents the discriminator with parameters $\phi$. During
the generative training stage, which utilizes $\mathcal{L}_{c}$ and
$\mathcal{L}_{\text{adv}}$ to optimize the color generation branch, $D_{\phi}$
remains frozen. The discriminator is trained by maximizing $(\log
D_{\phi}(\hat{I})+\log(1-D_{\phi}(I)))$. The adversarial loss is employed to
encourage the model to produce realistic results based on the given text
description $D$.

As illustrated in Figure 3, our Implicit Refinement Stage effectively captures
fine details, including long, thin structures. Moreover, this stage aids the
generation of plausible colors that are consistent with both the shape $S$ and
the text description $D$.

###  3.3. Text-Guided 3D Shape Modification

Figure 4.  Our pipeline for modifying (a) shape and (b) color. For shape, we
expand the diffusion model to include two additional input channels, the mask
$M$ and the unmasked partial shape $S^{\prime}$. We then fine-tune the model
to fill in the masked areas. For color, we can either incorporate the new text
features into the intermediate feature maps or directly generate color values
within the regions specified by $M$, without the need for fine-tuning.  Figure
5.  Top: directly combining the original and generated shapes may result in
size and style inconsistencies, whereas our inpainting-based method
effectively utilizes the context to produce consistent outcomes. Bottom: when
incorporating new parts, merely copying the color from the given shape may
cause noisy color predictions, while our shape-conditioned color generation
generates more plausible colors.

Given text description $D$, our goal is to modify input shape $S$ to align it
with text $D$ while preserving the unrelated regions. Our hybrid explicit-
implicit representation, namely EXIM, provides the required flexibility for
this task, as it defines a 3D topology in the explicit phase, refines the
shape details, and decouples the shape and color generation. This allows us to
perform a certain level of localized shape modification and also independent
editing of color and shape, according to the text description $D$.

Shape modification primarily occurs in the explicit stage, as it generates and
determines the shape’s topology explicitly, allowing for the injection of
certain local shape information, It is important to note that the wavelet
volume corresponds locally to the original shape, meaning that modifying the
data in wavelet domain is equivalent to that modifying the data in the
original domain. Figure 4 illustrates how we may modify the shape and color.
First, given 3D shape $S$ and text description $D$, we generate mask $M$ that
localizes the regions of interest to be modified. To create mask $M$, one can
either manually select/refine an existing mask or generate it using our pre-
trained explicit diffusion model in Section 3.2, following DiffEdit (Couairon
et al., 2023); details are given in the supplementary material. Second, with
mask $M$, a naive approach for modifying the shape would be to directly
generate a new shape $\bar{S}$ using text $D$ and combine it with the existing
shape $S$ based on $M$. However, size and style may not be consistent, as seen
in Figure 5 top (b).

To this end, we adapt our pre-trained explicit diffusion model for text-guided
shape inpainting by augmenting the input wavelet volume $C_{t}$ with two
additional channels: mask $M$ and shape $S^{\prime}$, which represents the
original shape $S$ with the masked areas excluded. We initialize the two
additional channel weights to zero and use the pre-trained network weights to
initialize the remaining parameters. Then, we fine-tune the model to paint the
masked areas and produce coherent manipulated shapes according to text $D$.
During the fine-tuning, we simulate mask $M$ using the unsupervised part-
segmentation approach (Chen et al., 2019) to produce part-level masks and
train the model to paint the masked regions based on the original text $D$ to
emulate the scenarios at inference. After the training, the model can enable a
certain level of localized shape modification, according to the text and
generate coherent shapes, as demonstrated in Figure 5 top (c).

The edited shape $\hat{S}$ is further processed by the Implicit Refinement
Stage for detail refinement and color generation. Notably, our implicit
refinement stage is conditioned on shape $\hat{S}$, enabling the production of
colors consistent with the edited 3D shape. As depicted in Figure 5 bottom
(c), our method creates plausible colors that align with the manipulated
shape, while other approaches fail to produce well-aligned colors, as seen in
Figure 5 bottom (b).

Furthermore, our decoupled design for coarse shape and color generation
enables us to independently manipulate color without affecting the shape.
Given mask $M$ and text description $D$ for color editing, we integrate the
new text features into the intermediate feature maps within the regions
determined by $M$, producing colors in accordance with text $D$. This allows
us to manipulate color within the regions of interest without impacting the
shape or other areas, as demonstrated in Figure 4 “color modification”.

###  3.4. Applications on New Categories and Indoor Scene Generation

Our model requires paired text-shape data (Chen et al., 2018) for training,
which is challenging to obtain through human annotations. Therefore, we
further explore the possibility of expanding existing text-to-shape
annotations to new categories by leveraging pre-trained vision-language
foundation models, such as BLIP-v2 (Li et al., 2023b). Specifically, we render
an image for a given shape and employ a foundation model (_i.e._ , BLIP-v2 (Li
et al., 2023b)) for image captioning to generate a pseudo text description of
the shape. Despite its simplicity, this strategy can effectively generate
descriptions such as “The chair is a modern, two-seater sofa with a metal
frame and a red fabric seat”. More analysis of the quality of the pseudo
annotations is provided in the supplementary material. This approach allows us
to expand the text-to-shape dataset (Chen et al., 2018) with new categories,
further enabling novel applications in scene-level generation, as described
below.

We investigate the potential of using text-to-shape generation to create
style-consistent room-level scenes by composing generated 3D shapes. Given the
generated shapes for room assets, such as tables, chairs, sofas, and cabinets,
we employ a 2D room layout map, which can be obtained using existing methods
(Paschalidou et al., 2021) or human design, to determine the object location,
orientation, and size. Subsequently, the shapes are arranged accordingly to
generate a room. It is crucial to note that we can promote the production of
style-consistent shapes by using text prompts with similar descriptions for
desired color/shape styles. Our preliminary attempt demonstrates that our
method can effectively facilitate the creation of indoor scenes with high
consistency, as shown in Figure 11.

Figure 6.  Swivel chairs generated by various approaches. Top row: results
from existing works. The outcomes of Stable-Dreamfusion (Tang, 2022) and
Point-E (Nichol et al., 2022b) are obtained using their official codes, while
the results from other methods are directly taken from their respective
papers. Bottom row: our method. High resolution figures are provided in the
supplementary material.  Figure 7. Comparison with TITG3SG (Liu et al., 2022)
and TAPS3D (Wei et al., 2023). High resolution figures are provided in the
supplementary material.

##  4\. Experiments

###  4.1. Dataset and Implementation Details

We train our framework on five ShapeNet categories (Chang et al., 2015),
including chair, table, sofa, cabinet, and airplane. For the chair and table
categories, we train one model with pseudo annotations and another with
annotated texts provided by (Chen et al., 2018) to show the efficacy of the
generated pseudo annotations. For the remaining categories, we train models
using pseudo annotations. Prior to model training, we convert the mesh to TSDF
and subsequently to wavelet volumes, following (Hui et al., 2022). Initially,
we train the Explicit Diffusion Stage for 2,000 epochs using the Adam
optimizer and then train the Implicit Optimization Stage additionally for 200
epochs with an initial learning rate of $1e^{-4}$. For shape modification, we
expand the input to three channels and fine-tune the model for 500 epochs
using the same learning rate. Following (Hertz et al., 2022; Luo and Hu, 2021;
Hui et al., 2022), we train a separate model for each category. We employ
classifier-free diffusion guidance (Ho and Salimans, 2022) in both the
Explicit Diffusion Stage and shape modification fine-tuning. Empirically, we
set the guidance weight $g=3$. To demonstrate the efficacy of our approach, we
primarily conduct experiments on the chair category due to its high complexity
and diverse topologies and colors.

###  4.2. Comparisons with Existing Works

##### Text-guided 3D Shape Generation

To evaluate the performance of our EXIM, we compare it with existing works
both qualitatively and quantitatively. As illustrated in Figure 6, our
approach significantly outperforms all the existing works (Sanghi et al.,
2022; Mittal et al., 2022; Liu et al., 2022; Fu et al., 2022; Li et al.,
2023a; Sanghi et al., 2023; Poole et al., 2023; Liu et al., 2023; Nichol et
al., 2022b; Wei et al., 2023). Specifically, CLIP-Forge, Auto-SDF, Diffusion-
SDF, CLIP-Sculpture, ISS, and Point-E cannot generate high-quality structured
details,  _e.g._ , wheels. TITG3SG and ShapeCrafter produce asymmetrical arms.
Stable-Dreamfusion produces an unrealistic seat and requires long-time
optimization, whereas TAPS3D creates unnatural textures. Also, none of the
above existing works can create high-quality and smooth surfaces. On the
contrary, our model is able to generate diverse, high-quality 3D shapes in
terms of structured details, colors, and surfaces. Moreover, Figure 7
demonstrates our model’s diversified generative capability from a single given
text description, surpassing (Liu et al., 2022; Wei et al., 2023) in terms of
both shape and color.

To quantitatively compare our EXIM with the existing works, we adopt CLIP-S,
following (Fu et al., 2022), to measure the text-shape consistency and use FPD
and FID (Wei et al., 2023; Heusel et al., 2017) to evaluate the generative
quality of shape and color, respectively. For a fair comparison, we adopt the
results of existing works reported in (Fu et al., 2022) and (Wei et al.,
2023). Table 1 shows that our model consistently outperforms the existing
works across all metrics, indicating its superior performance in terms of
generative quality and text consistency. “N.A.” means the result is not
reported in (Fu et al., 2022) and (Wei et al., 2023).

Table 1. Comparison with existing works.

Method |  CLIP-S ($\uparrow$) |  FPD ($\downarrow$) |  FID ($\downarrow$)
---|---|---|---
Text2Shape (Chen et al., 2018) |  16.27 |  N.A. |  N.A.
CLIP-Nerf (Wang et al., 2022a) |  N.A. |  N.A. |  48.4
CLIP-Forge (Sanghi et al., 2022) |  26.34 |  825.96 |  N.A.
TITG3SG (Liu et al., 2022) |  38.88 |  1566.76 |  N.A.
AutoSDF (Mittal et al., 2022) |  48.92 |  N.A. |  N.A.
ShapeCrafter (Fu et al., 2022) |  52.43 |  N.A. |  N.A.
TAPS3D (Wei et al., 2023) |  N.A. |  342.23 |  43.7
EXIM (Ours) |  71.75 |  192.17 |  36.8

##### Text-guided 3D Shape Modification

Next, we compare the text-guided modification capabilities of our EXIM with
existing works ShapeCrafter (Fu et al., 2022) and TITG3SG (Liu et al., 2022).
As Figures 8 and 9 show, our approach allows more accurate editing of the
regions of interest according to the text descriptions, whereas the two
baselines may significantly impact other regions when they modify the shapes.
For example, ShapeCrafter alters the seat and legs when editing the back; see
Figure 8 top row, whereas TITG3SG cannot keep the original color when editing
the shape and may alter the shape when modifying the color.

More importantly, our EXIM is able to modify both the geometry and color of 3D
shapes using text descriptions successively, as Figure 23 shows. For instance,
we can easily convert the legs of a chair to wheels and add slats in the
backrest of the chair without affecting the other unmentioned regions. More
results are provided in the supplementary material.

Figure 8.  Comparison with ShapeCrafter (Fu et al., 2022) for text-guided
modification. Green and blue: ShapeCrafter. Yellow and white: Ours. Left:
ShapeCrafter alters the legs (red arrow) and the seat (blue) while editing the
back. Right: ShapeCrafter modifies the legs (blue) when adding armrests and
adjusts one armrest (red) while editing the back. Our approach maintains
unmentioned regions unaltered. Note that our color prediction is disabled for
a fair comparison.

Figure 9. Text-guided modification

using TITG3SG (Liu et al., 2022) (left) and our method (right).

Figure 10. Our approach enables us to successively modify 3D shapes using the
text descriptions.

###  4.3. Indoor Scene Generation

We extend our approach to indoor scene generation by composing shapes
generated by conditioning the process on text prompts. As illustrated in
Figure 11, our approach can generate style-consistent room-level 3D scenes in
terms of both color and shape.

Figure 11. Our approach effectively generates style-consistent indoor scenes
by utilizing the text descriptions, considering both the color and shape
aspects. High resolution figures are provided in the supplementary material.

###  4.4. Ablation Studies and Generation Novelties

##### Analysis on Word-level Cross-Attention Modules

To validate the effectiveness of the word-level cross-attention module in
embedding localized information into features, we design a baseline that
directly feeds the global feature of the text description $f_{D}$ into the
network without using the attention module. Specifically, we transform $f_{D}$
to the same feature dimension as the feature map $F_{i}$ using an MLP and
multiply $F_{i}$ with $f_{D}$. Figure 12 shows that our approach, with word-
level cross-attention, can produce results that are more consistent with the
text descriptions.

##### Analysis on the Implicit Stage

To investigate the manner in which the implicit stage refines the shape, we
conduct a comparative analysis of the generative quality produced by the two
stages of EXIM. As illustrated in Figure 13, the implicit refinement stage
effectively recovers high-frequency details such as thin components, which are
challenging to capture using the hand-crafted wavelet components $C_{0}$. To
quantitatively assess the generative details, we employ the $COV$ and $MMD$
metrics, as they can reveal the quality of these details. The findings
presented in Table 2 demonstrate that the results refined by the implicit
stage consistently surpass the stage-1 outputs.

##### Analysis on Multi-scale Feature Fusion

To assess the effectiveness of our multi-scale feature fusion approach within
the Explicit Stage, we establish a baseline model. This baseline model
predicts the occupancy and RGB values using the preceding feature map
exclusively, rather than merging all the previous features from $f_{1}$ to
$f_{4}$. As illustrated in Figure 14, our multi-level feature fusion strategy
blends the semantic-rich deep features with detail-rich shallow features. This
integration helps to generate detail-preserved structures and colors that are
more consistent with the input texts.

##### Analysis on Modification

Designs

First, we examine the efficacy of our inpainting strategy for shape
modification. As depicted in Figure 15, our approach, in comparison to
directly combining the generated results with the given shape using mask $M$,
is capable of generating consistent modification results in terms of both size
and style, for instance, in the case of legs and arms. Next, we explore the
impact of the shape-color decoupled generation strategy on modification. As
illustrated in Figure 16, the newly created structures resulting from shape
modification exhibit noisy color due to a lack of shape awareness. Yet, our
approach is able to effectively produce plausible colors more consistently
with the manipulated shapes.

Table 2. Ablation study on the hybrid representation. The generative quality
of our Implicit Stage consistently outperforms the Explicit Stage, which is
inspired by (Hui et al., 2022)

.  Method COV ($\uparrow$) MMD ($\downarrow$) CD EMD CD EMD Explicit Stage
39.8 38.7 14.0 16.3 Implicit Stage 41.7 42.4 13.8 15.9

Figure 12. Generative results with feeding a text feature to the network
(left) vs. our cross-attention module (right).  Figure 13. The explicit stage
(violet) refines the output of the Implicit stage (yellow). Color prediction
is disabled for more clear comparisons. Figure 14. Visual ablation on the
generated results from a single scale feature (left) vs. ours multi-scale
features (right) in the Implicit Stage. The single-scale feature may lead to
artifacts, such as those indicated by the red arrows. Left: missing leg and
noisy bottom. Middle: missing connections and incomplete wheels. Right: the
yellow color is inconsistent with the words “green colored” in the input text.
Figure 15.  In comparison to the simple combination using the mask (blue), our
shape inpainting method (pink) enhances the consistency between the given and
generated shapes. Colors are used for visualization purposes.  Figure 16. Our
color prediction is conditioned on the predicted shape, enhancing the shape-
color consistency when modifying the shape.

##### Generation Novelties

We analyze our method’s ability to generate novel shapes instead of merely
replicating samples from the training set. As illustrated in Figure 17, we
plot the distribution of Chamfer Distance (CD) between each generated shape
and its nearest neighbors within the training set. In Figure 17, for each
given generated sample (the “green” shape), we display its four nearest
neighbors from the training samples, arranged from the most similar to the
least similar, from left to right. It is important to note that a substantial
difference between the generated and retrieved shapes exists at $CD=7(25\%)$.
Here, $CD=7(25\%)$ means that only 25% of all evaluated samples have a Chamfer
distance to their nearest neighbors smaller than $7$. This finding indicates
that a significant portion of the generated results from our approach
comprises novel shapes that go beyond the training set.

Figure 17.  Analysis of generation novelty. For each of the 2,000 generated
chairs, we identify the most similar chairs in the training data with the
smallest Chamfer Distance (CD). Top left: the distribution of CD values for
all 2,000 generated chairs in relation to their nearest neighbors in the
training data. Others: for each generated chair (green), we retrieve the top
four most similar chairs (blue). Notably, significant differences exist
between our generated chairs and the retrieved ones for 75% (from (b) to (e))
of the generated samples with $CD\geq 7$. Color prediction is disabled for
more clear comparisons on the shapes.

###  4.5. Additional Generative Results

We present additional generative results for multiple categories using our
pseudo annotations in Figure 18. The pseudo text enables us to generate a
diverse array of 3D shapes that are consistent with input texts, eliminating
the need for human-annotated texts. More generative results are illustrated in
the supplementary material.

Figure 18. Diversified generative results of our approach. High resolution
figures are provided in the supplementary material.

##  5\. Conclusion

We presented a new text-guided 3D shape generation approach. The key
contribution of this work is a hybrid explicit-implicit (EXIM) 3D shape
representation that combines the strengths of explicit and implicit shape
representations. Specifically, the explicit stage learns the topology of 3D
shapes and enables a certain level of local modification, while the implicit
stage adds the details and predicts the colors. Additionally, the shape and
color predictions are decoupled and color prediction is conditioned on shape
to promote shape-color consistency in the modification. Our approach surpasses
existing works in its ability to create high-quality 3D shapes and to
facilitate local modifications using texts. Also, we extend our approach to
additional categories by leveraging image captioning models to enable the
innovative application of generating style-consistent room-level scenes.
Extensive experiments demonstrate that our approach outperforms existing works
regarding generative quality and text-shape consistency. We hope our
investigation will inspire further research in text-to-shape generation and
dataset creation. Limitations are discussed in the supplementary material.

###### Acknowledgements.

The work has been supported by the Research Grants Council of the Hong Kong
Special Administrative Region (Project No. CUHK 14206320,14201921) and the
InnoHK of the Government of the Hong Kong Special Administrative Region via
the Hong Kong Centre for Logistics Robotics.

Supplementary Material

##  6\. Additional Generation Results

First, we show more generations results from texts in Figures 19, Figure 20,
and Figure 21. When trained on the 3D-FUTURE dataset (Fu et al., 2021), our
approach can generate more detail-rich shapes; see pillows and cushions in the
chairs and sofas of Figure 22.

These results further demonstrate our EXIM can effectively create high-
fidelity 3D shapes from texts that are highly consistent with the input texts.

Figure 19. Additional text-guided generation results on the chair category.
Figure 20. Additional text-guided generation results on the airplane and sofa
categories.  Figure 21. Additional text-guided generation results on the table
and cabinet categories.

##  7\. Additional Moification Results

We further show additional text-guided 3D shape modification results. As shown
in Figure 23, we can effectively edit a local region (swivel chair, Z-shape
leg, S-shape leg, and X-cross legs) and topology (tall legs and short legs)
using texts without affecting other unrelated regions. Note that we show more
results of shape modification since it is a more challenging task than editing
colors.

Figure 22. Results produced by our model trained on the 3D-FUTURE dataset (Fu
et al., 2021).  Figure 23. Additional text-guided modification results. Our
EXIM can effectively manipulate the interested region without affecting other
regions.  Figure 24. The pipeline of deriving the mask using DiffEdit
(Couairon et al., 2023).

##  8\. Details on the Mask Prediction

To perform local modification, we first generate a mask $M$ to identify the
region of interest and guide subsequent modification operations. We adopt the
recent DiffEdit approach (Couairon et al., 2023), which focuses on 2D image
editing tasks, and adapt it to our 3D shape modification problem.

To identify the region of interest described by the text $D$, this approach
exploits the differences in diffusion model predictions conditioned on two
distinct texts, $D$ and $D^{\prime}$ (an empty text). The pipeline for
deriving $M$ is depicted in Figure 24. Given a shape $S$, we initially
transform it into the wavelet domain, introduce noise to a predefined
timestamp $t$ according to Equation (1) in the main paper, and obtain the
noisy wavelet volume $C_{t}$. We then denoise $C_{t}$ to $C_{t-1}$ by
conditioning the diffusion model on the input text $D$ and an empty text
$D^{\prime}$, respectively. Given $C_{t}$, these two text prompts result in
different noise estimations: $C_{t-1}$ and $C_{t-1}^{\prime}$. For instance,
in the case illustrated in Figure 24, $D$ aims to create a swivel chair, while
the empty text prompt $D^{\prime}$ seeks to reconstruct the original shape
with four legs $S$. In other regions, such as the back and seat, both $D$ and
$D^{\prime}$ tend to predict similar noise to reconstruct the original shape
$S$, as these areas are less relevant to the text prompt “a swivel chair”.
Utilizing this observation, we can estimate the region of interest by
calculating the difference between $C_{t-1}$ and $C_{t-1}^{\prime}$, denoted
as $\Delta C_{t-1}$. We repeat this process ten times and normalize the
summation of $\Delta C_{t-1}$ to values between 0 and 1. Subsequently, we can
derive the binary mask $M$ using thresholding. As demonstrated in Figure 24,
we can effectively and automatically locate both the leg regions of the given
shape $S$ and the swivel region for generating new structures, which are
initially empty in $S$.

##  9\. Analysis on the Pseudo Annotations

###  9.1. Pseudo Annotation Creation

In existing work, TAPS3D (Wei et al., 2023) aims to generate pseudo captions
for 3D shapes using CLIP (Radford et al., 2021). However, this approach
requires the manual collection of ShapeNet-related nouns to create a
vocabulary set, which significantly restricts the richness of the resulting
descriptions. Furthermore, as explored in (Liu et al., 2023), there exists a
noticeable gap between CLIP’s text and image features, leading to potential
inconsistencies between CLIP-based attribute retrieval and the actual 3D
shape. As a result, the descriptions generated for ShapeNet chairs using this
method contain an average of only $2.84$ words, which is insufficient for
providing a detailed description of a shape.

To address the aforementioned issues, we employ the large-scale vision-
language model BLIP-v2 (Li et al., 2023b) to generate descriptions for
rendered images of 3D shapes using the prompt, ”Question: please describe the
[category] in as much detail as possible, including its color, shape,
structure, parts, and attributes. Answer: ”. As demonstrated in Figure 25, our
method is capable of producing detailed descriptions for the 3D shapes,
allowing us to extend our approach to a wider range of categories beyond
chairs and tables.

Figure 25. (a) TAPS3D (Wei et al., 2023) can only create simple captions while
(b) our pseudo annotations contain rich details.

###  9.2. Performance of Pseudo Annotations

To evaluate the effectiveness of the pseudo-annotations and examine the
potential of our approach for application to a broader range of categories, we
train our method using texts generated by the image captioning model BLIP-v2
(Li et al., 2023b). We perform an analysis on the chair category from ShapeNet
(Chang et al., 2015) in order to compare our results with those obtained using
human-annotated texts.

Figure 26 and Table 3 indicate that both of our models trained on pseudo-
annotations achieve performance comparable to those trained on human-annotated
texts. Specifically, the model trained using pseudo-annotations is slightly
inferior to the one trained with human-annotated texts in terms of text-shape
consistency and shape generation quality. However, our pseudo-annotations
outperform the annotated ones regarding color generation. This can be
partially attributed to the fact that nearly all pseudo-texts contain color
descriptions, whereas annotated texts may not always include them.

Figure 26. Qualitative evaluation on the pseudo annotation quality. Left:
trained with pseudo annotation. Right: trained with human-annotated data. Our
model can produce plausible shapes consistent with the human-annotated text
even trained with pseudo annotations generated by the image captioning model
BLIP-v2 (Li et al., 2023b).  Table 3. Quantitative analysis on our pseudo
annotations.

Method | CLIP-S ($\uparrow$) | FPD ($\downarrow$) | FID ($\downarrow$)
---|---|---|---
Pseudo annotation | 60.21 | 291.16 | 35.5
Human annotation | 71.75 | 192.17 | 36.8

##  10\. Comparison with Wavelet Representation

To further demonstrate the generation capability of EXIM, we create a baseline
following the state-of-the-art 3D shape generation approach (Hui et al., 2022)
which adopts wavelet representation. Specifically, given the coarse 3D shapes
$S$ generated by our Explicit Diffusion Stage, we adopt an additional high-
frequency wavelet domain component $H$ besides $C_{0}$ to predict details on
$S$ following (Hui et al., 2022) instead of our Implicit Refinement Stage.

As shown in Figure 27, our stage-2 refinement result (green) can better
recover high-frequency details (see the holes) on the seat while maintaining
the original structures compared with the wavelet high-frequent components
adopted in (Hui et al., 2022). Further, the quantitative results shown in
Table 4 demonstrate our approach outperforms our baseline consistently in all
metrics. Beyond that, our implicit stage can produce plausible colors while
the wavelet baseline cannot.

Figure 27. Based on the output of our explicit stage (brown), our implicit
stage (green) can better recover high-frequency details while keeping the
original topology compared with the wavelet high-frequent components adopted
in (Hui et al., 2022) (blue).  Table 4. Ablation study on hybrid
representation.

Method | COV ($\uparrow$) | MMD ($\downarrow$)
---|---|---
CD | EMD | CD | EMD
Stage 1 | 39.8 | 38.7 | 14.0 | 16.3
wavelet high freq. (Hui et al., 2022) | 40.3 | 39.8 | 13.9 | 16.1
EXIM stage 2 | 41.7 | 42.4 | 13.8 | 15.9

Figure 28. Our generative results may not always be consistent with the text
description, e.g., the numerical numbers.

##  11\. Details on TSDF and 3D Wavelet Transform

In this section, we provide additional information on TSDF and wavelet
transform as suggested in (Hui et al., 2022).

The Truncated Signed Distance Function (TSDF) is a mathematical function used
to quantify the distance from a query point $p$ to the surface of a 3D shape.
This distance possesses a sign, denoting positivity for points exterior to the
object and negativity for those within the object’s boundaries. The TSDF
undergoes truncation, implying that it’s calculated within a limited distance
threshold from the object’s surface. In this work, we follow (Hui et al.,
2022) to normalize each shape to be in the [-0.45, +0.45] range and perform
grid-based calculations for SDF values with a resolution of $256^{3}$ (Wang et
al., 2022b). Then we truncate the distance values in the range of [-0.1, +0.1]
to get the TSDF $S$.

To obtain the low-frequency wavelet component of the TSDF $S$ of the size
$256^{3}$, we decompose its TSDF volume into a high-frequency wavelet volume
$H^{1}$ and a low-frequency wavelet volume $C^{1}$ using the biorthogonal
wavelets filter (Cohen, 1992) and repeat the above process for $J$ times to
obtain the low-frequency wavelet component $C^{J}$ for conducting the
diffusion process. Note that the above process is reversible: given a low-
frequency wavelet component $C^{J}$, we can reconstruct the shape $\hat{S}$
with details corresponding to components of higher frequencies
$\\{H^{1},C^{1},C^{2},...,C^{J-1}\\}$ discarded. Following (Hui et al., 2022),
we choose $J$ to be $3$, resulting in a wavelet volume $C^{3}$ with size
$46^{3}$ and perform diffusion process in this compact wavelet space.

Readers may refer to (Hui et al., 2022) for more details.

##  12\. Limitations

Despite the superior performance of our approach, there are still some
limitations to this work. First, the performance of the categories trained on
pseudo-annotations is largely constrained by the image captioning model, such
as BLIP-v2 (Li et al., 2023b). Although our analysis in Section 9.2 of this
supplementary file demonstrates the satisfactory quality of our pseudo-
annotations, their quality still falls short compared to human-annotated ones.
However, it is important to note that our approach is orthogonal to
advancements in image captioning models, and we could adopt more recent
models, such as miniGPT4 (Zhu et al., 2023), to generate higher-quality
pseudo-annotations. Additionally, utilizing large-scale image-language models
for automatic annotation is a promising research direction. Second, in text-
guided 3D shape modification, automatically deriving an accurate mask $M$ from
a text prompt remains an unsolved problem. Although we can employ DiffEdit
(Couairon et al., 2023) to locate the region of interest, there is no
guarantee that the approach will always precisely segment the desired region.
Therefore, we provide a secondary option for users to manually refine the mask
or directly select the area of interest. While this manual strategy is a
common choice in many existing works (Nichol et al., 2022a; Lugmayr et al.,
2022), automatically locating the region of interest would be a more desirable
solution. Third, the generated shapes may not always be consistent with the
input text. For example, as shown in Figure 28, our model creates a table with
eight legs instead of the mentioned six and produces a four-layered table when
it should have been two layers. Also, while image captioning enables training
the generative model without paired text-shape data, our method still requires
large-scale, category-specific 3D data for captioning. This poses a limitation
compared to zero-shot methods like DreamFusion (Poole et al., 2023). At last,
although our approach can generate plausible colors on the 3D shapes in a
feed-forward manner, it’s still difficult for our current implementation to
generate high-fidelity textures like existing methods that can generate rich
color and textures often rely on a time-consuming per-scene optimization,
leveraging 2D priors, e.g., DreamFusion (Poole et al., 2023).

## References

  * (1)
  * Chang et al. (2015) Angel X. Chang, Thomas Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015.  _ShapeNet: An Information-Rich 3D Model Repository_.  Technical Report arXiv:1512.03012 [cs.GR].
  * Chen et al. (2018) Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. 2018.  Text2shape: Generating shapes from natural language by learning joint embeddings. In _ACCV_.
  * Chen et al. (2023) Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. 2023.  Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation.  _ICCV_ (2023).
  * Chen et al. (2022) Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. 2022.  Tango: Text-driven photorealistic and robust 3D stylization via lighting decomposition.  _NeurIPS_ (2022).
  * Chen et al. (2019) Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. 2019.  BAE-Net: Branched autoencoder for shape co-segmentation. In _CVPR_.
  * Chen and Zhang (2019) Zhiqin Chen and Hao Zhang. 2019.  Learning implicit fields for generative shape modeling. In _CVPR_.
  * Cheng et al. (2023) Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. 2023.  SDFusion: Multimodal 3D shape completion, reconstruction, and generation. In _CVPR_.
  * Chibane et al. (2020) Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. 2020.  Implicit functions in feature space for 3D shape reconstruction and completion. In _CVPR_.
  * Cohen (1992) Albert Cohen. 1992.  Biorthogonal wavelets.  _Wavelets: A Tutorial in Theory and Applications_ (1992).
  * Couairon et al. (2023) Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. 2023.  DiffEdit: Diffusion-based semantic image editing with mask guidance.  _ICLR_ (2023).
  * Fu et al. (2021) Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 2021.  3D-FUTURE: 3D furniture shape with texture.  _IJCV_ (2021).
  * Fu et al. (2022) Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. 2022.  Shapecrafter: A recursive text-conditioned 3D shape generation model.  _NeurIPS_ (2022).
  * Gao et al. (2022) Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. 2022.  GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images.  _NeurIPS_ (2022).
  * Hertz et al. (2022) Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine-Hornung, and Daniel Cohen-Or. 2022.  Spaghetti: Editing implicit shapes through part aware generation.  _TOG_ (2022).
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  _NIPS_ (2017).
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans. 2022.  Classifier-free diffusion guidance.  _NeurIPS Workshop_ (2022).
  * Hong et al. (2022) Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. 2022.  AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars.  _ACM TOG (SIGGRAPH)_ (2022).
  * Hui et al. (2022) Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. 2022.  Neural wavelet-domain diffusion for 3D shape generation. In _SIGGRAPH Asia 2022 Conference Papers_.
  * Hui et al. (2020) Le Hui, Rui Xu, Jin Xie, Jianjun Qian, and Jian Yang. 2020.  Progressive point cloud deconvolution generation network. In _ECCV_.
  * Jain et al. (2022) Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. 2022.  Zero-Shot Text-Guided Object Generation with Drefam Fields. In _CVPR_.
  * Jiang et al. (2023) Zutao Jiang, Guangsong Lu, Xiaodan Liang, Jihua Zhu, Wei Zhang, Xiaojun Chang, and Hang Xu. 2023.  3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation.  _AAAI_ (2023).
  * Jun and Nichol (2023) Heewoo Jun and Alex Nichol. 2023.  Shap-E: Generating conditional 3D implicit functions.  _arXiv preprint arXiv:2305.02463_ (2023).
  * Li et al. (2022) Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, and Dacheng Tao. 2022.  3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models.  _arXiv preprint arXiv:2211.14108_ (2022).
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ICML_ (2023).
  * Li et al. (2023a) Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. 2023a.  Diffusion-SDF: Text-to-Shape via Voxelized Diffusion.  _CVPR_ (2023).
  * Li et al. (2021) Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu. 2021.  SP-GAN: sphere-guided 3D shape generation and manipulation.  _ACM TOG (SIGGRAPH)_ (2021).
  * Lin et al. (2023) Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2023.  Magic3D: High-Resolution Text-to-3D Content Creation.  _CVPR_ (2023).
  * Liu et al. (2023) Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. 2023.  ISS: Image as stetting stone for text-guided 3D shape generation.  _ICLR_ (2023).
  * Liu et al. (2022) Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. 2022.  Towards Implicit Text-Guided 3$D$ Shape Generation. In _CVPR_.
  * Lugmayr et al. (2022) Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022.  Repaint: Inpainting using denoising diffusion probabilistic models. In _CVPR_.
  * Luo and Hu (2021) Shitong Luo and Wei Hu. 2021.  Diffusion probabilistic models for 3D point cloud generation. In _CVPR_.
  * Mescheder et al. (2019) Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019.  Occupancy networks: Learning 3D reconstruction in function space. In _CVPR_.
  * Michel et al. (2022) Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022.  Text2Mesh: Text-Driven Neural Stylization for Meshes. In _CVPR_.
  * Mittal et al. (2022) Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. 2022.  AutoSDF: Shape priors for 3D completion, reconstruction and generation. In _CVPR_.
  * Mohammad Khalid et al. (2022) Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. 2022.  CLIP-Mesh: Generating textured meshes from text using pretrained image-text models. In _SIGGRAPH Asia Conference Paper_.
  * Nichol et al. (2022a) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022a.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  _ICLM_ (2022).
  * Nichol et al. (2022b) Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. 2022b.  Point-E: A System for Generating 3D Point Clouds from Complex Prompts.  _arXiv preprint arXiv:2212.08751_ (2022).
  * Niemeyer et al. (2020) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020.  Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In _CVPR_.
  * Oechsle et al. (2019) Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. 2019.  Texture fields: Learning texture representations in function space. In _ICCV_.
  * Paschalidou et al. (2021) Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. 2021.  Atiss: Autoregressive transformers for indoor scene synthesis.  _NeurIPS_ (2021).
  * Poole et al. (2023) Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2023.  DreamFusion: Text-to-3D using 2D diffusion.  _ICLR_ (2023).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _ICML_.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _CVPR_.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding.  _NeurIPS_ (2022).
  * Sanghi et al. (2022) Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero. 2022.  CLIP-Forge: Towards zero-shot text-to-shape generation. In _CVPR_.
  * Sanghi et al. (2023) Aditya Sanghi, Rao Fu, Vivian Liu, Karl Willis, Hooman Shayani, Amir Hosein Khasahmadi, Srinath Sridhar, and Daniel Ritchie. 2023.  Text-Sculpture: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Text.  _CVPR_ (2023).
  * Shue et al. (2022) J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 2022.  3D Neural Field Generation using Triplane Diffusion.  _arXiv preprint arXiv:2211.16677_ (2022).
  * Smith and Meger (2017) Edward J Smith and David Meger. 2017.  Improved adversarial systems for 3D object generation and reconstruction. In _CoRL_.
  * Song et al. (2021) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021.  Denoising diffusion implicit models.  _ICLR_ (2021).
  * Tang (2022) Jiaxiang Tang. 2022.  Stable-dreamfusion: Text-to-3D with Stable-diffusion.  https://github.com/ashawkey/stable-dreamfusion.
  * Tang et al. (2019) Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong. 2019.  A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In _CVPR_.
  * Vahdat et al. (2022) Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al. 2022\.  LION: Latent Point Diffusion Models for 3D Shape Generation.  _NeurIPS_ (2022).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_.
  * Wang et al. (2022a) Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. 2022a.  CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields. In _CVPR_.
  * Wang et al. (2022b) Peng-Shuai Wang, Yang Liu, and Xin Tong. 2022b.  Dual octree graph networks for learning adaptive volumetric shape representations.  _TOG (SIGGRAPH)_ (2022).
  * Wei et al. (2023) Jiacheng Wei, Hao Wang, Jiashi Feng, Guosheng Lin, and Kim-Hui Yap. 2023.  TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision.  _CVPR_ (2023).
  * Wu et al. (2016) Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016.  Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling.  _NIPS_ (2016).
  * Yan et al. (2022) Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2022.  Shapeformer: Transformer-based shape completion via sparse representation. In _CVPR_.
  * Zhang et al. (2022) Biao Zhang, Matthias Nießner, and Peter Wonka. 2022.  3DILG: Irregular latent grids for 3D generative modeling.  _NeurIPS_ (2022).
  * Zhang et al. (2023) Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 2023.  3Dshape2vecset: A 3D shape representation for neural fields and generative diffusion models.  _arXiv preprint arXiv:2301.11445_ (2023).
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ (2023).