#  AToM: Amortized Text-to-Mesh using 2D Diffusion

Guocheng Qian1,2 Junli Cao1 Aliaksandr Siarohin1 Yash Kant1,3 Chaoyang Wang1
Michael Vasilkovsky1 Hsin-Ying Lee1 Yuwei Fang1 Ivan Skorokhodov1 Peiye
Zhuang1
Igor Gilitschenski3 Jian Ren1 Bernard Ghanem2 Kfir Aberman1 Sergey Tulyakov1
1Snap Research 2KAUST 3University of Toronto

###### Abstract

We introduce Amortized Text-to-Mesh (AToM), a feed-forward text-to-mesh
framework optimized across multiple text prompts simultaneously. In contrast
to existing text-to-3D methods that often entail time-consuming per-prompt
optimization and commonly output representations other than polygonal meshes,
AToM directly generates high-quality textured meshes in less than 1 second
with around $10\times$ reduction in the training cost, and generalizes to
unseen prompts. Our key idea is a novel triplane-based text-to-mesh
architecture with a two-stage amortized optimization strategy that ensures
stable training and enables scalability. Through extensive experiments on
various prompt benchmarks, AToM significantly outperforms state-of-the-art
amortized approaches with over $4\times$ higher accuracy (in DF415 dataset)
and produces more distinguishable and higher-quality 3D outputs. AToM
demonstrates strong generalizability, offering finegrained 3D assets for
unseen interpolated prompts without further optimization during inference,
unlike per-prompt solutions.

##  1 Introduction

Polygonal meshes constitute a widely used and efficient representation of 3D
shapes. As we enter a revolutionary phase of Generative AI [46, 41], the
creation of 3D meshes has become increasingly intuitive, with controls
transitioning from complex, handcrafted graphics handles [50] to simple
textual inputs [14]. Current mainstream text-to-mesh models [25, 8, 47] can
generate impressive textured meshes through score distillation sampling [38]
without 3D supervision. Despite the growing interest, these methods require a
per-prompt optimization that trains a standalone model for each prompt, which
is time and computational consuming. More importantly, per-prompt solutions
cannot generalize to unseen prompts.

Recently, ATT3D [29] presents amortized text-to-3D, which optimizes a text-
to-3D system in many prompts simultaneously unlike per-prompt solutions. This
amortized optimization not only significantly reduces training time but also
allows generalizability due to the feature sharing across prompts.
Unfortunately, ATT3D is limited predominantly to outputting 3D objects
represented by Neural Radiance Fields (NeRF) [31]. An amortized text-to-mesh
system is of more practical importance, but is under explored. First, mesh is
more widely used in most developments such as gaming and design. However,
converting NeRFs to meshes is inaccurate and might require further
optimization that is costly [28]. Second, training text-to-mesh directly
facilities a more efficient rasterizer, allowing higher-resolution renders
that help recover details in geometry and texture compared to text-to-NeRF
[25, 8].

Extending ATT3D to amortized mesh generation presents challenges in unstable
training that causes poor geometry. Our observations highlight two primary
factors contributing to this instability of ATT3D for mesh generation: the
architecture and the optimization. First, ATT3D adopted a HyperNetwork-
based[18] Instant-NGP [34] positional encoding for text-conditioned NeRF
generation. This HyperNetwork introduces numerical instability and demands
special layers such as spectral normalization [33] to alleviate. The
instability is more severe in large-scale datasets, leading to
indistinguishable 3D content for different prompts, limiting generalizability
of the system. See the two similar robots generated from two distinct prompts
in $4^{th}$ row $1^{st}$ and $3^{rd}$ column in Fig. 5. Second, the end-to-end
optimization for text-to-mesh also triggers instability due to the topological
limitations of differential mesh representation [42], leading to suboptimal
geometry. Overall, naively extending ATT3D to mesh generation results in
divergent optimization and the inability to generate any 3D object after
training. Refer to the second column 1 in Fig. 1 for illustration.

Figure 1:  Per-prompt text-to-mesh [47] generates high-quality results but
demands expensive optimization. Naively extending ATT3D for mesh generation
leads to divergent training and poor geometry. AToM introduces a triplane-
based architecture with two-stage amortized optimization for enhanced
stability. AToM efficiently generates textured meshes for various text prompts
in under one second during inference.

We thus introduce AToM, the first amortized approach for direct Text-to-Mesh
generation. To address architecture instability, AToM introduces a text-to-
triplane network in replacement of HyperNetwork for the positional encoding.
Our text-to-triplane demonstrates greater resilience to parameter changes and
generally yields higher-quality and significantly more distinguishable 3D
content compared to the ATT3D’s HyperNetwork counterpart. We then propose to
use triplane features as input to subsequent signed distance function (SDF),
deformation, and color networks to generate geometry and texture for
differentiable mesh [42].

Moreover, to stabilize optimization, we propose a two-stage amortized training
in contrast to naive end-to-end optimization. Our first stage trains text-to-
triplane, SDF, and color networks through low-resolution ($64\times 64$)
volumetric rendering. Volumetric rendering’s consideration of multiple points
per ray contributes to a stable optimization of the SDF network. In our second
stage, these networks undergo refinement, and an additional deformation
network is learned to manipulate the mesh vertices for finegrained details.
Utilizing efficient mesh rasterization allows for $512\times 512$ resolution
renders in this phase. After training, AToM enables ultra-fast inference,
generating textured meshes in under one second. The main contributions of this
work can be summarized as follows:

  * •

We propose AToM, the first amortized text-to-mesh model that is optimized
across multiple text prompts without 3D supervision. AToM trains a triplane-
based mesh generator, which contributes to stable optimization and
generalizability to large-scale datasets.

  * •

We introduce a two-stage amortized optimization, where the first stage uses
low-resolution volumetric rendering, and the second stage utilizes high-
resolution mesh rasterization. Our two-stage amortized training significantly
improves the quality of the textured mesh.

  * •

AToM generates high-quality textured meshes in less than $1$ second from a
text prompt and generalizes to unseen prompts with no further optimization.

##  2 Related Work

Figure 2: Inference and training of AToM. AToM inference (up): AToM generates
textured meshes from given prompts in less than a second in inference. The
text-to-mesh generator proposed in AToM consists of three components: a) a
text encoder that tokenizes the input prompt, b) a text-to-triplane network
that outputs a triplane representation from the text embedding, and c) a 3D
network that generates SDF, vertex deformation, and color to form a
differential mesh from positions and triplane features. AToM Training
(bottom): AToM utilizes a two-stage amortized optimization, where the first
stage leverages stable volumetric optimization to train only the SDF and
texture modules using low-resolution renders. The seconds stage uses mesh
rasterization to optimize the whole network through high-resolution renders.
In both stages, AToM is trained simultaneously on many prompts through the
guidance of a text-to-image diffusion prior without any 3D data supervision.

Feed-Forward 3D Generation. The evolution of feed-forward 3D generation models
has followed the success of 2D generation, drawing inspiration from generative
adversarial networks [15, 22, 5] and autoregressive networks [56, 32] to
diffusion models [35, 21, 10, 17, 7, 57]. Various 3D representations have been
studied, including point clouds [54, 23, 44], volumes [26, 45, 55, 11], and
meshes [58, 9, 37, 50]. Despite their success, these methods are bounded by
the availability of high-quality 3D data and thus most previous works merely
applied to certain categories, such as cars and human faces [14, 4]. The
concurrent work Instant3D [24] shows the potential to train a generalizable 3D
generative model in the recent large-scale 3D dataset [13]. We note that
training in 3D dataset or through score distillation sampling (SDS) [38] are
two orthogonal directions. The latter does not require any 3D data, which
aligns with our interest. We elaborate the text-to-3D by SDS next.

Per-prompt 3D Optimization. Recent studies have suggested leveraging
pretrained text-to-image models for 3D generation without 3D data supervision
[20, 38, 30, 39, 6]. Subsequent works introduced enhancements in directions
such as multiview image gradient aggregation [49], two-stage training
optimization [25], representation techniques [8], increased diversity [53],
and optimization techniques [59]. Being able to generate high-quality 3D
content, per-prompt optimization is receiving increasing interest. However,
these methods are time-intensive, as each asset needs a separate optimization
process and usually requires tedious parameter tuning. Per-prompt optimization
also overfits to the single training prompt. Instead, we are interested in
generalizable and efficient text-to-3D.

Amortized Optimization. Unlike the time-consuming per-prompt optimization,
ATT3D [29] proposed to amortize [1] the optimization across multiple prompts.
This enables more efficient synthesis of 3D objects in seconds, facilitating
interpolation between prompts and generalization to unseen prompts. However,
ATT3D is limited to small-scale datasets, generating 3D content
indistinguishable between prompts in larger-scale benchmark, _e.g_. DF415 (415
prompts from DreamFusion [38]). Additionally, ATT3D solely produces NeRF that
limits the quality. A recent concurrent work HyperFields [3] attempted to
improve ATT3D with a stronger dynamic hypernetwork. In this work, we are more
interested in amortized text-to-mesh, that generates textured meshes in under
one second and can be applied to large-scale prompt datasets.

##  3 Method

###  3.1 AToM Pipeline

Fig. 2 demonstrates the pipeline of the proposed Amortized Text-to-Mesh
(AToM). Unlike mainstream per-prompt solutions [38, 25, 47, 8] that train a
standalone 3D model for a specific prompt, AToM trains a text-conditioned mesh
generator, which can efficiently produce textured meshes from various text
prompts in inference stage. The network architecture of AToM consists of three
components: (1) a text encoder, (2) a text-to-triplane network, and (3) a
triplane-to-mesh generator.

Text encoder embeds the input text prompt. For simplicity, we use the same
frozen pretrained T5 XXL [40] as the text-to-image diffusion model DeepFloyd
IF [12]. The text embedding $e\in\mathbb{R}^{L_{e}\times C_{e}}$ obtained is
used as input to the following networks to generate the desired 3D content,
where $L_{e}$ and $C_{e}$ represent the number of tokens (_e.g_. 77) and the
dimension of their embedding (_e.g_. 4096), respectively.

Text-to-triplane network $T$ outputs a triplane representation from the text
embedding $e$. $T$ is composed of two parts. The first part is a linear
projection that maps the averaged text embedding from $\mathbb{R}^{C_{e}}$ to
$\mathbb{R}^{3C_{T}H_{T}W_{T}}$, which is then reshaped to a triplane [4]
representation $\mathbb{R}^{3\times C_{T}\times H_{T}\times W_{T}}$. Note that
$C_{T}$ is the number of triplane features and $H_{T}\times W_{T}$ denotes the
height and width of each plane. The second part of $T$ is a text-conditioned,
3D-aware Triplane ConvNeXt network to enhance the triplane features. We
construct the network by stacking $N$ ConvNeXt blocks, where each block
consists of four modules. The first is a multihead cross-attention module [48]
between text embedding and triplane features. Each pixel of a plane is used as
one query token, while each text token serves as the key and value. This
cross-attention module is beneficial for higher generalizability and better
quality especially when the dataset is large-scale (see §5). The second module
is a 3D-aware convolution borrowed from [52]. Due to the fact that each pixel
$(i,j)$ in a plane can be associated with the whole column or row in the other
two planes, this 3D-aware convolution is proposed to concatenate the features
of $(i,j)$ with the averaged features of $(i,:)$ and $(:,j)$, and then perform
a 2D convolution. The third is a depth-wise convolution with kernel size
$7\times 7$ to aggregate spatial information per channel, as inspired from
ConvNeXt [27]. The last is a feedforward network that is made up of two
inverted linear layers to mix channel information. We perform convolutions and
linear layers per plane, since we empirically find otherwise the information
of non-correlated regions will be mixed that might slightly degrade the
performance. The 3D-aware convolution and the feedforward network can be
efficiently implemented by group convolutions using PyTorch. A residual
connection is added to each module to avoid gradient vanishing. See Appendix
for the illustration of our text-to-triplane achitecture and the Triplane
ConvNeXt block.

Triplane-to-Mesh generator $\nu$ generates a differential mesh from the
triplane features. We use DMTet [42] as the mesh representation. DMTet
represents a differential tetrahedral geometry as a signed distance field
(SDF) defined on a deformable tetrahedral grid. The mesh vertices $V$ and
their connections (mesh faces) are predefined on the initial grid. A
deformation network is optimized to offset these predefined vertices for
refined triangle meshes and finer details. The SDF values of the shifted
points are learned by an SDF network to represent the final distance to the
surface. The zero-level set of SDF values represents the surface of the
triangle meshe. Moreover, a color network is leveraged to learn the color of
each vertex. Overall, we employ three separate networks, _i.e_. the SDF
network, the deformation network, and the color network, to optimize DMTet.
The input of each network is the concatenation of the triplane features and
the positions with their sinusoidal encoding of the predefined mesh vertices.

(a) AToM

(b) AToM Per-Prompt

Figure 3: Comparing AToM to AToM Per-Prompt on the Pig64 compositional prompt
set (“a pig activity theme”), where each row and column represent a different
activity and theme, respectively. The models are trained using 56 prompts and
tested on all 64 prompts, while 8 unseen prompts are evaluated on the
diagonal. As depicted in (a), AToM consistently generates pigs with a similar
identity and a uniform orientation, indicating that AToM also promotes feature
sharing across prompts, similar to ATT3D [29]. Also, AToM generates 3D content
with consistent quality, while per-prompt optimization cannot as shown in (b).
Additionally, per-prompt optimization is more prone to overlooking certain
details, such as the top hat in row 2 column 4 and the shovel in row 4 column
2 in (b), while AToM preserves them. More importantly, AToM performs well on
unseen prompts without further optimization, unlike the per-prompt solution.

The inference stage of AToM is a feedforward progress that gets a textured
mesh directly from the text input, and is finished in less than one second
without the need of optimization. During inference, once a text is given, AToM
first gets the text embedding $e$ through the text encoder and next passes $e$
to the triplane generator to obtain the features. AToM then inputs the
vertices of the DTMet grid to query the triplane features, encodes the
vertices positions, and passes the concatenation of triplane features,
positional encoding, and positions to the SDF, the deformation, and the color
networks to output the textured mesh.

###  3.2 Two-Stage Amortized Optimization

Optimizing a text-to-mesh end-to-end is problematic due to the topology
constraints _e.g_. triangle connections, inherent in the differentiable mesh.
Fantasia3D [8] makes such a direct training of text-to-mesh possible, but
requires sophisticated parameter tuning for each prompt. Refer to the rather
poor geometry without per-prompt tuning in Appendix. GET3D [14] also shows the
possibility of training an unconditional mesh generator, but is limited to
specific categories such as chair and car, and requires the ground-truth 3D
data during training. We show that a trivial end-to-end training for AToM
leads to divergent optimization (§5). To address this unstable optimization,
we propose a two-stage amortized optimization: a NeuS volumetric training as
warmup followed by a differentiable mesh training.

First stage: volumetric optimization. We use volumetric rendering in the first
stage to warmup the SDF network. In this stage, we use the NeuS [51]
representation, and only optimize the triplane generator, the SDF network, and
the color network. We render a low-resolution (_e.g_. $64\times 64$) image by
accumulating points’ colors along the ray. The obtained renderings are added
noise and passed to the text-to-image diffusion prior to provide the guidance
through SDS loss [38]. Mathematically, given a pixel in the rendered image,
the ray emitted from this pixel is denoted as
$\left\\{\mathbf{p}_{i}=\mathbf{o}+t_{i}\mathbf{v}\mid i=\right.$
$\left.1,\ldots,n,t_{i}<t_{i+1}\right\\}$, where $\mathbf{o}$ is the center of
the camera, $\mathbf{v}$ is the unit direction vector of the ray, $n$ is the
number of points per ray, $t$ denotes the distance from $\mathbf{o}$. NeuS
volumetric rendering is given by:

$\displaystyle\hat{C}$ $\displaystyle=\sum_{i=1}^{n}T_{i}\alpha_{i}c_{i},\quad
T_{i}=\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right)$ (1)
$\displaystyle\alpha_{i}$
$\displaystyle=\max\left(\frac{\Phi_{s}\left(f\left(\mathbf{p}\left(t_{i}\right)\right)\right)-\Phi_{s}\left(f\left(\mathbf{p}\left(t_{i+1}\right)\right)\right)}{\Phi_{s}\left(f\left(\mathbf{p}\left(t_{i}\right)\right)\right)},0\right)$

where $\phi_{s}(x)=se^{-sx}/\left(1+e^{-sx}\right)^{2}$ is the logistic
density distribution. $s$, $f$, $c$ are the learnable parameter of NeuS, the
SDF network, and the color for point $i$, respectively.

Second stage: mesh optimization. For the second stage, we use differentiable
mesh representation. Fast and memory-efficient mesh rasterization is
leveraged, allowing the system be trained with high-resolution renderings
(_e.g_. $512\times 512$). The same SDS loss as the first stage is used as
guidance. The deformation network initialized with zeros is included in the
optimization, _i.e_. the vertices of the initial mesh grid without offsets are
used as query points. Since SDF and color networks are warmed up during the
first stage, the main goal of the second stage is improve the quality of the
geometry and texture through the high-resolution renders in mesh
representation.

##  4 Experiment

We conduct comprehensive experiments on various benchmarks to show the
effectiveness of AToM. We showcase our strong generalizability to unseen
prompts, while the per-prompt solutions [38, 47, 8, 25] cannot. We also
demonstrate quantitatively and qualitatively that AToM outperforms ATT3D, the
state-of-the-art amortized text-to-3D. We show the capability of AToM in the
large-scale benchmark while per-prompt solutions are prohibitive to train, and
ATT3D produces indistinguishable results.

Table 1: Compare AToM to the state-of-the-art. CLIP R-probability$\uparrow$ is
reported. The per-prompt methods in seen prompts are deemphasized. Per-prompt
solutions have not been experimented in Animal2400 and DF415 due to their
prohibitive computation. ATT3D’s results are from the original paper [29].
ATT3D-IF† denotes our reproduced version using Deep Floyd [12] as prior.

Method/Dataset | Pig64 unseen | Pig64 seen | Pig64 all | Animal2400 | DF27 | DF415
---|---|---|---|---|---|---
DreamFusion-IF | 0 | 0.7143 | 0.6250 | - | 0.8889 | -
TextMesh-IF | 0 | 0.8036 | 0.7031 | - | 0.9259 | -
Fantasia3D | 0 | 0.5357 | 0.4688 | - | 0.7037 | -
Magic3D-IF | 0 | 0.8036 | 0.7031 | - | 0.8519 | -
ATT3D | 0.3750 | 0.6071 | 0.5781 | 0.11 | 0.6296 | -
ATT3D-IF† | 0.6250 | 0.6429 | 0.6406 | 0.1671 | 0.8519 | 0.1880
AToM (Ours) | 0.7500 | 0.7500 | 0.7500 | 0.3442 | 0.9259 | 0.8193

Data. We evaluate on two compositional datasets from ATT3D [29]: Pig64 and
Animal2400. Pig64 is structured according to the pattern “a pig {activity}
{theme}” from 8 activities and $8$ themes. In total, there are $64$ prompts in
Pig64, where $56$ are used as training and all prompts including $8$ unseen
ones are used as testing. Animal2400 is constructed following the template
“{animal} {activity} {theme} {hat}”. There are 10 animals, 8 activities, 6
themes, and 5 hats. We experiment on the hardest spit ($12.5\%$ split), where
only 300 prompts are used in training, and all 2400 prompts are tested
including 2100 unseen ones. See Appendix for dataset details. We also evaluate
on two datasets demonstrated in DreamFusion [38]: DF27, the $27$ prompts shown
in their main paper and DF415, the $415$ prompts available on their project
webpage.

Evaluation Metrics. We employ the same evaluation metric, the CLIP
R-probability, as in ATT3D. The CLIP R-probability gauges the average distance
of the associated text with $4$ uniformly rendered views from the generated 3D
model. This distance score indicates the level of confidence CLIP holds in
regards to the relevance between the text prompt and the mutiple renderings
from each 3D asset.

Implementation Details. We implement our AToM, reimplement ATT3D, and run per-
prompt baselines [38, 47, 25, 8] using the ThreeStudio [16] library. For all
methods except Fantasia3D [8] that requires using Latent Diffusion Model [41],
we utlize Deep Floyd [12] as the text-to-image prior, as it is found to offer
higher quality across methods [16]. For text embedding, we use the same frozen
T5 text encoder for prompt processing and text-to-triplane input. During the
first stage, we render $64\times 64$ resolution images with $64$ uniformly
sampled points per ray. One can use $32$ points without significant
difference. We optimize networks with learning rate 4e-4 and batch size 16
using 2 GPUs for 20K iterations on DF27, 4 GPUs for 10K iterations on Pig64, 8
GPUs for 100K iterations on DF415. For the second stage, we optimize with
learning rate 2e-4 and batch size 16 using 4 GPUs for 10K iterations on DF27
and Pig64, and 8 GPUs for 50K iterations on DF415. See Appendix for details.

Figure 4: Gallery of AToM evaluated in DF415. Here ^ and $ denote “a zoomed
out DSLR photo of” and “a DSLR photo of”, respectively.

###  4.1 Unseen Interpolation Experiments

As a significant benefit, AToM generalizes to interpolated prompts that are
unseen during training. This generalizability is not possessed by the per-
prompt solutions. Fig. 3 showcases the differences of AToM compared to AToM
per-prompt in the Pig64 compositional dataset. We highlight that AToM per-
prompt shares the same architecture but is trained in a per-prompt fashion. We
observe the following:  (1) AToM can produce high-quality results of unseen
prompts without further optimization, while per-promt optimziation cannot, as
shown in the diagonal in Fig. 3; (2) AToM generates pigs with a similar
identity and a uniform orientation, which is not observed in per-prompt
experiments, indicating that AToM promotes feature sharing across prompts; (3)
Per-prompt optimization is more prone to overlooking certain details, such as
the top hat in row 2 column 4 and the shovel in row 4 column 2, due to the
necessity for per-prompt parameter tuning, while AToM yilds a consistent
quality across prompts. In Appendix, we further illustrates the training
dynamics of AToM compared to AToM per-prompt, AToM significantly outperforms
its per-prompt version under the same training budgets. Trained both to
convergence, we observe a reduction of training iterations by over 20 times of
AToM _vs_. AToM per-prompt. Appendix also qualitatively compare AToM to ATT3D
in Pig64 and the harder dataset Animal2400, where we again show the obvious
improvements of AToM against ATT3D. Refer to Appendix for details.

ATT3D-IF† AToM (Ours) ATT3D-IF† AToM (Ours)

Figure 5: Compare AToM to ATT3D-IF† evaluated in DF415. In each row, we mostly
show results from two similar prompts. While ATT3D producing indistinguishable
results for similar prompts, AToM handles the complexity of prompts and
achieves significantly higher quality than ATT3D. ^ in the text denotes “a
zoomed out DSLR photo of ”. One can also observe clear improvements of AToM
over the original ATT3D by cross-referencing with their paper.

###  4.2 Compare to the State-of-the-Art

Tab. 1 presents our quantitative results in terms of CLIP R-probability on
Pig64, DF27, and DF415 benchmarks, compared to the amortized text-to-NeRF
method ATT3D, and per-prompt approaches [38, 47, 8, 25]. In addition to
reporting the official results, We also reproduce ATT3D using the same
diffusion prior [12] as AToM, denoted ATT3D-IF† for a fair comparison. From
the experiments, one can observe the following: (1) AToM achieves a higher
CLIP R-probability of $75.00\%$ than ATT3D ($64.29\%$) on Pig64’s unseen
prompts, indicating its stronger capability to generalize to unseen prompts.
(2) Across the training (seen) prompts in Pig64 and DF27, AToM surpasses
DreamFusion[38] and Fantasia3D [8] on both datasets. In comparison to TextMesh
[47] and Magic3D [25], AToM slightly lags in CLIP R-probability in Pig64 seen
prompts; however, visually, AToM exhibits more consistent results as shown in
Fig. 3. (3) Across all benchmarks shown in Tab. 1, AToM showcases superior
performance compared to ATT3D, highlighting AToM’s effectiveness over ATT3D.
Specifically, in DF415, AToM attains $81.93\%$ accuracy, much higher than
ATT3D ($18.80\%$).

Fig. 5 show the qualitative comparison between AToM and ATT3D in the large-
scale bechmark DF415. ATT3D mostly outputs indistinguishable 3D assets across
various prompts. Conversely, AToM excels in managing large-scale benchmarks,
handling complex prompts, and achieving consistently higher accuracy and
higher quality than ATT3D. For qualitative comparisons against ATT3D and per-
prompt solutions in Pig64 and DF27, see Appendix. We observe AToM can achieve
a comparable performance to the state-of-the-art with consistent quality
across prompts unlike per-prompt solutions.

##  5 Analysis and Ablation Study

We perform ablation studies in DF415 in Fig. 6. We investigate: (1) the effect
of two-stage training by comparing AToM with an end-to-end single-stage
training version, (2) the effects of the second stage by comparing to the
first-stage output, (3) the effect of triplane by comparing to AToM with
HyperNet-InstantNGP (Hyper-INGP for short) used in ATT3D as a replacement for
positional encoding, (4) the effect of architecture designs including
ConvNeXt, 3D-aware convolution, and cross attention.

Figure 6: Ablation study. We compare AToM full pipeline in column 2 against
the end-to-end approach without two-stage training in column 1, the first-
stage output without second-stage refinement in column 3, AToM first stage
without triplane but employing Hyper-INGP used in ATT3D [29] in column 4, AToM
first stage without ConvNeXt blocks but using two linear layers with spectral
normalization for text-to-triplane in column 5. Quantitative results in
average R-Probability evaluated in the entire DF415 dataset are provided at
the bottom. $ in the text denotes “a DSLR photo of ”.

Two-stage training significantly improves convergence. Training AToM end-to-
end using a single stage, _i.e_. training solely by the second stage from
scratch, leads to poor optimization. In DF415, we observe that the training
diverges at a very early stage (in iteration 2000), generating black images
without objects after that. We provides the results without two-stage training
at iteration 2000 before the divergence in Fig. 6 column 1. AToM single stage
results in the lowest accuracy ($7.47\%$), significantly worse than AToM full
($81.93\%$). Fig. 6 demonstrates the coarse and noisy visual results due to
its divergent training. We also ablate single-stage training in a smaller
dataset DF27 provided in Appendix, where the model trained through single
stage can converge but still produces lower-quality results than the baseline
for many prompts. Overall, these experiments clearly show the importance of
two-stage training.

Second-stage training yields meshes with improved visual quality. Fig. 6
column 2&3 shows the usage of our second stage can slightly improves the
accuracy. This is expected because the CLIP R-probability is used to estimate
the correspondence between the generated 3D shape and the text prompt, for
which, the first stage output is good enough. The higher-quality renderings
from the second stage only have neglectable effect on the CLIP R-probability
metric. Visually speaking, the second stage of AToM increases the rendering
resolution, reduces noise, and enhances sharpness.

Triplane _vs_. HyperNetworks. We use text-to-triplane as a text-conditioned
positional encoding instead of Hyper-INGP used in ATT3D. Hyper-INGP is a
HyperNetworks-based [18] positional encoding, which utilizes two linear layers
to predict the weights for the Instant-NGP positional encoding network [34].
Hyper-INGP is not numerically stable and therefore requires special layers
such as spectral normalization [33] to stabilize optimization [29]. Due to the
difficulty of learning the weights of another neural network, Hyper-INGP shows
low model capacity in the large-scale dataset, delivering a poor accuracy
($15.18\%$) in DF415 as indicated in Tab. 1. Its low capacity is also verified
in our ablation study where we replace our text-to-triplane with Hyper-INGP in
Fig. 6 column 4: the accuracy of AToM first stage drops from $81.69\%$ to only
$35.18\%$. Visually speaking, less distinguishable results with Hyper-INGP are
produced compared to our text-to-triplane network. To verify this performance
drop is not due to the reduced complexity of the network, we also removed all
ConvNeXt blocks in text-to-triplane and used two linear layers with spectral
normalization to predict the features, which still significantly outperforms
the Hyper-INGP counterpart, as indicated in column 5 _vs_. column 4 ($77.11\%$
_vs_. $35.18\%$). We highlight that only difference between columns $4\&5$ is
the positional encoding (Instant-NGP or triplane). These experiments clearly
indicate the strength of our proposed text-to-triplane positional encoding.

Triplane ConvNeXt designs. In Fig. 6 column 5 (w/o ConvNeXt), we experiment
AToM without Triplane ConvNeXt. We observe an accuracy drop of $4.8$ points.
The proposed Triplane ConvNeXt blocks are helpful in enhancing details,
reducing noises, and more importantly, preserving components of complex
prompts. We also tried to replace ConvNeXt blocks into Transformer blocks but
found Transformer did not converge. We hypothesize that Transformer requires a
significantly larger amount of data. We also perform ablation studies on the
components of Triplane ConvNeXt blocks to investigate the effectiveness of
3D-aware convolution and cross attention, and reach lower CLIP R-Prob
$79.76\%$ and $79.28\%$, respectively. These indicate that both 3D-aware
convolution and cross attention improve the performance.

##  6 Conclusion

This work proposes AToM, the first amortized text-to-mesh framework. AToM
introduces a 3D-aware text-to-triplane network, which leads to superior
quality compared to the HyperNetworks counterpart used in ATT3D. AToM also
adopts a two-stage amortized optimization to stabilize the text-to-mesh
generation. AToM significantly reduces training time compared to per-prompt
solutions due to geometry sharing of amortized optimization. More importantly,
AToM demonstrates strong generalizability, producing high-quality 3D content
for unseen prompts without further optimization. Compared to ATT3D, AToM
achieves an accuracy more than $4$ times higher in DF415. Qualitatively, AToM
outperforms ATT3D by providing distinguishable 3D assets with better geometry
and texture. We believe AToM, along with the code, the pretrained models, and
the generated 3D assets that will be made publicly available, will contribute
to push the boundaries of text-to-mesh generation.

AToM: Amortized Text-to-Mesh using 2D Diffusion
— Appendix —

##  A Implementation Details

###  A.1 AToM implementation details

AToM uses a similar camera and rendering setup to TextMesh [47] in the first
stage and similar to Magic3D [25] in the second stage. We bound the objects in
$2$ meters and set the camere $3$ meters away from the object center. We
employ a field of view ranging from $40$ to $70$ in the first stage and from
$30$ to $40$ in the second stage. Soft rendering with $50\%/50\%$
probabilities for textureless/diffuse shading is used to avoid learning flat
geometry. We implement SDF, deformation, and color networks using three
separate three-layer MLPs with hidden channels $64$. We empirically find that
these separate networks slightly improve the generation quality than the
single model used in ATT3D. Original texts without direction are used as input
for the text-to-mesh network, while directional prompts with “, front view”,
“, side view”, “, back view”, “, overhead view” are used as text condition in
the diffusion model. We utilize Deep Floyd [12] as the text-to-image prior
with guidance scale 20. A random noise from (0.2, 0.98) and (0.02, 0.5)

###  A.2 ATT3D reimplementationdetails

In the main paper, we report the quantitative results of the original ATT3D
and our re-implemented version ATT3D-IF†.

Original ATT3D is not yet released. We retrieve the quantitative results from
Figure 6 in their original paper. We compare with the original ATT3D in Table
1 in our manuscript.

ATT3D-IF† denotes our reimplementation using the exact same architecture,
training parameters, and camera setups as mentioned in the original paper,
except for the unavaiable ones (where we use the same as AToM). The only
difference of ATT3D-IF† from the original ATT3D is the diffusion prior: while
the original ATT3D used their internal version, we adopt the publicly
available IF model from Deep Floyd [12]. We cannot achieve exact the same
performance as ATT3D mostly due to the different diffusion prior.

Figure I: Text-to-Triplane architecture. Triplane is generated from the
averaged text embedding followed by a linear projection and then refined by
multiple text-conditioned 3D-aware ConvNeXt blocks.

(a) AToM Stage 1

(b) AToM Stage 2

Figure II: Results of AToM first stage (left) and second stage (right) on the
Pig64 compositional prompt set. The mesh refinement stage (second stage) turns
the NeuS representation to a high-resolution mesh representation and sharply
increases visual quality.  Figure III: Training dynamics comparisons between
AToM and AToM Per-prompt. Amortized training significantly reduces training
cost per-prompt.

##  B Method Details

Method Comparison. AToM is trained on many prompts simultaneously through SDS
loss without any 3D data and generates textured meshes in less than 1 second
during inference. This differs AToM from previous 3D reconstruction models
such as GET3D [14], which requires 3D Ground Truth and is thus limited by the
availability of 3D data and the low diversity of 3D shapes. Compared to
Magic3D [25], the well-known per-prompt text-to-mesh work, we are similar in
two-stage training, but differ from each other. Magic3D uses the results of
the first stage to initialize the SDF parameters of the second stage through
an optimization progress [25], leading to inaccurate initialization and cannot
be trained amortized. Conversely, the AToM network remains unchanged across
both stages. The first stage training in AToM serves as a warmup phase for the
second stage. This approach uses the same SDF and color networks in both
stages and eliminates the need for optimization of the SDF parameters, unlike
Magic3D. Last but not least, AToM differs from ATT3D [29] in two important
aspects: (1) AToM is the first to enable amortized training for text-to-mesh
generation, while ATT3D only supports text-to-NeRF; (2) AToM uses triplane to
condition the generation of 3D content, which is more robust to training
parameters and is more stable in optimization compared to the HyperNet-based
solutions in ATT3D.

Triplane ConvNeXt. We provides a pseudo code for the proposed Triplane
ConvNeXt in Algorithm 1. We illustrate Triplane ConvNeXt in Fig. I.

Algorithm 1 Code for Triplane ConvNeXt (PyTorch [36] like)

⬇

import torch.nn.functional as F

def forward(text_emb):

# map averaged text_emb to triplane

avg_emb = text_emb.mean(dim=1,keepdims=False)

x = self.linear(avg_emb)

# reshape to triplane

x = x.reshape(-1, self.c, self.h, self.w)

# Triplane ConvNeXt blocks

for i in range(self.num_blocks):

inp = x

# cross attention

x = x + self.crossatt(x,text_emb)

# 3D aware convolution

x = x + F.relu(self.aware3dconv(x))

# FeedForard network

x = x + self.ffn(x)

# residual connection

x = inp + x

##  C Additional Results

###  C.1 Pig64

Dataset details. Pig64 is structured according to the pattern “a pig
{activity} {theme}” from 8 activities and $8$ themes.

activities = [“riding a motorcycle”, “sitting on a chair”, “playing the
guitar”, “holding a shovel”, “holding a blue balloon”, “holding a book”,
“wielding a katana”, “riding a bike”].

themes = [“made out of gold”, “carved out of wood”, “wearing a leather
jacket”, “wearing a tophat”, “wearing a cape”, “wearing a party hat”, “wearing
a sombrero”, “wearing medieval armor”]

Two stages of AToM on Pig64. Fig. II shows the comparisons of AToM first-stage
outputs and AToM second-stage outputs. The mesh refinement stage (second
stage) turns the NeuS representation to a high-resolution mesh representation
and sharply increases visual quality.

Training dynamics. Fig. III shows the training dynamics of AToM compared to
AToM Per-prompt (per-prompt optimized versions of AToM network). Amortized
training significantly reduces training cost per prompt. While per-prompt
optimization typically requires 2000 - 8000 training batches, amortized
optimization with AToM reaches the same-level accuracy with only 142 training
batches per-prompt. In other words, AToM reduces the training time in this
compositional dataset by more than $10\times$.

(a) Original ATT3D

(b) ATT3D-IF†

Figure IV: Compare our reproduced ATT3D-IF† at right to original ATT3D [29] at
left. Due to the distinct diffusion employed in ATT3D-IF†, the disparate
outcomes from original ATT3D are expected. As strength, our reimplementation
using Deep Floyd guidance facilitates more geometry sharing, yields results
with less variances, and reduces noises. Especially, we highlight the unseen
examples in the diagonal, where ATT3D-IF† shows better generalizability then
origianl ATT3D. As drawbacks, our reimplementation handles prompts sometimes
worse than the original version, _e.g_. not all pigs are made out of wood in
the second column. Desipte the different implementation, AToM outperforms both
versions, see Fig. II for qualitative improvement and Tab. 1 in main paper for
quantitative improvements.

ATT3D reimplementation. We compare our reimplemented ATT3D-IF† to the original
ATT3D in Fig. IV.

###  C.2 Animal2400

Dataset details. We also include comparisons of AToM to ATT3D on Animal2400
$12.5\%$ split, where only 300 prompts are used in training and all 2400
prompts are used in testing. Animal2400 is constructed following the template
“{animal} {activity} {theme} {hat}”. There are 10 animals, 8 activities, 6
themes, and 5 hats.

animals = [“a squirrel”, “a raccoon”, “a pig”, “a monkey”, “a robot”, “a
lion”, “a rabbit”, “a tiger”, “an orangutan”, “a bear”]

activities = [“riding a motorcycle”, “sitting on a chair”, “playing the
guitar”, “holding a shovel”, “holding a blue balloon”, “holding a book”,
“wielding a katana”, “riding a bike”]

themes = [“wearing a leather jacket”, “wearing a sweater”, “wearing a cape”,
“wearing medieval armor”, “wearing a backpack”, “wearing a suit”]

hats = [“wearing a party hat”, “wearing a sombrero”, “wearing a helmet”,
“wearing a tophat”, “wearing a baseball cap”]

Results. AToM significantly outperforms ATT3D-IF† as shown in Fig. V.
Quantitatively, AToM achieves 0.3422 CLIP R-Probability, higher than the
original ATT3D (0.11) and ATT3D-IF† (0.1671). AToM trained in this $12.5\%$
split seems even outperforms the original ATT3D trained in $50\%$ split by
cross referencing Fig. 8 in ATT3D [29].

(a) ATT3D-IF†. CLIP R-Prob in 2400 prompts: 0.1671.

(b) AToM. CLIP R-Prob in 2400 prompts: 0.3422.

Figure V: Compare AToM to ATT3D-IF† on Animal2400 $12.5\%$ split. Trained only
in 300 prompts, AToM also generalizes to all 2400 prompts, and significantly
outperforms ATT3D and ATT3D-IF†. See the overall improved quality and how AToM
perserves the prompts when ATT3D-IF† overlooks (_e.g_., the backpacks in the
second column).  Figure VI: Compare Single-Stage AToM with AToM Full. Single-
stage training can converge in smaller dataset (DF27), but still suffers from
poor geometry, compared to the two-stage training of AToM.

DreamFusion-IF [38, 16] |  TextMesh-IF [47, 16] |  Magic3D-IF [25, 16] |  Fantasia3D [8, 16] |  AToM Per-prompt (Ours) |  ATT3D-IF† [29] |  AToM (Ours)
---|---|---|---|---|---|---

Figure VII: Visual comparisons of AToM against the state-of-the-art per-prompt
solutions (first four columns), AToM Per-prompt, and our reproduced ATT3D in
DF27 dataset. AToM achieves higher quality than ATT3D and a performance
comparable to the per-prompt solutions.

###  C.3 DF27

We compare AToM with per-prompt solutions and ATT3D-IF† in Fig. VII. Note that
we do not expect the performance of amortized training to be better than per-
prompt training for the seen prompts.

##  D Additional Ablation Study

Single-stage training in DF27. In manuscript, we show that single-stage
training in DF400 leads to divergent optimzation. Here, we further demonstrate
that single-stage training can converge in smaller dataset (DF27), but still
suffers from poor geometry. See examples in Fig. VI.

##  E Limitation and Future Work

First, the quality of AToM is bounded by the diffusion prior employed.
Throughout the work, we utilized IF-stage1 [12] as the diffusion prior, which
limits the high-frequency details due to its low-resolution input. The use of
a higher-resolution diffusion like Stable Diffusion [41] (SD) and IF-stage2
might improve the quality of AToM further. We made initial attempts to use SD,
SD’s variants including VSD [53] and MVDream [43], and IF-stage2 but resulted
in worse performance. We hypothesize the lower quality stems from the
difficulties of their optimization in the amortized settings. We believe how
to correctly utilize stronger diffusion for amortized text-to-mesh is
promising. Second, AToM used DMTet with SDF as the mesh representation, which
is not capable of modeling surfaces of nonzero genus. More advanced mesh
representation can be adopted to address this limitation, which is orthogonal
to our study. Third, Janus problem also exists in some results of AToM,
despite the fact that AToM alleviate it a lot mostly by geometry sharing in
amortized optimization. We also tried progressive views, debiasing scores
[19], Perp-Neg [2], but empirically found that they did not work trivially in
the amortized setting. We leave these limitations and their potential
solutions as future work.

Figure VIII: More results of AToM evaluated in DF415.

## References

  * [1] Brandon Amos et al.  Tutorial on amortized optimization.  Foundations and Trends® in Machine Learning, 16(5):592–732, 2023.
  * [2] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou.  Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond.  arXiv preprint arXiv:2304.04968, 2023.
  * [3] Sudarshan Babu, Richard Liu, Avery Zhou, Michael Maire, Greg Shakhnarovich, and Rana Hanocka.  Hyperfields: Towards zero-shot generation of nerfs from text.  arXiv preprint arXiv:2310.17075, 2023.
  * [4] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16123–16133, 2022.
  * [5] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5799–5809, 2021.
  * [6] Dave Zhenyu Chen, Haoxuan Li, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors.  arXiv preprint arXiv:2311.17261, 2023.
  * [7] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  * [8] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22246–22256, October 2023.
  * [9] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  Advances in Neural Information Processing Systems (NeurIPS), 2019.
  * [10] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [11] Zezhou Cheng, Menglei Chai, Jian Ren, Hsin-Ying Lee, Kyle Olszewski, Zeng Huang, Subhransu Maji, and Sergey Tulyakov.  Cross-modal 3d shape generation and manipulation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  * [12] DeepFloyd, StabilityAI.  Deepfloyd if, 2023.  GitHub repository.
  * [13] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi.  Objaverse-xl: A universe of 10m+ 3d objects.  arXiv preprint arXiv:2307.05663, 2023.
  * [14] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  Advances in Neural Information Processing Systems (NeurIPS), 35:31841–31854, 2022.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems (NIPS), 2014.
  * [16] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang.  threestudio: A unified framework for 3d content generation.  https://github.com/threestudio-project/threestudio, 2023.
  * [17] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oğuz.  3dgen: Triplane latent diffusion for textured mesh generation.  arXiv preprint arXiv:2303.05371, 2023.
  * [18] David Ha, Andrew M. Dai, and Quoc V. Le.  Hypernetworks.  In International Conference on Learning Representations (ICLR). OpenReview.net, 2017.
  * [19] Susung Hong, Donghoon Ahn, and Seungryong Kim.  Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation.  arXiv preprint arXiv:2303.15413, 2023.
  * [20] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 867–876, 2022.
  * [21] Heewoo Jun and Alex Nichol.  Shap-e: Generating conditional 3d implicit functions.  arXiv preprint arXiv:2305.02463, 2023.
  * [22] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401–4410, 2019.
  * [23] Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov.  Point cloud gan.  arXiv preprint arXiv:1810.05795, 2018.
  * [24] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi.  Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.  CoRR, abs/2311.06214, 2023.
  * [25] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [26] Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, and Sergey Tulyakov.  Infinicity: Infinite-scale city synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  * [27] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [28] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998.
  * [29] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas.  Att3d: Amortized text-to-3d object synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17946–17956, October 2023.
  * [30] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360$\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 405–421. Springer, 2020.
  * [32] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  Autosdf: Shape priors for 3d completion, reconstruction and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [33] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations (ICLR). OpenReview.net, 2018.
  * [34] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In ACM Transactions on Graphics (SIGGRAPH), 2022.
  * [35] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  arXiv preprint arXiv:2212.08751, 2022.
  * [36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  * [37] Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurelien Lucchi.  Convolutional generation of textured 3d meshes.  Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [38] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  International Conference on Learning Representations (ICLR), 2022.
  * [39] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022.
  * [42] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 6087–6101, 2021.
  * [43] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  CoRR, abs/2308.16512, 2023.
  * [44] Dong Wook Shu, Sung Woo Park, and Junseok Kwon.  3d point cloud generative adversarial network based on tree structured graph convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [45] Edward J Smith and David Meger.  Improved adversarial systems for 3d object generation and reconstruction.  In Conference on Robot Learning (CoRL), 2017.
  * [46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Proceedings of the International Conference on Machine Learning (ICML), pages 2256–2265. PMLR, 2015.
  * [47] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari.  Textmesh: Generation of realistic 3d meshes from text prompts.  In 2024 International Conference on 3D Vision (3DV), 2024.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  * [49] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [50] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  * [51] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  * [52] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4563–4573, June 2023.
  * [53] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  arXiv preprint arXiv:2305.16213, 2023.
  * [54] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  Advances in Neural Information Processing Systems (NeurIPS), 2016.
  * [55] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu.  Learning descriptor networks for 3d shape synthesis and analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [56] Biao Zhang, Matthias Nießner, and Peter Wonka.  3dilg: Irregular latent grids for 3d generative modeling.  Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [57] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka.  3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.  ACM Transactions on Graphics (SIGGRAPH), 2023.
  * [58] Song-Hai Zhang, Yuan-Chen Guo, and Qing-Wen Gu.  Sketch2model: View-aware 3d modeling from single free-hand sketches.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [59] Joseph Zhu and Peiye Zhuang.  Hifa: High-fidelity text-to-3d with advanced diffusion guidance.  arXiv preprint arXiv:2305.18766, 2023.