(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: NVIDIA00footnotetext: *Authors contributed equally.

#  LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

Kevin Xie∗ and Jonathan Lorraine∗ and Tianshi Cao∗ and Jun Gao and
James Lucas and Antonio Torralba and Sanja Fidler and Xiaohui Zeng NVIDIA

###### Abstract

Recent text-to-3D generation approaches produce impressive 3D results but
require time-consuming optimization that can take up to an hour per prompt
[39, 21]. Amortized methods like ATT3D [26] optimize multiple prompts
simultaneously to improve efficiency, enabling fast text-to-3D synthesis.
However, they cannot capture high-frequency geometry and texture details and
struggle to scale to large prompt sets, so they generalize poorly. We
introduce Latte3D, addressing these limitations to achieve fast, high-quality
generation on a significantly larger prompt set. Key to our method is 1)
building a scalable architecture and 2) leveraging 3D data during optimization
through 3D-aware diffusion priors, shape regularization, and model
initialization to achieve robustness to diverse and complex training prompts.
Latte3D amortizes both neural field and textured surface generation to produce
highly detailed textured meshes in a single forward pass. Latte3D generates 3D
objects in 400ms, and can be further enhanced with fast test-time
optimization.

Figure 1: Samples generated in $\sim$400ms on a single A6000 GPU from text
prompts.  Objects without prompt labels are generated by our text-to-3D model
trained on $\sim\\!100$k prompts, while labeled objects are generated by our
3D stylization model trained on $12$k prompts. See the project website for
more.

##  1 Introduction

Recent advances in text-to-3D synthesis via pre-trained image diffusion models
mark significant progress in democratizing 3D content creation using natural
language [39, 21, 57, 4, 55, 11, 31]. However, these methods often involve an
expensive and time-consuming optimization that can take up to an hour to
generate a single 3D object from a text prompt [39, 21]. To turbocharge these
tools, we want techniques to generate various high-quality 3D objects in real-
time, enabling rapid iteration on outputs and empowering user creativity in 3D
content creation. We aim to achieve real-time text-to-3D synthesis for diverse
text prompts and support fast test-time optimization when a further quality
boost is desired.

Figure 2: A quantitative comparison of SOTA text-to-3D methods on unseen
prompts. We plot different methods’ user study preference rates compared to
Latte3D. For MVDream, we report results with varying optimization times.

Seminal work [39] proposed to optimize a neural field using the score
distillation sampling (SDS) loss to generate 3D assets. It unlocked open-
vocabulary 3D content creation but was (a) lacking high-frequency geometric
and texture details, (b) expensive, and (c) prone to failing to generate the
desired object. Follow-ups introduced a surface-based fine-tuning stage [21,
57, 4] to generate high-frequency details. While this two-stage pipeline
defines most current state-of-the-art text-to-3D approaches, it remains
expensive and prone to prompt failure.

ATT3D [26] was the first to make text-to-3D generation fast by simultaneously
training a single model on a set of prompts in a process called _amortized
optimization_. Amortization reduces training time by optimizing a shared text-
conditioned model on a set of prompts, enabling the model to generalize to new
prompts at inference time. Despite promising results on curated prompts, ATT3D
– and the underlying per-prompt methods – remain prone to failing on general
prompts. ATT3D’s simple architecture has limited capacity and lacks strong
inductive biases for 3D generation, struggling to scale with dataset size and
rendering resolution, limiting the method to small scale (100s-1000s) prompt
sets and low-fidelity textures. Also, ATT3D only amortizes the first stage of
3D synthesis, producing a neural field representation and thus cannot generate
high-frequency details.

To address these issues, we introduce Latte3D, a Large-scale Amortized Text-
to-Enhanced3D synthesis method that can produce high-quality 3D content in
real-time. Our work makes the following technical contributions to boost the
quality, robustness, scale, and speed of text-to-3D generative models:

  * •

We propose a novel text-to-3D amortization architecture that can scale to
orders of magnitude larger prompt sets.

  * •

We leverage 3D data in training to improve quality and robustness, through
3D-aware diffusion priors, regularization loss, and weight initialization
through pretraining with 3D reconstruction.

  * •

We amortize the surface-based refinement stage, greatly boosting quality.

##  2 Related Work

Early work on 3D object synthesis was typically limited to generating objects
from a given class [58, 60, 10, 1, 64, 62, 13, 27, 30, 34, 3, 53], e.g. cars
or chairs. Recent extensions use captioned 3D shapes to train text-to-3D
models [16, 35] generating shapes from diverse categories, but requiring 3D
supervision, restricting them to synthetic datasets of limited size,
diversity, and visual fidelity.

The advent of differentiable rendering, both volumetric [32] and surface-based
[33], opened the door to inverse image rendering [39, 10], unlocking the use
of powerful text-to-image generative models [45, 43, 46] in the 3D synthesis
process. DreamFusion [39] proposed the SDS loss to optimize a neural field
using a text-to-image diffusion model to generate 3D assets. Follow-up work
introduced a surface-based refinement stage [21], allowing the synthesis of
high-frequency details. This two-stage optimization pipeline now defines most
of the state-of-the-art text-to-3D approaches [57, 4, 31, 55]. The recently
released large-scale 3D dataset Objaverse-XL [9, 8] has spurred researchers to
explore the benefits of 3D supervision in the synthesis process. To date, the
most successful approaches make text-conditioned image diffusion models 3D
aware [23, 50, 19, 49] by fine-tuning 2D diffusion models using rendered
multiview images. SDS from multiview diffusion models is then used to obtain
3D shapes [49, 50, 23].

However, these approaches require a lengthy, per-prompt optimization process,
hindering applicability to real-world content creation. To address this, two
lines of work have been pursued. The first type of method uses a text-to-
image-to-3D approach that generates images with a text-to-image diffusion
model and trains an image-to-3D lifting network [19, 49, 22, 52]. They offer
improved speed but with limited quality details compared to optimization
methods, but still take $5-20$ seconds per prompt [19, 52] for sampling from
the image diffusion model.

In an alternate line of work, ATT3D [26] proposed an amortized framework that
optimizes multiple prompts simultaneously with a unified model. Follow-up
works expanded the network architecture to achieve better quality [20, 41].
Latte3D extends this line of work to larger scales by incorporating 3D
knowledge to achieve prompt robustness during training, among other design
improvements like better architecture. Amortized text-to-3D offers an
attractive quality vs. speed trade-off as it uses a single feed-forward
architecture without requiring sampling of diffusion models - see Fig. 2.

##### Concurrent Works

Concurrently, AToM [41] also amortizes two-stage training with a triplane
representation, but they use DeepFloyd [51] and do not scale beyond the
smaller sets of thousands of prompts from ATT3D. In contrast, we scale to the
order of $100\,000$ prompts, using 3D-aware techniques such as MVDream.
Another concurrent work of ET3D [5] achieves fast generation by training a GAN
model, but it is only trained on a small-scale compositional dataset of $5000$
prompts, and only shows compositional generalization results like ATT3D and
AToM. Furthermore, it does not guarantee view consistent 3D outputs as it uses
a 2D neural upsampler. Lastly, LGM [52] concurrently works in the text-to-
image-to-3D direction that generates Gaussian splats in $5$ seconds,
representing the latest state-of-art. We experimentally compare our method
against this method and demonstrate competitive performance.

##  3 Methodology

An established pipeline for high-quality text-to-3D generation consists of two
stages, each performing per-prompt optimization using the SDS loss with a
text-to-image diffusion model [21, 57, 4]. Stage-1 optimizes a volumetric
representation, which is typically a neural radiance field. A coarse
resolution is often used to speed up rendering. Meshes afford real-time
rendering but are hard to optimize from scratch with image supervision alone.
Stage-2 uses the output of the neural field to initialize a signed distance
field (SDF) and a texture field, from which a mesh can be derived using
differentiable isosurfacing [47, 48]. This surface-based representation is
then optimized via differentiable rasterization [33], which is fast even at
$1024$ rendered image resolution – enabling supervision from higher-resolution
text-to-image diffusion models.

ATT3D [26] amortizes optimization of a neural field over a set of prompts with
a hypernetwork mapping a text prompt to a neural field, which is trained with
the SDS loss over a prompt set, referred to as _seen prompts_. Latte3D
introduces a new architecture that amortizes both stages of the generation
process, aiming to produce high-quality textured meshes in real-time. We scale
the approach to a magnitudes larger and more diverse set of prompts by
leveraging 3D knowledge in the amortization loop.

Our approach is illustrated in Fig. 4. We initialize Latte3D with a
reconstruction pretraining step to stabilize training (Sec. 3.1). The
architecture consists of two networks, one for geometry and the other for
texture (Sec. 3.2). Amortized model training with diffusion priors is done
through a two-stage pipeline consisting of a volume-based stage-1 and a
surface-based stage-2 (Sec. 3.3.1). To reconcile reconstruction pretraining,
which uses 3D geometry as input, with the task of text-to-3D generation, we
anneal the network’s input during stage-1 to gradually fully replace the
dependence on 3D input with a single dummy input. During inference (Sec. 3.4),
our model generates a 3D textured mesh from just a text prompt in 400ms and
allows an optional lightweight test-time refinement to enhance the quality of
geometry and textures (Sec. 3.5).

###  3.1 Pretraining to reconstruct shapes.

Although SDS loss can synthesize text-aligned 3D features from scratch,
empirically, it is found to be a less robust source of supervision due to high
variance. A good initialization could stabilize the optimization process. We
find that pretraining the model first to be able to encode-decode 3D shapes
makes it easier to optimize with amortized SDS subsequently. We initialize our
model $\mathcal{M}$ by pretraining it on a dataset of 3D assets using image
reconstruction losses, similar to the reconstruction stage used in [12], and
we show the pipeline in Fig. 3. $\mathcal{M}$ takes as input a sampled point
cloud and outputs a predicted 3D shape. We then render multiple views of the
predicted 3D shape $o$ to compare with the input shape $s$ renderings. We use
an $\ell_{2}$ loss on the rendered opacity and RGB image:

$\mathcal{L}_{\textnormal{recon}}(o,s,c)=||\mathbf{R}_{\text{opacity}}(o,c)-\mathbf{R}_{\text{opacity}}(s,c)||_{2}+||\mathbf{R}_{\text{RGB}}(o,c)-\mathbf{R}_{\text{RGB}}(s,c)||_{2}$

(1)

We denote the renderer $\mathbf{R}$, the opacity and RGB rendering with
$\mathbf{R}_{\text{opacity}}$ and $\mathbf{R}_{\text{RGB}}$, respectively, and
the camera with $c$, randomly sampled on a sphere.

###  3.2 Model Architecture

Our model $\mathcal{M}$ consists of two networks, $G$ and $T$, for predicting
geometry and texture, respectively. After pretraining, we introduce text-
conditioning by adding cross-attention layers. We use CLIP [42] embeddings for
encoding the text prompt $p$. In stage-1 training, we tie the geometry and
texture networks’ weights (except for separate MLP decoder heads), effectively
training a single input encoder. In stage-2, we freeze the geometry network
$G$ and refine the texture network $T$ with an additional trainable upsampling
layer. The final model $\mathcal{M}$ used in inference is the frozen $G$ from
stage-1 and a refined $T$ from stage-2.

Figure 3:  We overview our reconstruction pretraining here, which we use to
achieve our shape initialization to improve prompt robustness.  Figure 4:
Latte3D consists of two networks: a texture network $T$ and geometry network
$G$. When amortizing the first stage, the encoders of both networks share the
same set of weights. The training objective includes an SDS gradient from a
3D-aware image prior and a regularization loss that compares the rendered
predicted shape’s mask with the rendered masks of 3D assets in a library. When
amortizing surface-based refinement in stage-2, we freeze the geometry network
$G$ and update the texture network $T$.

##### The geometry network

$G$ consists of a U-Net encoder on a triplane representation [3, 10], followed
by a decoder containing another triplane U-Net and a neural volumetric density
field which predicts the output shape from triplane features. Specifically, a
point cloud is fed through a PointNet, and the features are converted to a
triplane representation via average scattering based on geometric projection
to each plane. During training, the point cloud comes from the 3D data, and in
inference, the point cloud is replaced by a dummy input of a sphere point
cloud. The triplane representation is processed through the encoder and
decoder U-Nets. The text embedding is fed into every residual U-Net block in
the encoder via cross-attention. For any point in 3D space, we extract
triplane features by projecting into each plane and bilinearly interpolating
the feature map. An MLP then predicts the density value at that point.

##### The texture network

$T$ and geometry network $G$ share the same encoder and decoder U-Net
architecture in pretraining and stage-1. We employ another non-shared MLP
neural texture field to predict RGB values at query points. In stage-2, we
upsample the triplane features produced by the encoder to a higher resolution
to gain extra capacity to generate high-frequency texture details.
Specifically, we first bilinearly upsample the latent triplane from $128$
resolution to $480$ and add a residual MLP which maps the text-embedding to
the residual of the triplane feature, mimicking ATT3D’s _mapping
(hyper)network_. The decoder then takes this upsampled triplane as input.
Details are in the Appendix.

##### Rendering.

To train the model, we render the generated 3D object into 2D images with
different methods in stage-1 and 2. In stage-1, the output 3D geometry is a
triplane-based neural field, on which we use volume rendering to get images of
$256$ resolution using the formulation from VolSDF [61], allowing us to
parameterize the density via an approximate signed distance field whose zero-
level set defines the object geometry. This choice allows a simple conversion
to surface-based rendering. For the geometry network, we observe more stable
training with volume rendering with SDS compared to surface rendering. In
stage-2, we extract the isosurface from the density field with Marching Cubes
[24] and render images via rasterization [18]. Meshes can be rendered at a
$1024$ image resolution in real-time, helping capture finer details for
learning textures.

Figure 5: Stylization application: Our model learns to generate diverse
stylizations of the same shapes. Left column shows the original shape to be
stylized.

###  3.3 Amortized Learning

Amortized learning distills knowledge from image diffusion priors into our
model. We perform amortized training on volumetric geometry training (stage-1)
and surface texture training (stage-2) sequentially. We incorporate 3D
information into our training process (a) implicitly via 3D-aware 2D SDS
losses and (b) explicitly with regularization.

##### Curating prompt sets.

We create large prompt sets for training using rule-based text generation or
ChatGPT [37]. For the former, we start with the categories names from the
captions of Objaverse [28], and design rules, like “object A in style B is
doing C” with a set of styles and activities.

Alternatively, we input the captions into ChatGPT and ask it for detailed and
diverse prompts describing a similar 3D object. Fig. 4 shows that when
generating _“Amigurumi fiddler crab wearing a top hat”_ , we guide shape
synthesis towards resembling a crab by retrieving a related 3D shape used in a
regularization loss. Other details of dataset construction are in Sec. 4.1.1.

####  3.3.1 Amortizing Stage-1 Generation

In this stage, we train $\mathcal{M}$ with our datasets. We use the SDS loss
with a 3D-aware 2D prior and regularization to the paired 3D shape, which we
now explain in detail:

##### 3D-aware SDS loss.

During stage-1 training we leverage a 3D aware diffusion prior, which provides
a stronger, multiview consistent supervisory signal. Specifically, we use the
model from MVDream [50], which was trained on four rendered views of objects
from the Objaverse dataset [9] by modifying Stable Diffusion [45] to generate
multiview consistent images.

##### Regularization loss.

While 3D-aware SDS loss with MVDream offers a strong multiview prior, it can
still fail to correctly capture the full 3D geometry for some prompts since
MVDream only supervises four views and can create enlarged geometry or
floaters in the shape. Some failures are shown on the right of Fig. 10.

Combining regularization using 3D shapes with the SDS loss helps training and
improves geometry. As shown in Fig. 4, for each training prompt, we regularize
the output shape $o$ by comparing its rendered mask to the mask of a shape $s$
retrieved from the 3D dataset with the prompt:

$\mathcal{L}_{\textnormal{reg}}(o,s,c)=||\mathbf{R}_{\textnormal{opacity}}(o,c)-\mathbf{R}_{\textnormal{opacity}}(s,c)||_{2},$
(2)

where $\mathbf{R}_{\textnormal{opacity}}$ is the opacity mask (volume-rendered
density). We balance the regularization $\mathcal{L}_{\textnormal{reg}}$ and
the SDS loss $\mathcal{L}_{\textnormal{SDS}}$ by simply blending them using a
convex combination with weight $\alpha$. Our final training loss is:

$\mathcal{L}_{\textnormal{train}}=(1-\alpha)\mathcal{L}_{\textnormal{SDS}}+\alpha\mathcal{L}_{\textnormal{reg}}.$
(3)

During training, we render five views for each generated shape. We average
across four views to compute $\mathcal{L}_{\textnormal{SDS}}$, and one is used
to compute $\mathcal{L}_{\textnormal{reg}}$.

##### Input point cloud annealing

To bridge the gap between the real point cloud inputs in training and dummy
point cloud in inference, we gradually anneal the input point clouds towards
dummy point clouds during stage-1 training. Specifically, we randomly replace
a subset of points in the input point cloud with points from the dummy point
cloud using a probability that increases linearly from $0$ to $1$ in the last
$5000$ training iterations. With point cloud annealing, the model gets better
results in inference when the dummy point cloud is used.

####  3.3.2 Amortizing Stage-2 Generation

During stage-2 training, we freeze the network for our geometry and only tune
the texture since training both leads to instabilities. We use depth-
conditional ControlNet [63] for SDS guidance, allowing a higher, $512$
resolution supervision. Depth conditioning encourages the SDS loss to guide
the texture to align with the geometry from the geometry network, thereby
improving the 3D texture consistency.

###  3.4 Inference

During inference, our model inputs the user’s text prompt and a dummy point
cloud. Our $\mathcal{M}=(G,T)$ outputs a textured mesh, where $T$ is the final
texture network refined in stage-2, while $G$ is trained in stage-1 only and
then frozen. Predicting one shape and rendering the image only takes us 400ms
on an A6000 GPU in inference, allowing the generation of user results at
interactive speeds. A single A6000 can generate $4$ samples simultaneously.
Once the user selects a desired sample, textured meshes can be exported from
Latte3D. We use a UV mapping tool to UV parameterize the mesh output by $G$
and use $T$ to predict the albedo colors of each texel in a UV-mapped texture
image. The resulting mesh and texture image is compatible with standard
rendering software.

###  3.5 Test Time Optimization

Our method supports test time optimization if the user wants to boost the
quality of a particular prompt. This can be useful if the user’s prompt
deviates significantly from the seen prompts. As in our stage-2 training, we
freeze the geometry and finetune the texture network $T$ with depth-
conditional SDS guidance. Our test time optimization is significantly faster
to converge than MVDream [50] or other optimization-based text-to-3D methods
[21, 39, 57] since we optimize our amortized network instead of a randomly
initialized network.

###  3.6 3D Stylization

Latte3D can also be trained and deployed as a 3D stylization method, allowing
users to cheaply create variations from existing 3D assets. During training,
we skip the input point cloud annealing step to maintain point cloud reliance
and amortize over a dataset where each 3D shape corresponds to many style
prompts. Our blend of regularization and SDS guidance drives the model to
produce shapes structurally similar to the original shapes yet semantically
aligned with the style prompt. Amortization significantly reduces the
computational cost per shape-prompt pair compared to optimizing each pair
separately. During inference, point clouds from the training set can be
combined with novel prompts as input to the model to produce variations of the
original shape.

Figure 6:  Qualitative comparison between Latte3D and ATT3D [26], MVDream
[50], Instant3D/3DTopia [19] and LGM [52]. The top row is a training prompt:
“a futuristic robot with shiny metallic armor and glowing blue accents..”, and
the last two rows are unseen prompts: “a red convertible car with the top
down”, “an ice cream sundae”.

##  4 Experiments

We first describe our experimental setup, including datasets, metrics, and
baselines for comparison (Sec. 4.1). We then present our quantitative and
qualitative results (Sec. 4.2), demonstrating the performance and
generalization ability of Latte3D (Sec. 4.3), followed by design choices
ablations (Sec. 4.4).

###  4.1 Experimental Setup

We now discuss dataset construction, model pretraining, and evaluation
metrics.

####  4.1.1 Datasets

Table 1: Comparing the sizes of amortization training sets to prior works.

name | #prompts | Source
---|---|---
Animal2400 [26, 41] | $2400$ | Rule based
DF415 [41] | $415$ |  DreamFusion[39]
animal-style (Ours) | $12\,000$ | Rule based
gpt-101k (Ours) | $101\,608$ | ChatGPT

##### Training Set

To investigate scaling amortized optimization beyond what has been done in
prior works – see Tab. 1 – we construct a new dataset gpt-101k that consists
of 101k text prompts and 34k shapes. For the shapes, we use the “lvis” subset
from Objaverse [9] with 50k human-verified shapes. We filter out poor shapes,
_e.g_. flat images and scans, and retain 34k shapes for our dataset. We use
ChatGPT to augment each object’s caption to gather $3$ prompts per object.

##### Evaluation Sets

We use two prompt sets for evaluation. To evaluate the benefit of amortized
optimization in our method, we evaluate our model on gpt-101k. However, due to
the computational cost of running the baselines, we randomly select a $50$
prompt subset in gpt-101k to form the seen prompt set for benchmarking.
Furthermore, to gauge the generalization abilities of our amortized model, we
construct an unseen prompt set by filtering a subset of $67$ DreamFusion[39]
prompts that are close in distribution to lvis categories.

####  4.1.2 Baselines and Evaluation Schemes

##### Baselines.

We compare Latte3D with the baselines: ATT3D [26], MVDream [50], Instant3D
[19] and LGM [52]. We re-implement ATT3D [26] with a hypernetwork. We use the
open-source threestudio [11] implementation of MVDream. For MVDream, we train
each 3D model for $10\,000$ iterations, with a batch size of $8$ on a single
A100 GPU with provided default settings. For Instant3D, since the source code
has not been released, we use the re-implementation from
3DTopia111https://github.com/3DTopia/3DTopia. For LGM, we use the official
open-source implementation.

##### Evaluation metrics.

We quantitatively compare our model with baselines by evaluating the generated
3D content fidelity and its consistency to the text prompt for the
optimization cost. We use three metrics to evaluate fidelity.

Render-FID: We compute the FID [14] between the renderings of generated 3D
content and a set of images sampled from Stable Diffusion [45] with the same
text prompts as input. This metric measures how well the generated shapes
align with those from the 2D prior in visual quality.

CLIP Score: We compute the average CLIP scores between the text prompt and
each rendered image to indicate how well the generated 3D shape aligns with
the input text prompt.

User Study: We evaluate the overall 3D geometry and texture through Amazon
Mechanical Turk user studies. For each baseline, we present videos of rendered
3D shapes generated by our method and a baseline side-by-side, with
corresponding text prompts, and ask users for their preference. We present
each comparison to three users and average the preference across users and
prompts to obtain the average user preference for each baseline relative to
Latte3D.

Timing: We measure our optimization cost by GPU time per prompt to gauge the
total compute cost on each prompt. We divide the total wall-clock time for
pretraining, stage-1, and stage-2 by the prompt set’s size, then multiply by
the number of GPUs used. We measure inference time for Instant3D, LGM, and
Latte3D by the time the model takes from in-taking the text input to
outputting the final triplane features before rendering. For MVDream [50], we
measure the average time to optimize a single prompt on an A100 GPU.

Table 2: Quantitative metrics and average user preference ($\%$) of baselines
over Latte3D trained on gpt-101k using seen and unseen prompts. We also report
test-time optimization, which takes $10$ min.

Model | Time | Render-FID $\downarrow$ | CLIP-Score $\uparrow$ | Preference % $\uparrow$
---|---|---|---|---
| seen | unseen(df) | seen | unseen(df) | seen | unseen(df)
Latte3D | 400ms | 180.58 | 190.00 | 0.2608 | 0.2605 | - | -
Latte3D-opt |  $10$min | 171.69 | 178.37 | 0.2800 | 0.3000 | - | -
MVDream |  $6$min | 203.53 | 199.23 | 0.2240 | 0.2698 | 2.5 | 12.4
MVDream |  $18$min | 191.77 | 188.84 | 0.2440 | 0.2950 | 26.5 | 52.2
MVDream |  $36$min | 158.11 | 143.44 | 0.2830 | 0.3180 | 48.5 | 89.3
3DTopia | 20s | 217.29 | 222.76 | 0.1837 | 0.2171 | 20.8 | 25.6
LGM | 5s | 178.67 | 166.40 | 0.2840 | 0.3063 | 26.2 | 36.4

###  4.2 Experimental Results

We observe robust generalization of Latte3D by training on the gpt-101k
dataset to unseen prompts (df, from DreamFusion [39]), as demonstrated by
qualitative (Fig. 6) and quantitative (Fig. 2, Tab. 2) results. We highlight
the relative user preference of Latte3D versus different baselines over
inference time in Fig. 2. Latte3D produce results of competitive user
preference to SOTA baselines at a uniquely fast inference time.

Figure 7:  Results of user study showing the average preference of MVDream at
different amounts of optimization time to Latte3D gpt-101k on seen prompts.

From the quantitative comparisons in Tab. 2, our performance on both seen and
unseen prompts is competitive with baselines while our inference time is at
least one order of magnitude faster. We note that 3DTopia[52] and LGM[52],
which reconstruct 3D representations from four input views, are advantaged
under image-based metrics (FID and CLIP score) as they tend to produce 3D
inconsistent results that are not obvious from still-images. Interested
readers can refer to rendered videos of all methods in the supplementary
materials.

####  4.2.1 Total optimization cost

A benefit of amortized optimization over many prompts is that our total
optimization cost on gpt-101k is much lower than optimizing MVDream per-
prompt. Fig. 7 illustrates that to achieve user preference on par with
Latte3D, MVDream would require $36$ GPU minutes per prompt, whereas Latte3D
spent only $215$ GPU seconds per prompt, representing a $10\times$ reduction
in optimization cost.

###  4.3 Application

We illustrate Latte3D’s capacity for enhanced quality through test time
optimization and how to adapt Latte3D for stylizing 3D content.

|  |
---|---|---
|  |
400ms |  $1$ min |  $10$ min
Figure 8: Test time optimization on seen and unseen prompt. The model is
trained on the gpt-101k dataset. First column: without test time optimization,
400ms. Other columns: test time optimization with the given time budget
denoted. Top: seen prompt “..animal with a blue and white floral pattern..”
Bottom: unseen prompt “orchid in clay pot”.

####  4.3.1 Per Prompt Test-time Optimization

We show experimentally that Latte3D quickly adapts to new prompts during test
time to improve quality. In Fig. 8, we qualitatively compare Latte3D with up
to $600$ steps. Each optimization iteration takes an average of $0.98$ seconds
on an A100 GPU, giving us a total compute budget of less than $10$ GPU minutes
per prompt. Even with test-time optimization, our method is still an order of
magnitude faster than per-prompt optimization methods such as MVDream, as in
Tab. 2. Test-time optimization is particularly beneficial on unseen prompts,
where FID drops by $11.6$ for unseen vs. $8.8$ for seen, and the CLIP score
gains $0.04$ for unseen vs. $0.02$ for seen.

####  4.3.2 Stylizing 3D content

We illustrate how Latte3D architecture offers us the flexibility to adapt it
for targeted 3D content stylization as described in Sec. 3.6. To test this, we
manually curate a dataset of 100 animal shapes from Objaverse as a basis and
augment the species name of each animal shape name with combinations of
activities (_e.g_. “riding a skateboard”) and styles (_e.g_. “voxel style”) to
create our 12000 prompts animal-style dataset. We train a Latte3D model
(stage-1 and stage-2) on animal-style without the point cloud annealing stage.
The resulting model adapts to both the text prompt and the point cloud inputs.
In Fig. 5, each row contains the outputs of Latte3D given different prompts
while fixing the point cloud input, demonstrating the effectiveness of Latte3D
for text-controlled stylization. In Fig. 11, we conduct a user study comparing
Latte3D’s outputs on animal-style to MVDream and find that Latte3D is
competitive with MVDream with $4000$ steps of optimization, representing a
$10\times$ reduction in optimization cost (compared to running MVDream for
each prompt). Latte3D on animal-style also generalizes to held-out
combinations of animal activities and styles with little quality drop, as
indicated by its similar relative preference to MVDream. Therefore, Latte3D
enables users to cheaply create variations of their 3D assets via amortized
optimization and even explore novel prompt combinations at test time.
Additional quantitative and qualitative results can be found in the
supplementary materials.

###  4.4 Ablation Studies

We now analyze the value of each component we have introduced in Latte3D, with
more results in the Appendix.

Table 3: Ablation of components in stage-1 training. Trained on gpt-101k data
and evaluated on seen and unseen prompts. Preference indicate average user
preference of baseline over Latte3D. Latte3D is better than all ablated
settings in quantitative metrics and is preferred by users on the unseen
prompt set.

Exp | MV | Unet | Pretrain | Reg | Seen | Unseen
---|---|---|---|---|---|---
Mask-FID$\downarrow$ | Render-FID$\downarrow$ | Clip-Score$\uparrow$ | Preference
ATT3D |  |  |  |  | 274.44 | 275.01 | 0.2091 | 28.3 | 24.2
+MV | $\checkmark$ |  |  |  | 243.24 | 236.73 | 0.1826 | 34.8 | 26.9
+MV+UNet | $\checkmark$ | $\checkmark$ |  |  | 184.80 | 203.33 | 0.2214 | 48.0 | 45.6
+MV+UNet+PT | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 189.09 | 189.51 | 0.2191 | 51.8 | 47.7
Latte3D (S1) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 176.44 | 186.84 | 0.2379 | - | -

##### Stage-1 ablation.

Tab. 3 summarizes our quantitative comparison of the following ablations:
ATT3D baseline using a hypernet, ATT3D+MV uses MVDream as diffusion guidance.
ATT3D+UNet+MV replaces the hypernet with our model architecture, but is
initialized from scratch and given a dummy point cloud as input.
ATT3D+UNet+MV+PT is initialized from the reconstruction pretrained (PT) model.
We introduce the Mask-FID metric to quantitatively assess geometry adherence
to the shape dataset by computing the FID between the rendered binary masks of
the generated shapes and those of the 3D dataset.

We find that each of the components we introduce improves performance.
Comparing ATT3D with ATT3D+MV, we see benefits from using the MVDream guidance
as it mitigates geometric artifacts like Janus faces. Comparing ATT3D+MV with
ATT3D+MV+UNet we see a performance boost from our architecture. Further,
adding pretraining (ATT3D+UNet+MV+PT) helps, especially in recovering fine
geometric and texture details. Finally, adding shape regularization (full
Latte3D) makes the generated geometry adhere better to the 3D data, as
evidenced by the drop in Mask-FID. An additional user study is shown in Tab.
3. We will show the convergence speed in the Appendix.

##### Ablation on using stage-2 refinement.

Stage-2 refinement gives large improvements in texture details over stage-1
training. We show the comparison in Fig. 10.

Figure 9: Comparison of stage-1 and stage-2 results. Stage-2 refinement
significantly improves the texture quality.

“fun sheep wearing skis in pixel art style, voxel”
a
0.3 0.2 0.1 0

Figure 10: Qualitative comparison of the impact of blending factor $\alpha$,
where $\alpha=0$ corresponds to no regularization.

##### Regularization

We now investigate the design choice of how to weight $\alpha$ between the SDS
loss and the regularization loss as in Eq. 3. In Tab. 3, we show that adding
regularization loss improves the quantitative results. Here we qualitatively
contrast training on different fixed weightings in Fig. 10 and Appendix,
respectively. We see greater consistency with the input shape by increasing
shape regularization with higher blend factors $\alpha$.

Table 4: An ablation of unseen DreamFusion inputs in inference and annealing in training. Users prefer the annealed model with a dummy input in $51.2\%$ of cases. Model | without anneal | with anneal
---|---|---
Input | Top-1 | Dummy | Dummy
CLIP | 0.2628 | 0.2577 | 0.2605
FID | 186.46 | 193.28 | 190.00
Figure 11:  Results of user study showing the average preference rate for
MVDream at different amounts of optimization time to Latte3D stylization
results on animal-style.

##### Annealing in training and different input point clouds in inference.

In training, we take retrieved point clouds from the dataset as input and
anneal the input to become a fixed dummy point cloud. In inference, we only
use the dummy point cloud as input. In Tab. 4, we quantitatively ablate the
role of this point cloud annealing process on the behavior of LATTE3D on
gpt-101k. Our model without annealing training is somewhat sensitive to the
point cloud input. There is a small performance drop when using a dummy as
input in inference compared to a retrieved point cloud. However, the
performance gap is reduced by introducing point cloud annealing so the model
trains with dummy input. Further, we display qualitative results in Fig. 12,
showing that point cloud annealing improves the results when the dummy input
is used in inference.

##  5 Conclusion and Limitation

We presented a scalable approach to perform amortized text-to-enhanced-3D
generation. To successfully scale amortization to larger datasets, we used 3D
data through (1) 3D-aware 2D SDS, (2) pretraining, and (3) 3D regularization
of the amortization process. We further improve the model architecture to be
more scalable. Our model generates high-quality shapes within 400ms. Moreover,
quality and generalization can be improved further via a speedy test time
optimization. Our model also has limitations. First, our model uses SDS and
thus relies on the understanding of the text-to-image models, which can often
fail to respect fine-grained details such as part descriptions. Second, the
geometry is frozen in stage-2 and test-time optimization, so the geometry
flaws from stage-1 cannot be fixed. Developing amortized training for stable
geometry changes in stage-2 is left for future work.

Figure 12:  Qualitative comparison of LATTE3D samples with a retrieved (top
row) or dummy point cloud (middle row) before annealing. The performance is
very similar except for a few geometry failures that inherit the dummy sphere
geometry (highlighted in red). We also show samples with dummy input after
annealing (bottom row). After annealing, the model no longer exhibits this
behavior on the dummy sphere inputs.

## Acknowledgements

We thank Matan Atzmon, Or Perel, Clement Fuji Tsang, Masha Shugrina, and her
group for helpful feedback. The Python community [54, 36] made underlying
tools, including PyTorch [38] & Matplotlib [15].

## Disclosure of Funding

NVIDIA funded this work. Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao,
and Xiaohui Zeng had funding from student scholarships at the University of
Toronto and the Vector Institute, which are not in direct support of this
work.

## References

  * [1] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3d point clouds. In: International conference on machine learning. pp. 40–49. PMLR (2018)
  * [2] Amos, B.: Tutorial on amortized optimization for learning to optimize over continuous domains. arXiv e-prints pp. arXiv–2202 (2022)
  * [3] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., Mello, S.D., Gallo, O., Guibas, L., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3D generative adversarial networks. In: arXiv (2021)
  * [4] Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2023)
  * [5] Chen, Y., Li, Z., Liu, P.: ET3D: efficient text-to-3d generation via multi-view distillation. arXiv 2311.15561 (2023)
  * [6] Community, B.O.: Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http://www.blender.org
  * [7] Cremer, C., Li, X., Duvenaud, D.: Inference suboptimality in variational autoencoders. In: International Conference on Machine Learning. pp. 1078–1086. PMLR (2018)
  * [8] Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S.Y., VanderBilt, E., Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L., Farhadi, A.: Objaverse-xl: A universe of 10m+ 3d objects (2023)
  * [9] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13142–13153 (June 2023)
  * [10] Gao, J., Shen, T., Wang, Z., Chen, W., Yin, K., Li, D., Litany, O., Gojcic, Z., Fidler, S.: Get3d: A generative model of high quality 3d textured shapes learned from images. Advances in Neural Information Processing Systems (2022)
  * [11] Guo, Y.C., Liu, Y.T., Shao, R., Laforte, C., Voleti, V., Luo, G., Chen, C.H., Zou, Z.X., Wang, C., Cao, Y.P., Zhang, S.H.: threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio (2023)
  * [12] Gupta, A., Xiong, W., Nie, Y., Jones, I., Oğuz, B.: 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371 (2023)
  * [13] Henzler, P., Mitra, N.J., Ritschel, T.: Escaping plato’s cave: 3d shape from adversarial rendering. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019)
  * [14] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)
  * [15] Hunter, J.D.: Matplotlib: A 2D graphics environment. Computing in Science & Engineering (2007)
  * [16] Jun, H., Nichol, A.: Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 (2023)
  * [17] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [18] Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics 39(6) (2020)
  * [19] Li, J., Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong, Y., Sunkavalli, K., Shakhnarovich, G., Bi, S.: Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.06214 (2023)
  * [20] Li, M., Zhou, P., Liu, J.W., Keppo, J., Lin, M., Yan, S., Xu, X.: Instant3d: Instant text-to-3d generation. arXiv preprint arXiv:2311.08403 (2023)
  * [21] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [22] Liu, M., Xu, C., Jin, H., Chen, L., Xu, Z., Su, H., et al.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16928 (2023)
  * [23] Liu, R., Wu, R., Hoorick, B.V., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object (2023)
  * [24] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Seminal graphics: pioneering efforts that shaped the field, pp. 347–353 (1998)
  * [25] Lorraine, J., Duvenaud, D.: Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv:1802.09419 (2018)
  * [26] Lorraine, J., Xie, K., Zeng, X., Lin, C.H., Takikawa, T., Sharp, N., Lin, T.Y., Liu, M.Y., Fidler, S., Lucas, J.: Att3d: Amortized text-to-3d object synthesis. In: Proceedings of the International Conference on Computer Vision (ICCV) (2023)
  * [27] Lunz, S., Li, Y., Fitzgibbon, A., Kushman, N.: Inverse graphics gan: Learning to generate 3d shapes from unstructured 2d data. arXiv preprint arXiv:2002.12674 (2020)
  * [28] Luo, T., Rockwell, C., Lee, H., Johnson, J.: Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07279 (2023)
  * [29] MacKay, M., Vicol, P., Lorraine, J., Duvenaud, D., Grosse, R.: Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv:1903.03088 (2019)
  * [30] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4460–4470 (2019)
  * [31] Metzer, G., Richardson, E., Patashnik, O., Giryes, R., Cohen-Or, D.: Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 (2022)
  * [32] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [33] Munkberg, J., Hasselgren, J., Shen, T., Gao, J., Chen, W., Evans, A., Müller, T., Fidler, S.: Extracting triangular 3d models, materials, and lighting from images. In: CVPR. pp. 8280–8290 (2022)
  * [34] Nash, C., Ganin, Y., Eslami, S.M.A., Battaglia, P.W.: Polygen: An autoregressive generative model of 3d meshes (2020)
  * [35] Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 (2022)
  * [36] Oliphant, T.E.: Python for scientific computing. Computing in Science & Engineering (2007)
  * [37] OpenAI: Gpt-4 technical report. ArXiv abs/2303.08774 (2023), https://api.semanticscholar.org/CorpusID:257532815
  * [38] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in PyTorch. Openreview (2017)
  * [39] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [40] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652–660 (2017)
  * [41] Qian, G., Cao, J., Siarohin, A., Kant, Y., Wang, C., Vasilkovsky, M., Lee, H.Y., Fang, Y., Skorokhodov, I., Zhuang, P., et al.: Atom: Amortized text-to-mesh using 2d diffusion. arXiv preprint arXiv:2402.00867 (2024)
  * [42] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [43] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821–8831. PMLR (2021)
  * [44] Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: International conference on machine learning. pp. 1278–1286. PMLR (2014)
  * [45] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684–10695 (June 2022)
  * [46] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [47] Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
  * [48] Shen, T., Munkberg, J., Hasselgren, J., Yin, K., Wang, Z., Chen, W., Gojcic, Z., Fidler, S., Sharp, N., Gao, J.: Flexible isosurface extraction for gradient-based mesh optimization. ACM Trans. Graph. 42(4) (jul 2023). https://doi.org/10.1145/3592430, https://doi.org/10.1145/3592430
  * [49] Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C., Wei, X., Chen, L., Zeng, C., Su, H.: Zero123++: a single image to consistent multi-view diffusion base model (2023)
  * [50] Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv:2308.16512 (2023)
  * [51] Shonenkov, A., Konstantinov, M., Bakshandaeva, D., Schuhmann, C., Ivanova, K., Klokova, N.: If by deepfloyd lab at stabilityai (2023), github.com/deep-floyd/IF
  * [52] Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 (2024)
  * [53] Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In: Proceedings of the IEEE international conference on computer vision. pp. 2088–2096 (2017)
  * [54] Van Rossum, G., Drake Jr, F.L.: Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam (1995)
  * [55] Wang, H., Du, X., Li, J., Yeh, R.A., Shakhnarovich, G.: Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774 (2022)
  * [56] Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltru${\bm{s}}$aitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., Guo, B.: Rodin: A generative model for sculpting 3d digital avatars using diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2022)
  * [57] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213 (2023)
  * [58] Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems 29 (2016)
  * [59] Wu, M., Choi, K., Goodman, N., Ermon, S.: Meta-amortized variational inference and learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 6404–6412 (2020)
  * [60] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: Pointflow: 3d point cloud generation with continuous normalizing flows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)
  * [61] Yariv, L., Gu, J., Kasten, Y., Lipman, Y.: Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems 34 (2021)
  * [62] Zeng, X., Vahdat, A., Williams, F., Gojcic, Z., Litany, O., Fidler, S., Kreis, K.: Lion: Latent point diffusion models for 3d shape generation. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
  * [63] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models (2023)
  * [64] Zhou, L., Du, Y., Wu, J.: 3d shape generation and completion through point-voxel diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5826–5835 (2021)

Supplementary Material
LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis Kevin Xie∗
Jonathan Lorraine∗ Tianshi Cao∗ Jun Gao
James Lucas Antonio Torralba Sanja Fidler Xiaohui Zeng

In this appendix, we provide additional details on Latte3D. In Sec. 0.A, we
provide details on our model architecture, amortized learning framework, and
test time optimization. In Sec. 0.B, we provide details on the experimental
settings, additional results of our methods, comparisons with baselines, and
additional ablation studies. In Sec. 0.C, we show the failure cases of our
method and its limitations. In Sec. 0.D, we show an additional capability for
amortizing the regularization strength, allowing enhanced user-controllability
and interpolations.

Table A.5: Glossary and notation ATT3D | Amortized Text-to-3D [26]
---|---
SDS | Score Distillation Sampling [39]
SD | Stable Diffusion
SDF | Signed distance field
MV | Shorthand for MVDream [50]
PT | Pretraining
S1, S2 | Stage-1 and stage-2
FID | Fréchet Inception Distance
OOD | Out of distribution
MLP | Multilayer Perceptron
$p$ | A text prompt
$s$ | An input shape point cloud
$G$ | The geometry network in our model
$T$ | The texture network in our model
$\mathcal{M}=(G,T)$ | Our model to generate 3D objects
$o=\mathcal{M}(p,s)$ | A predicted output shape
$c$ | A sampled camera
$\mathbf{R}(o,c)$ | A rendering function
$\mathbf{R}_{\text{opacity}},\mathbf{R}_{\text{RGB}}$ | A renderer for the opacity or RGB
$\mathcal{L}_{\textnormal{reg}}(o,s,c)$ | The shape regularization loss using only opacity
$\mathcal{L}_{\textnormal{recon}}(o,s,c)$ | The pretraining reconstruction loss using RGB
$\mathcal{L}_{\textnormal{SDS}}$ | The SDS loss for our text-prompt
$\alpha\in[0,1]$ | Blend factor between regularization and SDS
$\mathcal{L}_{\textnormal{train}}=(1-\alpha)\mathcal{L}_{\textnormal{SDS}}+\alpha\mathcal{L}_{\textnormal{reg}}$ | The training loss
$\mathcal{M}^{\prime}(p,s,\alpha)$ | Our model for amortizing over blend factors

##  Appendix 0.A Additional Methodology Details

In this section, we provide additional details on the model architecture for
both geometry and texture network (Sec. 0.A.2), with more details on the
amortized learning in Sec. 0.A.3 and test time optimization in Sec. 0.A.4.

###  0.A.1 Background on Amortized Learning

Amortized optimization methods predict solutions when we repeatedly solve
similar instances of the same problem [2], finding use in generative modelling
[17, 44, 7, 59], hyperparameter optimization [25, 29] and more. The goal is to
solve the problem more efficiently by sharing optimization efforts somehow.

A typical amortized optimization strategy involves (a) finding some context or
description of the problem you are solving, (b) making that an input to a
network, and (c) training the network to output problem solutions given an
input problem context, by sampling different problems during training.
Specific methods differ in how they construct the problem context, the
architecture for inputting the context to the network, and how they sample the
context during training. This strategy circumvents the need to completely
solve any single problem, scaling more efficiently than naïvely generating a
large dataset of solved problems and training a network to regress them. Many
feedforward networks can be viewed as amortizing over some type of problems,
but the amortized viewpoint is particularly useful when the target associated
with the network input is the result of (an expensive) optimization.

We amortize the optimization of text-to-3D methods, which repeatedly solve an
expensive optimization problem to generate a 3D object consistent with a
provided text prompt using the SDS loss. The problem context is the text
prompt embedding, and the solution is the output 3D object.

###  0.A.2 Model Architecture

We include an architecture diagram showing the individual components in our
model in Fig. 0.A.13.

Figure 0.A.13: We show a pipeline diagram with our architecture’s individual
components. Note that the additional residual MLP is only active in the
texture network upsampling in stage-2. Dummy point cloud

####  0.A.2.1 Geometry Network

We compute the token embeddings from the input text prompt using a frozen CLIP
[42] text encoder, following Stable Diffusion 2.1 [45]. The tokens are padded
or truncated to $77$ for easy batching. In addition, we uniformly sample
$20\,480$ points from the reference input 3D shape and feed in the point’s
positions and RGB values to the network.

We adopt PointNet [40] as our point cloud encoder but use triplanes for local
pooling. We use scatter operations at every block to aggregate the point
features onto the triplane via geometric projection. This can be regarded as a
local pooling among the points projected into the same pixel in the triplane.
We then gather the feature from the triplane and concatenate it with the
initial point feature for each point as input to the residual stream MLP.

We use $128\times 128$ resolution triplanes for our pooling and the same
resolution for the output triplane feature in the point encoder. In our
experiments, the point encoder has $5$ residual blocks and a hidden dimension
of $32$. The output of our point encoder is a $128\times 128$ triplane with
$32$ channels obtained after a final projection of the point features.

After obtaining the point encoder’s triplane feature, we use a U-Net (encoder)
to combine the features with the text embeddings, followed by another U-Net
(decoder) to process the triplane features. The encoder and decoder U-Nets are
nearly identical, except the encoder conditions on the text embeddings via
cross-attention layers inserted after every U-Net residual block.
Specifically, in our residual blocks, we apply a $3\times 3$ rolled
convolution (convolution is applied to height-stacked triplanes) followed by
$1\times 1$ 3D-aware convolution [56, 12] and another $3\times 3$ rolled
convolution. We use Swish activation and GroupNorm with $32$ groups
throughout. Our U-Net has a depth of $5$, and the lowest resolution is $8$. We
use a single res block per depth, and the base number of channels is $64$. The
final output of the U-Net is a $128$ resolution, $32$ channel triplane
feature.

In the encoder, the cross-attention layer uses the flattened triplane features
as the query and the text token embeddings as context. It uses multi-head
attention with $4$ heads, a head dimension of $64$, and layernorm.

We first compute a feature by summing bilinearly interpolated features from
each feature plane in the triplane to predict the SDF for each query position.
We then pass this feature vector through an MLP network to obtain an
approximate SDF value. In our implementation, we add the predicted SDF value
with the SDF value for a sphere, stabilizing the training process. The final
SDF value will be converted to density for volume rendering or isosurface
extraction for surface-based rendering. The MLP network has one hidden layer
of size $32$.

####  0.A.2.2 Texture Network

The texture network $T$ has the same architecture as the geometry network
described in Sec. 0.A.2.1. However, we have an upsampling layer to increase
the triplane resolution to learn high-frequency texture details in stage-2. We
describe the details for this upsampling layer below.

##### Texture network upsampling

We first upsample the triplane from a resolution of $128$ to $480$ using
nearest-neighbors and add a residual MLP that maps the CLIP embedding to the
latent triplane, following the _mapping network_ from ATT3D [26].
Specifically, with the input prompt text-embedding $p$ and output triplane
features ${\mathbf{h}}$ we use:

${\mathbf{v}}=\smash{\textnormal{linear}_{\textnormal{w/
bias}}}\left(\textnormal{normalize}\left(\tanh{\left(0.1\cdot\textnormal{flatten}\left(p\right)\right)}\right)\right)$
(4)
${\mathbf{h}}=\textnormal{reshape}\left(\smash{\textnormal{linear}_{\textnormal{no
bias}}}\left({\mathbf{v}}\right)\right)$ (5)

Here, ${\mathbf{v}}$ is of size $32$, and the final layer uses no bias, as in
ATT3D. We found spectral normalization was not required, and thus, we
discarded it in our model. The reshape operation takes a flattened vector to
our $32$ channel feature triplane shape of $[3,32,480,480]$. We use an
activation of $\tanh(0.1\cdot)$ on the text-prompt embedding to scale the
entries to $(-1,1)$ for optimization stability, where $0.1$ is to prevent
saturation. We also normalized each prompt embedding’s activation components
to have mean $0$ and unit variance for optimization stability.

The texture network is then refined via the second-stage optimization using
surface renderings. We find that upsampling the geometry triplane leads to
unstable training, we only upsample the texture network and freeze the
geometry in the second stage.

###  0.A.3 Details on Amortized Learning

##### Pretraining.

For pretraining with reconstruction, we use surface-based rendering for its
efficiency. We train the model using a batch size of $64$ objects and render
$2$ random views per object, which gives us $128$ total views.

##### Stage-1 training.

In stage-1, we use $4$ training views per output shape, as MVDream [50] only
supports computing SDS loss on $4$ views. We use $8$ shapes per batch
totalling $32$ views for all experiments except for gpt-101k where we increase
the batch size by a factor of $4$ to $32$ shapes. We use $96$ samples per ray
for volume rendering to render an image at a resolution of $$256$\times$256$$.
We also render one extra view per shape to compute our shape regularization
loss.

##### Input point cloud annealing

As in the main text, towards the end of stage-1, we anneal away the dependence
on the input point clouds by gradually replacing them with the dummy point
cloud. Specifically, we randomly replace a subset of points in the input point
cloud with points from the dummy point cloud using a probability that
increases linearly from $0$ to $1$ in the last $5000$ training iterations.
During this phase, we also freeze the decoder network to only modify the point
cloud encoder and encoder U-Net. This puts additional constraints on the
output shapes to not change much in appearance from before annealing.

##### Stage-2 training.

In stage-2, we randomly sample $10$ training views per shape, with $8$ output
shapes per batch, totaling $80$ views for all experiments. To render the
image, we first use differentiable marching cubes with a grid resolution size
of $256$ and render images with a resolution $$1024$\times$1024$$ which is
bilinearly downsampled to $512$ for our image-denoising steps.

##### Hyperparameters.

We optimize $T$ for $600$ iterations to reduce runtime and anneal the timestep
sampling range for the SDS loss over the first $400$ iterations. Further
iterations do not bring noticeable improvements in our experiments. We use the
Adam optimizer with a $0.0003$ learning rate and $\beta_{1}=$0.9$$,
$\beta_{2}=$0.99$$. We apply a $0.001$ multiplier when computing the text
guidance loss ($\mathcal{L}_{\textnormal{SDS}}$) to scale it with our shape
regularization loss ($\mathcal{L}_{\textnormal{reg}}$). Different amounts of
shape regularization may be optimal depending on the alignment between the
input reference shapes and the associated prompts. For our main reported
results, we used $\alpha=0.2$ for animal-real, $\alpha=$0.1$$ for animal-style
and $\alpha=$0.05$$ for gpt-101k.

###  0.A.4 Test Time Optimization

For test-time optimization, we decrease the learning rate to $0.0001$. We
anneal the sampled timestep range from $[0.5,0.9]$ to $[0.1,0.5]$ over $400$
iterations. We sample $8$ training views per iteration and one $\epsilon$
noise sample per training view. Other settings, such as architecture and
guidance, are the same as in stage-2.

##  Appendix 0.B Additional Experiments

In this section, we provide additional experiments. We first provide the
details on our experimental setup in Sec. 0.B.1.

###  0.B.1 Experimental Setup Details

####  0.B.1.1 Details on Baselines

##### MVDream.

We use an open-source implementation of MVDream [50] from threestudio [11],
with the original (publicly available) $4$-view diffusion model. We follow the
default settings from threestudio, which we find to perform well. We fit each
3D model using SDS for $10\,000$ training iterations, with a batch size of $8$
on a single GPU.

ATT3D. We re-implement ATT3D [26] as a baseline in our experiments. We use
their amortized optimization with either (a) the original hypernetwork or (b)
our triplane architecture. Our reproduction is consistently better than the
original paper, with more detailed results in Sec. 0.B.2. Our architecture
includes a triplane latent input representation instead of a multi-resolution
voxel grid, as in ATT3D. We also have a point-cloud encoder in our
architecture – absent in ATT3D – that feeds a dummy, spherical point cloud for
a fair comparison.

3DTopia We use the first stage of the open-source implementation from
https://github.com/3DTopia/3DTopia. We use the default settings for sampling
the diffusion model (DDIM sampler with $200$ steps, guidance scale $7.5$. We
decode and volumetrically render the sampled triplane into $256$ by $256$
images for the user study. Note that the $20$ seconds compute time only
includes the compute cost of the diffusion sampling, as time for rendering the
triplane depends both on the number of camera poses and the desired
resolution.

LGM We use the open-source implementation from https://github.com/3DTopia/LGM.
We use the text-to-3D pipeline implemented in the code base. To sample multi-
view images from MVDream, we use the default settings in the LGM code base
(DDIM sampler with $30$ steps, guidance scale $7.5$). Using the LGM
repository, we render the generated Gaussian Splatting for the user study. As
with 3DTopia, the $6$ seconds compute time only includes sampling MVDream and
running the Gaussian reconstruction model, though the per-frame LGM rendering
time is generally negligible compared to 3DTopia.

####  0.B.1.2 Dataset

Table 0.B.6: We show details for various amortized datasets, where the source generated the prompts. Each row is one curated dataset, with our assigned name for each dataset. Each prompt is tied to one or multiple shapes. The last column shows example prompts that are tied to the same training shape. name | #shape | #prompts | source |  prompt examples
---|---|---|---|---
animal-real | 100 | 1,000 | ChatGPT |  (1) Gray wolf (2) Arctic wolf (3) Coyote (4) Dingo (5) German shepherd dog (6) Siberian husky (7) Alaskan malamute (8) Fox
animal-style | 100 | 12,000 | Rules |  (1) Penguin wearing a top hat animal built out of lego (3) wood carving of penguin in a Chef’s Hat
gpt-101k | 33,958 | 101,608 | ChatGPT |  (1) A sleek silver dumbbell with a digital display that shows the weight and number of reps (2) A futuristic dumbbell made of transparent glass with swirling neon lights inside (3) A chrome-plated dumbbell with a textured grip surface for better hold

##### Curating prompt sets.

To create our prompt sets, we start with captions for the objects in our shape
set [28]. We then use GPT or a rules-based approach to create multiple prompts
based on the caption.

Specifically for the animal-real dataset, we ask GPT to recommend $10$ other
animals with the same coarse shape. We found it important to add additional
instructions to ensure diverse responses. The instruction used was: “Based on
the animal named in the query, provide $10$ different visually distinct animal
species or breeds that have roughly a similar body shape (they can be
different scale). Start from similar animals and gradually make them much less
similar. The last $4$ should be very different. Use common name and do not
include esoteric variants. Keep your answers short and concise."

For constructing gpt-101k, we broaden the scope of the instruction to handle
general object types and gather $3$ prompts per caption. The instructions used
were: “I am using text conditioned generative models, it take a base shape +
target prompt as input, and deforms the shape to a similar object, with a
slightly different shape / appearance. Now I have a caption describing the
input object, imagine the shape of the object, and think about $3$ possible
target prompts that this object can deform into? Make the prompt as detailed
as possible. Attention: 1) Don’t describe the background, surroundings. 2)
output is a python list"

##### Curating shape datasets.

We found undesirable shapes in the Objaverse [9] dataset, such as those
resulting from scans or abstract geometry, which we remove in a semi-manual
process. Specifically, we first cluster shapes into $1000$ clusters through
K-means using the average multiview CLIP image embeddings. We then manually
filter the clusters that have undesirable shapes. This gives us 380k shapes in
total. To get the reference shapes for gpt-101k, we take the intersection
between our filtered subset and the subset with lvis categories released by
Objaverse [9]. For shapes in animal-real and animal-style datasets, we
manually pick $100$ animals with an acceptable coarse full-body geometry and
represent a diversity of animals.

##### Data pre-processing.

We use Blender [6] to preprocess the dataset from Objaverse. We first
normalize all shapes to be centered, with the same maximum length of $0.8$. We
then randomly select $32$ cameras on an upper sphere and render $32$ images
for each shape. To obtain the point cloud, we render the depth images and
unwarp them into 3D to obtain the point cloud with colors.

####  0.B.1.3 Evaluation Metrics

##### Render-FID.

The Render-FID is computed as the FID score between the images generated by
Stable Diffusion 2.1 and those rendered from our model’s generated shapes.
Specifically, for each prompt in the test set, we first augment them with $4$
view-dependent prompts, and then for each augmented prompt, we use Stable
Diffusion 2.1 to generate $4$ different images. For our rendered images, we
generate one shape from one text prompt and render four images for each
generated shape from $0$, $90$, $180$, and $270$ degrees.

##### CLIP scores.

We compute the average CLIP scores between the text prompt and each rendered
image to indicate how well the generated 3D shape aligns with the input text
prompt. It is computed between the CLIP embeddings of the input text prompt
and renderings from our output shape and averaged over multiple views. We
compute the CLIP score by using the
`laion/CLIP-ViT-H-14-laion2B-s32B-b79K` version of the CLIP model.

##### Mask-FID

We additionally introduced the Mask-FID metric to quantitatively assess the
adherence of our geometry to the shape dataset. We compute the FID between
binary masks of the generated shapes and those of the 3D dataset. In
particular, we take $100$ input shapes and render binary masks from them from
$4$ $90$ degree rendering views. We compare these with binary masks rendered
from $4$ views of our output shapes. We threshold the opacity at $0.5$ for
volume rendering to obtain the mask.

##### User study.

We use Amazon Mechanical Turk to operate a user study. We present two videos
from two different methods for each user with the corresponding text prompt.
One of the videos is from the baseline, and the other is from our method. We
ask the user to answer six questions:

  * •

Which 3D object has a more natural visual appearance?

  * •

Which 3D object has a visual appearance with more details?

  * •

Which 3D object has a more natural shape?

  * •

Which 3D object has a shape with more geometric details?

  * •

Which 3D object has an overall more appealing look?

  * •

Which 3D object more faithfully follows the textural description?

We provide a screenshot of the user study in Fig. 0.C.29. We use $200$ prompts
per method, and each comparison is shown to three different users to reduce
variance. We randomize the order of the videos to remove bias due to ordering.

###  0.B.2 Validating ATT3D Reproduction

We assess our reproduction of the ATT3D baseline method with qualitative
comparisons by training models on DF27, the $27$ prompts from DreamFusion re-
used in ATT3D. Fig. 0.B.14 shows that our baseline with a triplane+UNet
architecture and only stage-1 training achieves a quality similar to that of
the source ATT3D.

Furthermore, we see improvements with stage-2 amortized refinement (without 3D
data), dubbed _MagicATT3D_ or ATT3D+S2, as we introduce amortizing over the
refinement stage of Magic3D [21]. Fig. 0.B.15 shows the quality improvement
from stage-1 to stage-2 is comparable to Magic3D’s quality improvement over
single-stage methods like DreamFusion. These experiments demonstrate the
successful reproduction of ATT3D in our framework.

ATT3D SourceATT3D, Our Arch.Our ATT3D + Stage-2 Figure 0.B.14:  We compare
examples from the source ATT3D (top) with a proxy ATT3D baseline trained with
our triplane architecture after surface extraction (middle) and our ATT3D with
additional stage-2 refinement (bottom). Our ATT3D re-implementation achieves a
similar quality to the source ATT3D. Further, stage-2 fine-tuning with our
ATT3D – dubbed MagicATT3D - provides significant quality improvements without
requiring 3D data. See Fig. 0.B.15 for more examples.

DreamFusionMagic3DDreamFusionMagic3DDreamFusionMagic3DATT3DMagicATT3D(Us)ATT3DMagicATT3D(Us)ATT3DMagicATT3D(Us)

Figure 0.B.15:  We show source examples from DreamFusion [39], which is
improved by the second stage fine-tuning from Magic3D [21]. Our ATT3D+UNet
with stage-2 finetuning – dubbed MagicATT3D – provides a quality boost over
ATT3D+UNet analogous to Magic3D’s quality boost over DreamFusion, while
maintaining ATT3D+UNet’s speed benefits. See Fig. 0.B.14 for more examples.

###  0.B.3 Additional Results

We provide additional results for the three datasets we utilized in our main
paper. For animal-style and gpt-101k, we provide additional qualitative
examples of our model output for unseen prompts, highlighting our
generalization ability.

##### Additional results on animal-real.

We present additional qualitative examples in Fig. 0.C.20.

##### Additional results on animal-style.

We show qualitative examples of Latte3D on the animal-style dataset for both
seen and unseen prompts in Fig. 0.C.21 and Fig. 0.C.22 respectively. We see
virtually no degradation in our performance on the unseen prompts,
corroborating our quantitative results and demonstrating our model’s
combinatorial generalization.

##### Additional results on gpt-101k.

In the main manuscript, we evaluated our model trained on gpt-101k on a set of
unseen prompts from DreamFusion. These prompts are generally further out of
distribution compared to our training prompts. To evaluate unseen prompts that
are more similar to our training prompts, we use ChatGPT to suggest $50$ more
prompts based on $20$ of our training prompts given as reference. We show
qualitative examples in Fig.0.C.19.

##### Additional Comparisons to Instant3D

We apply our model trained on gpt-101k to unseen prompts from DreamFusion. As
in the main text, we select a subset of $65$ prompts closer to the training
prompt distribution. Instant3D also presents their generations for the
DreamFusion prompts, so we show more qualitative side-by-side comparisons with
their reported results in Fig. 0.C.25.

##### Additional Results on Test-time Optimization

We present additional results on test-time optimization in Fig 0.C.31. Columns
of Fig 0.C.31 correspond to test-time optimization after $0$ (no
optimization), $60,120,300$, and $600$ steps, respectively.

###  0.B.4 Additional Ablation Studies

##### Additional results for ablations on stage-1.

In the main manuscript we presented the quantitative ablations of the main
components LATTE3D introduced. Due to space constraints, we only reported full
quantitative metrics on seen prompts (and user preference ratings on both seen
and unseen.

Table 0.B.7: Ablation of components in stage-1 training. Trained on gpt-101k
data and evaluated on unseen prompts. Preference indicate average user
preference of baseline over Latte3D.

Exp | MV | Unet | Pretrain | Reg | Unseen
---|---|---|---|---|---
Mask-FID$\downarrow$ | Render-FID$\downarrow$ | Clip-Score$\uparrow$ | Preference$\uparrow$
ATT3D |  |  |  |  | 294.18 | 292.38 | 0.1779 | 24.2
+MV | $\checkmark$ |  |  |  | 263.70 | 256.67 | 0.1834 | 26.9
+MV+UNet | $\checkmark$ | $\checkmark$ |  |  | 168.11 | 216.43 | 0.2283 | 45.6
+MV+UNet+PT | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 176.01 | 199.09 | 0.2330 | 47.7
Latte3D (S1) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 157.88 | 201.17 | 0.2447 | -

On the CLIP score, the ATT3D ablation outperforms adding MVDream. We posit
that this is due to the higher rate of Janus face artifacts when using SDS
over MVDream, which can maximize the CLIP score metric at the cost of
geometric errors, as evidenced by the higher Mask-FID.

##### Ablation on the upsampling layer in stage-2.

We provide an ablation study on the upsampling layer in stage-2 in Table 0.B.8
with qualitative examples in Fig. 0.C.24. Adding the upsampling layer for the
latent triplane significantly improves Render-FID and retains a strong CLIP-
Score.

Table 0.B.8:  Ablation over whether we use upsampling on the latent triplanes during stage-2 using $\alpha=.9$ on the realistic animals with samples shown in Fig. 0.C.24. Upsampling the triplane gets better performance.  Setup | Render-FID$\downarrow$ | Clip-Score$\uparrow$
---|---|---
Stage-2 w/ upsampling | 96.75 | $0.260$
Stage-2 w.o. upsampling | $104.32$ | 0.261
Stage-1 only | $122.24$ | $0.264$
Table 0.B.9: Ablation over different guidance at stage-2. Depth-SD and SD are significantly better than MVDream and VSD. While Depth-SD and SD are close to each other regarding the render-FID and CLIP score, our user study shows that Depth-SD is still preferable. The reason is that the texture generated with Depth-SD is more faithful to the input shape. Guidance | Render-FID$\downarrow$ | CLIP-Score$\uparrow$ | User-Preference ($\%$)$\uparrow$
---|---|---|---
Depth-SD | $96.75$ | $0.260$ | 30.56
SD | 94.42 | 0.266 | 29.11
MVDream | $110.13$ | $0.248$ | 18.33
VSD | $102.92$ | $0.250$ | 22.00

##### Ablation on stage-1 guidance choices.

We contrast the SDS guidance in stage-1 between Stable Diffusion [45] and
MVDream [50]. We provide qualitative examples in Figs. 0.B.16 and 0.C.30.
Notably, Fig. 0.B.16 shows examples of prompts where stage-1 SDS can produce
incorrect geometry leading to Janus-face-type artifacts propagating to the
second stage, while stage-1 MVDream avoids Janus-faces.

Stage-1: Stable DiffusionStage-1: MVDream Figure 0.B.16:  Here, we compare our
SDS guidance choices for stage-1 – Stable Diffusion [45] and MVDream [50]. We
show examples of prompts where SDS generates Janus-face-type artifacts, which
persist through stage-2 refinement. On these prompts, using MVDream leverages
3D data to guide the geometry to regions which avoid Janus faces. Further
examples are shown in Figure 0.C.30.

As MVDream is already finetuned on 3D consistent data, its geometric
consistency is already quite good. In the first stage, reference geometry
regularization is especially important when using Stable Diffusion instead of
MVDream.

##### Ablation on stage-2 guidance choices.

In Tab. 0.B.9, we explore different guidance choices in our stage-2 training.
Overall, Depth-SD and SD perform consistently better than MVDream and VSD.
MVDream performs poorly in the refinement stage because it inputs lower-
resolution images. We found that SD and Depth-SD perform close to each other
quantitatively. We use Depth-SD since it occasionally resolves Janus face
issues (Fig. 0.B.17).

Figure 0.B.17: We compare training Latte3D stage2 with depth-conditioned
ControlNet against Stable Diffusion. Without depth conditioning, we observe
the Janus face issue [39] in textures.

##  Appendix 0.C Failure Cases and Limitations

We display typical failure cases of our model in Fig. 0.C.18. Our model
trained on gpt-101k can miss details in composed prompts and often only
generates one object in multi-object prompts. We expect such failure cases to
diminish by increasing the diversity of prompts used for training. Some thin
feature details may also be lost since geometry is frozen from stage-1 and
converted from volume to surface rendering. Stabilizing geometry training in
stage-2 would help recover some of these details.

Figure 0.C.18:  We show our model’s failure to generate compositions of
objects following the prompt (top row) and thin feature geometry details
(bottom row).  Figure 0.C.19:  We show example results of our model on the
unseen in-distribution set created using further ChatGPT suggestions. From
left to right, the prompts for the first 2 rows are: 1) A set of sleek and
modern salt and pepper grinders with adjustable coarsene…; A sleek and modern
salad bowl with a curved design and vibrant stripes…; A set of artisanal
handmade soap bars in various shapes and colors…; A classic and timeless
leather watch with a simple dial and a genuine leath…; A vintage-inspired
typewriter with a metallic finish and classic keycaps…; A vibrant and
patterned beach towel with a tassel fringe…; A pair of stylish sunglasses with
mirrored lenses and a tortoiseshell frame… 2) A classic red wagon with sturdy
wooden sides and large rubber wheels…; A charming ceramic salt and pepper
shaker set shaped like playful hedgehogs…; A vibrant yellow school bus with
cheerful cartoon characters painted on the…; A sleek and modern electric
kettle with a transparent water level indicator…; A set of ceramic nesting
bowls with hand-painted floral motifs…; A set of elegant wine glasses with a
gold rim and a delicate floral pattern…; A modern glass chess set with
minimalist pieces and a reflective board… 1)2)3)4)5)6)7) Figure 0.C.20:  We
show additional example results of Latte3D on the seen prompts from animal-
real. The rows represent generations based on the same point cloud input. For
each row the animal in the text prompts are:  1) Asian Elephant; Sumatran
Elephant; Woolly Mammoth; Mastodon; White Rhinoceros; Hippopotamus; Indian
Rhinoceros 2) Zebra; Paint Horse; Pinto Horse; Brindle Horse; Friesian Horse;
Grevy’s Zebra; Mountain Zebra 3) Koi ; Shubunkin; Oranda Goldfish; Fantail
Goldfish; Betta Fish; Gourami; Tetra Fish 4) Honey bee; Carpenter bee; Yellow
jacket; Paper wasp; Hornet; Cicada ; Grasshopper 5) Emperor Penguin; Gentoo
Penguin; Macaroni Penguin; Rockhopper Penguin; Puffin; Atlantic Puffin; Tufted
Puffin 6) Great White Shark; Tiger Shark; Bull Shark; Mako Shark; Whale Shark;
Blue Shark; Dolphin 7) Texas Longhorn; Charolais Bull; Brahman Bull; Hereford
Bull; Simmental Bull; Bison; Water Buffalo 1)2)3)4)5)6)7)ABCDEFG Figure
0.C.21:  We show additional example results of Latte3D on the seen prompts
from animal-style. Each column shares the same style combination and each row
is a different animal. From left to right, the style combinations are: From
left to right, the style combinations are:  A) engraved golden cast sculpture
of .. on top of a skateboard B) fun cute .. on top of a skateboard animal
built out of lego pieces C) fun kids play doh .. on a Magic Carpet D)
cyberpunk neon light .. on top of a snowboard 3D model E) highly detailed ..
wearing skis painted in pastel delicate watercolors F) colorful 3D mosaic art
suplture of .. wearing a top hat, unreal engine, dimensional surfaces G) ..
wearing a party hat, low poly 3D model, unreal engine  From top to bottom, the
animals are: 1) cobra 2) pufferfish 3) basenji dog 4) camel 5) dog 6) boar 7)
leopard  1)2)3)4)5)6)7)ABCDEFG Figure 0.C.22:  We show additional example
results of Latte3D on the unseen prompts from animal-style demonstrating
combinatorial generalization. Each column shares the same style combination
and each row is a different animal. From left to right, the style combinations
are:  A) gray textured marble statue of .. on top of a skateboard B) engraved
golden cast sculpture of .. on top of a snowboard C) amigurumi .. wearing skis
D) .. wearing a top hat, low poly 3D model, unreal engine E) brilliantly
colored iridescent .. wearing a party hat F) DSLR full body photo of majestic
.. wearing a crown, HD, green planet, national geographic G) colorful 3D
mosaic art suplture of .. in a Chef’s Hat, unreal engine, dimensional
surfaces.  From top to bottom, the animals are: 1) elephant 2) brown and white
striped horse 3) goldfish 4) bee 5) giraffe 6) meerkat 7) shark  Mask-FIDMask-
FIDRender-FIDRender-FIDRealistic AnimalsStylized AnimalsStage 1 SDSStage 1
MVDream Figure 0.C.23:  We show the frontier of the tradeoff between Mask FID
and Render-FID at various blending $\alpha$ (annotated) on the realistic (top)
and stylized (bottom) animals for training (red) and testing (green) prompts
in stage-1 (solid lines) and stage-2 (dashed lines) as optimization progresses
from low-alpha in the start to high-alpha at the end of training. We display
results of $5$ evenly spaced points in training for stages 1 and 2, with
optimization horizons of $50$k iterations for realistic and $100$k for
stylized. Notably, the gap between seen (training) and unseen (testing)
prompts at the end of training is small, showing effective generalization. Our
Render-FID improves over training with a large jump from stage-1 to 2. Our
Mask-FID improves at the start of stage-2 when we convert to surface
rendering. The mask-FID is often lower for larger shape regularization,
showing shape regularization improves consistency with the source object,
illustrating that we can use $\alpha$ to enhance user-controllability towards
the input.  Stage 1Stage 2No-upsampleUpsample Figure 0.C.24:  We show the
results of our stage-2 fine-tuning with and without upsampling and the stage-1
result. Upsampling allows our method to better learn high-frequency details,
with quantitative assessments in Table 0.B.8.  Figure 0.C.25: Comparison
between Instant3d (left) and Latte3D (right).  We display our results next to
the results cropped from Instant3D paper. The Instant3D results are displayed
on the left for each pair of columns, with our result on the right. From left
to right, top to bottom the prompts are: a blue tulip, the Imperial State
Crown of England, a teal moped, a pile of dice on a green tabletop, a Space
Shuttle, a brain in a jar, a tree stump with an axe buried in it, a roulette
wheel, Coffee cup with many holes, a red convertible car with the top down, an
origami crane, a dalmation wearing a fireman’s hat, a dachsund wearing a
boater hat, a golden goblet, low poly, a majestic sailboat, a palm tree, low
poly 3d model, a pair of tan cowboy boots, studio lighting, product
photography, a typewriter

Figure 0.C.26:  Comparison between Instant3d (left) and Latte3D (right).  From
left to right, top to bottom, the prompts are: an orchid flower planted in a
clay pot, a model of a house in Tudor style, a bald eagle, a plate piled high
with chocolate chip cookies, an origami motorcycle, a stack of pancakes
covered in maple syrup, a bear dressed as a lumberjack, a hotdog in a tutu
skirt, edible typewriter made out of vegetables, a knight chopping wood, a
colorful camping tent in a patch of grass, a metal sculpture of a lion’s head,
highly detailed, a rotary telephone carved out of wood, a roast turkey on a
platter, an ice cream sundae, a drum set made of cheese, a beautiful violin
sitting flat on a table, a lion’s mane jellyfish

Figure 0.C.27:  Comparison between Instant3d (left) and Latte3D (right).  From
left to right, top to bottom, the prompts are: a small saguaro cactus planted
in a clay pot, a vintage record player, a ripe strawberry, a robot tiger, a
toy robot, Two locomotives playing tug of war, Wedding dress made of
tentacles, a steampunk space ship designed in the 18th century, a green
monster truck, a completely destroyed car, a baby grand piano viewed from far
away, a porcelain dragon, a spanish galleon sailing on the open sea, a cake in
the shape of a train, a human skull with a vine growing through one of the eye
sockets, a rabbit, animated movie character, high detail 3d model, a woolly
mammoth standing on ice, a classic Packard car

Figure 0.C.28:  Comparison between Instant3d (left) and Latte3D (right).  From
left to right, top to bottom, the prompts are:  a red cardinal bird singing, a
lionfish, a pug made out of modeling clay, A DMC Delorean car, a recliner
chair, an old vintage car, a ghost eating a hamburger, a red rotary telephone,
a plush t-rex dinosaur toy, studio lighting, high resolution, A DSLR photo of
pyramid shaped burrito with a slice cut out of it, a shiny silver robot cat,
an expensive office chair

Figure 0.C.29:  We show a screenshot of one user study we conducted in our experiment. The visualizations for objects A and B were displayed as a video of a camera sweeping over the azimuth.  Reconstruct$\sim\alpha=1$Retexture$\alpha=0.2$$\alpha=0.1$S1=SDS$\alpha=0$S1=MVDream _King Cobra_ _White-tailed Deer_ _Rottweiler_ _Blue crab_ Text-prompt Figure 0.C.30:  A qualitative comparison of our methods using various blending values $\alpha$. We display results using our method with $\alpha=1$ in the first stage to reconstruct geometry purely, then with retexturing for the text-prompt or purely reconstructing the input point-cloud RGB in stage-2. For $\alpha=0$, we also show results with an S1 guidance of MVDream, while for all other results, we use an S1 guidance of SDS.  400ms |  1min |  2min |  5min |  10min
---|---|---|---|---
Figure 0.C.31: Test time optimization. First column: without test time
optimization, $400ms$. Other columns: test time optimization with the given
time budget denoted.

##  Appendix 0.D Additional New Capability: User-controllability via
Amortizing the Regularization Strength

We also train our model in a way that allows users to select the
regularization strength $\alpha$ at inference. This provides another axis for
users to guide generation with 3D objects in real-time, analogous to guiding
generated results towards provided images in text-to-image generation tools.

We do this by simultaneously training on multiple different $\alpha$
(uniformly sampled in a range), which are also input to the network – i.e.,
amortizing over $\alpha$. We input $\alpha$ simply by linearly scaling the
text-conditional cross-attention residual added at each layer by $1-\alpha$.
Intuitively, this forces the magnitude of the text influence on the features
to be smaller for higher $\alpha$ and prevents any dependence at $\alpha=1$.

We contrast results with and without amortized loss weighting qualitatively
and quantitatively in Figs. 0.D.32 and 0.D.34 respectively, showing a minimal
loss in quality from non-amortized runs. This means that, for a small impact
on quality and compute, we can allow a user to select the loss weighting
dynamically at inference time.

Fig. 0.D.33 illustrates an example where we blend the RGB loss from the 3D
object, allowing users to smoothly interpolate between the 3D object and the
object from only the text-prompt. This is a 3D analog to the feature of text-
to-image models having a test-time user-controlled weighting towards a source
image. The interpolation is a 4D (a 3D animation), generating a separate 3D
object for each $\alpha\in[0,1]$.

$\alpha=.3$$\alpha=.3$$\alpha=.2$$\alpha=.2$$\alpha=.1$$\alpha=.1$$\alpha=.1$Amortized
BlendSingle Blend Figure 0.D.32:  We compare results with and without
amortized blending factors across various weightings $\alpha$ and
prompt/point-cloud pairings. Notably, we achieve comparable quality when
amortizing over the blending factor, allowing users to select the loss
weighting at inference time without re-training the model. Training each model
for the fixed $\alpha$ of $.3$, $.2$, or $.1$ costs the same amount of compute
as training the single amortized-weighting model, making the top row only a
third of the cost of the bottom row. Our strategy to amortize over $\alpha$ is
to make it an input to the network and sample it during training. Further,
Fig. 0.D.33 shows interpolation from $\alpha=0\to 1$ recovering the source 3D
data at $\alpha=1$ and purely the text-prompt at $\alpha=0$. Quantitative
results are shown in Fig. 0.D.34.  Interpolating $\alpha$ from $1\to 0$ with 1
model at inferencePoint-cloud Input _“Nemo (Clownfish)”_ _“King Cobra”_
_“European rabbit”_ _“Big Horn Sheep”_ _“Garden snail”_ _“Domestic Cat”_
Figure 0.D.33:  We display the results for an interpolation in the loss
blending value $\alpha$ for a single trained model amortizing over the
regularization strength (Sec. 0.D). The final output is a series of 3D objects
for each $\alpha$, effectively forming a 3D animation (4D). Here, the
regularization loss also includes an RGB loss on the point-cloud input.
Takeaway: Our model enhances controllability by allowing users to select how
to mix the reconstruction and text-prompt results in real-time during
inference. This is a 3D analogue of a user providing a source image and
weighting towards the image in text-to-image generation models.  Mask IoUMask
IoURender FIDRealistic AnimalsStylized Animals Figure 0.D.34:  We compare
models with an amortized blend-factor $\alpha$ and single blend-factor by
showing the frontier of the tradeoff between Mask IoU and Render-FID, at
various blending $\alpha$ (annotated) on the realistic (top) and stylized
(bottom) animals, for training (red/blue) and testing (green/cyan) prompts, in
stage-1 (solid lines) and stage-2 (dashed lines). The amortized-blend-factor
model receives the same compute budget as each single-blend-factor model. As
such, the single-blend-factor models receive more training at their respective
$\alpha$ values. Notably, after stage-2, we achieve similar qualities as
measured by FID, with stronger adherence to 3D data per $\alpha$ as measured
by a higher mask IoU. Qualitative results are shown in Figure 0.D.32.

## Author Contributions

All authors have significant contributions to ideas, explorations, and paper
writing. Specifically, XHZ and GJ led the research and organized team efforts.
KX developed fundamental code for experiments and led the experiments of pre-
training, stage-1 training, data curation, and evaluation. JoL led experiments
for stage-2 training and evaluation. TSC led experiments for integrating
MVDream prior, stage-1 training, and test-time optimization. JaL led
experiments for baselines for user study. GJ led the data preprocessing, pre-
training, and user study. XHZ led experiments for stage-1 training and data
curation. SF advised the research direction and designed the scope of the
project.