# GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and
Consistent 3D Generation

Baorui Ma1 Haoge Deng 2,1∗ Junsheng Zhou3,1 Yu-Shen Liu3 Tiejun Huang1,4
Xinlong Wang1
1 Beijing Academy of Artificial Intelligence 2 BUPT 3 Tsinghua University 4
Peking University Equal contribution.
Correspondence to<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Text-to-3D generation by distilling pretrained large-scale text-to-image
diffusion models has shown great promise but still suffers from inconsistent
3D geometric structures (Janus problems) and severe artifacts. The
aforementioned problems mainly stem from 2D diffusion models lacking 3D
awareness during the lifting. In this work, we present GeoDream, a novel
method that incorporates explicit generalized 3D priors with 2D diffusion
priors to enhance the capability of obtaining unambiguous 3D consistent
geometric structures without sacrificing diversity or fidelity. Specifically,
we first utilize a multi-view diffusion model to generate posed images and
then construct cost volume from the predicted image, which serves as native 3D
geometric priors, ensuring spatial consistency in 3D space. Subsequently, we
further propose to harness 3D geometric priors to unlock the great potential
of 3D awareness in 2D diffusion priors via a disentangled design. Notably,
disentangling 2D and 3D priors allows us to refine 3D geometric priors
further. We justify that the refined 3D geometric priors aid in the 3D-aware
capability of 2D diffusion priors, which in turn provides superior guidance
for the refinement of 3D geometric priors. Our numerical and visual
comparisons demonstrate that GeoDream generates more 3D consistent textured
meshes with high-resolution realistic renderings (i.e., 1024 $\times$ 1024)
and adheres more closely to semantic coherence. Our code and evaluation of 3D
metric are available at: GeoDream

##  1 Introduction

Diffusion models [41, 40, 39] have significantly advanced text-to-image
synthesis. This remarkable achievement has been reached by training scalable
generative models on a vast corpus of paired text-image data. Inspired by
their success, it is appealing to lift this success from 2D to 3D because this
achievement holds significant potential impacts on the modern game and media
industry. Template-based generators [3] and 3D native generative models [19,
52, 32, 34, 15] provide a natural and direct approach to the lift. However,
due to the massive and diverse 3D data required to train such generalized
models, these methods are usually limited to specific categories with
relatively simple topology and texture. Recently, the Score Distillation
Sampling (SDS) [35] and Variational Score Distillation (VSD) [53] have been
introduced to optimize 3D representations such that images rendered from any
viewpoints maintain a high likelihood, as evaluated by diffusion model
conditioned on a given text. This is an exciting direction because it allows
for generating 3D assets from any given text prompts, circumventing the need
for any 3D data. Despite these methods yielding satisfactory results on a wide
range of geometrically symmetrical 3D shapes, empirical observations indicate
that SDS and VSD losses still suffer from inconsistent 3D geometric structures
(Janus problems) [54] and severe artifacts [53, 45] with asymmetric geometry.
This is primarily due to the lack of 3D awareness in 2D diffusion models,
which inherently makes the lifting from 2D observations into 3D ambiguous.

Figure 1:  GeoDream alleviates the Janus problems by incorporating explicit 3D
priors with 2D diffusion priors. GeoDream generates consistent multi-view
rendered images and rich details textured meshes. We remove rendering
background to achieve a clearer visualization.

As a remedy, learning 3D priors from 3D datasets seems theoretically
reasonable and correct. However, 3D data remains expensive and sparse compared
to the plentifully available images. Therefore, the most promising avenue [37,
45, 47] presently is to equip 2D diffusion priors with 3D priors learned from
relatively limited 3D data, aiming to achieve the best of both worlds.
Recently, with the release of large-scale 3D datasets, Objaverse [6] and
Objaverse-XL [5], a few works [24, 20, 45, 58] have attempted to finetune pre-
trained 2D diffusion models using multi-view images rendered from 3D dataset.
This involves obtaining multi-view images from the fine-tuned diffusion model
conditioned on camera parameters and utilizing the clues of predicted multi-
view consistency to infer 3D information. Nevertheless, these methods rely
heavily on the consistency of content predicted across different source views.
Despite their efforts to employ 3D self-attention to exchange features between
different views [45], to correlate multi-view features using 3D-aware
attention [58], or to transform RGB predictions into coarser Canonical
Coordinates Map predictions [20] to mitigate the negative impact of
inconsistencies. Such inconsistencies between the predicted multiple views
become particularly noticeable, especially in imaginative and uncommon cases
beyond the training data distribution, resulting in over-smoothing and the
loss of semantic geometries in the generated 3D assets.

To resolve this issue, we introduce GeoDream, a novel method that incorporates
explicit generalized 3D priors with 2D diffusion priors to enhance the
capability of obtaining unambiguous 3D consistent geometric structures, while
maintaining diversity and high fidelity. Our contributions are listed below.
i) In stark contrast to the methods mentioned above that hinge heavily upon
the consistency between multi-view priors, we propose to obtain 3D native
priors within the 3D world space, which are well-suited to handle the inherent
lack of perfect consistency within the multi-view predicted priors, and
naturally free from inconsistencies caused by camera viewpoint transition. ii)
We justify that disentangling 3D and 2D priors is a potentially exciting
direction for maintaining both the generalization of 2D diffusion priors and
the consistency of 3D priors. In other words, providing hints through 3D
priors to unlock the great potential of 3D awareness in 2D diffusion priors,
without the need for invasive finetune 2D diffusion models.

Specifically, we start by reconstructing cost volume as native 3D priors by
aggregating the predicted multi-view 2D images into 3D space. Such aggregation
operations have been widely used in MVS-based techniques [57, 59, 26, 23],
which are known to be robust and generalized to provide valuable cues for
geometric reasoning. We find that such operations are well-suited for handling
imperfect and inconsistent multi-view predictions. The reason is that they
involve multi-view information aggregation, which helps filter out
inconsistent content to some extent, rather than dealing with each view
individually. Foremost, we conduct extensive experiments to demonstrate that
our proposed 3D priors adapt to multiple views predicted by various off-the-
shelf multi-view diffusion models, such as Zero123 [24], MVDream [45] and
Zero123++ [44]. Moreover, we introduce a critical viewpoint sampling strategy
to promote the stability of the 3D priors.

We further propose incorporating 3D priors with 2D diffusion priors in a
disentangled solution. Existing multi-view diffusion priors are equipped with
2D diffusion priors in a coupled way, including generating multiple views as
supervision [24, 44] or distilling the probability density as a loss [45, 20,
47, 37] to compute gradients for optimizing 3D representations. Instead, we
justify that leveraging the geometric clues provided by 3D priors can
effectively unleash the great potential 3D awareness capability inherent in 2D
diffusion priors, referred to as “disentangled design”. Very recent works have
started to explore how to evoke 3D-aware ability in 2D diffusion by altering
score functions [13] or negative text prompts [1]. These efforts have made
surprising progress, yet the performance remains unstable regarding 3D
consistency. Our insight is that going through geometric priors to unlock the
great potential of 3D awareness in 2D diffusion is a promising direction that
is both general and stable. Moreover, we rely solely on the awakened 3D-aware
capability of 2D priors to guide the optimization of Neural Implicit Surfaces
(NeuS) [50] without the supervision of 3D priors, thereby avoiding
compromising the inherent advantages of 2D priors in terms of generalization
and creativity. We show that 3D priors can be further refined to boost
rendering quality and geometric accuracy. The 2D diffusion priors benefit from
gradually evolved 3D priors, which in turn provide superior guidance for
unleashing the 2D priors. Finally, we use DMTet [43] to extract textured mesh
from optimized NeuS for mesh fine-tuning. Unlike previous work [35, 53]
attempt to increase the rendering resolution, which typically suffer from
over-saturation issues, we successfully increase the rendering resolution from
512 to 1024. We hypothesize that the improved results are aided and abetted by
3D priors that provide more plausible geometry and realistic texture, making
the optimization easier, because the rendered image is closer to diffused
distributions. To comprehensively evaluate semantic coherence, to our
knowledge, we are the first to propose $\rm{Uni3D_{score}}$ metric, lifting
the measurement from 2D to 3D.

Table 1: Comparison of design space.

Method | Prolific [53] | MVDream [45] | GSGEN [4] | Ours
---|---|---|---|---
Repr. | NeRF+DMTet | NeRF | Gaussian | NeuS+DMTet
Resolution | 512 | 512 | 512 | 1024
3D guidance | - | Multi-Views | Point-E | Cost volume
3D&2D | - | Entangled | Entangled | Disentangled
3D priors | Fixed | Fixed | Fixed | Optimizable

As summarized in Tab.1, we compared the latest methods[53, 45, 4] in design
space, including 3D representation, rendering resolution, forms of 3D
guidance, the disentangling of 3D and 2D priors and the optimizability of 3D
priors. As shown in Fig.1, GeoDream can yield 1024 $\times$ 1024 high-
resolution rendered images and high-fidelity textured meshes while greatly
alleviating the notorious Janus problems. In Sec.4.1, we conduct comprehensive
evaluations that demonstrate the superiority of the 3D assets generated by
GeoDream in terms of plausible geometry and delicate rendering details in
visual appearance.

##  2 Related Work

3D Generation Guided by 2D Priors. Deep generative models have driven the
field of 3D generation. Some efforts utilize Variational Auto Encoders (VAEs)
[17] for texture generation [11, 12], while Generative Adversarial (GAN)
Models [8] investigate 3D-aware GAN training [2, 7]. Thus far, diffusion
models have exhibited relatively better generalizability and training
stability for diverse object generation compared to GANs and VAEs, and thus
have gradually become recent focal points in 3D generation. Specifically,
recent endeavors attempt to leverage the potent 2D diffusion priors to aid 3D
generation by coupling it with a 3D representation, such as NeRF [31], DMTNet
[31], or NeuS [50], among others, bypasses the necessity for extensive text-3D
datasets for training 3D generative models. Such methods involve various
techniques, including score distillation sampling schedules like SDS [49], SJC
[35], and VSD [53] losses, which optimize the 3D representation by enhancing
high likelihood evaluated by the 2D diffusion models. A coarse-to-fine
training strategy [3] strengthens texture representation, decoupling geometric
and texture aspects of 3D representation for finer optimization [22],
improving 3D representation [48, 4]. Although these methods demonstrate the
ability to generate photo-realistic and diverse 3D assets with user-provided
textual prompts, they are prone to the notorious 3D inconsistency issues
(Janus problems) during the lifting process due to their reliance on 2D
diffusion models for training, which lack 3D knowledge. Despite some current
methods attempting to address 3D inconsistency by altering score functions
[13] or negative text prompts [1], performance remains instability in terms of
3D consistency. In this work, we aim to explore the distinctive advantages of
incorporating explicit 3D priors with 2D priors, enabling the generation of
highly detailed 3D objects while remarkably mitigating 3D inconsistency
issues.

3D Generation Guided by 3D Priors. Learning 3D priors from 3D datasets seems
theoretically reasonable and correct for enhancing the coherency of 3D
generation [24, 23, 22, 30, 55, 36]. Therefore, various 3D latent diffusion
models trained on 3D data have been recently introduced, including those using
Tri-plane [46] or feature grid [51, 16, 25] encoding 3D representations into
the latent space. Additionally, OpenAI has explored models aiming to directly
generate 3D formats using several million internal 3D shapes, such as point
clouds [34] or neural radiance fields [15]. However, their generalizability to
the scope of their 2D counterparts remains unverified, due to the relative
sparsity of 3D data compared to the abundance of available 2D images.
Consequently, the most promising avenue currently is to equip 2D diffusion
priors with 3D priors learned from relatively limited 3D data, intending to
achieve the best of both worlds. Recently, with the release of a large-scale
3D dataset called Objaverse [6] and Objaverse-XL [5], some work [24, 56, 23,
20, 45, 58] has attempted to fine-tune pre-trained 2D diffusion models using
multi-view images rendered from 3D data. This aims to generate multi-view
images from the fine-tuned diffusion model conditioned on camera parameters
and utilize the clues of predicted multi-view consistency to assist in
inferring 3D information. Nevertheless, these methods heavily depend on the
absolute consistency of content predicted across different views. Nonetheless,
their efforts to utilize 3D self-attention [45, 56] for feature exchange
between different views, to correlate multi-view features using 3D-aware
attention [58], or to transform RGB predictions into coarser Canonical
Coordinates Map predictions [20] for mitigate the negative impact of
inconsistencies. The performance of such methods frequently exacerbates
inconsistencies and unrealistic rendering quality in uncommon cases, due to
the absence of explicit constraints between different predicted viewpoints
within 3D space. In this work, we incorporate explicit generalized 3D priors
into 2D diffusion priors. These explicit 3D priors fundamentally ensure
consistency in 3D space and avoid the independence of multi-view priors across
source views.

Figure 2: The overview of GeoDream. (a) 3D priors training. (b) Incorporating
3D priors with 2D diffusion priors.

##  3 Method

We focus on generating 3D content with consistently accurate geometry and
delicate visual detail, by equipping 2D diffusion priors with the capability
to produce 3D consistent geometry while retaining their generalizability. The
overview of GeoDream is shown in Fig.2. GeoDream consists of the following two
stages. i) During 3D priors training, we build upon the One-2-3-45 [23], which
encodes geometry into cost volume $V$ and geometry MLP decoder $f_{g}$. In
addition, the appearance of the object is modeled to cost volume $V$ and
texture MLP decoder $f_{t}$. We refer to the trained geometric decoder $f_{g}$
and appearance decoder $f_{t}$ with cost volume $V$ as native 3D geometric
priors and appearance priors, as shown in Fig.2 (a). Detials in Sec.3.1. ii)
During priors refinement, we show that geometric priors can be further fine-
tuned to boost rendering quality and geometric accuracy by combining a 2D
diffusion model, as shown in Fig.2 (b). Detials in Sec.3.2.

###  3.1 Generalizable 3D Priors Training

We start by reconstructing cost volume $V$ as native 3D priors by aggregating
the 2D image features into 3D space, which provides valuable cues for
geometric reasoning in the priors refinement stage.

Cost Volume Construction. Following MVS-based methods [57, 59, 26, 23], given
multi-view images $I=\\{(I_{i})^{N-1}_{i=0}\\}$, we extract 2D feature maps
$F=\\{(F_{i})^{N-1}_{i=0}\\}$ using a 2D feature network $f_{2D}$. Volume
reconstruction model takes posed 2D feature maps $F$ as input and outputs cost
volume $V$ with per-voxel neural features in voxels. Specifically, for each
voxel centered at 3D location $h$, per-voxel neural feature is computed by
projecting each location $h$ to $N$ image feature planes and then fetching the
variance of the features at the the location of the projection. We use
$\mathrm{~{}Var}$ to denote the variance operation and $P$ to denote the
projection procedure. We then use a sparse 3D CNN $f_{3D}$ to process the
variance features per voxel to regress the cost volume, as formulated by,

$V=f_{3D}(\mathrm{~{}Var}\\{P(F_{i},h)\\}_{i=0}^{N-1}),$ (1)

where the variance operation is invariant to the number $N$ of input images.
We find that such an operation is well-suited for handling imperfect and
inconsistent multi-view predictions, due to involving information aggregation
rather than dealing with each view individually.

Geometry and Texture Decoder. The cost volume $V$ is directly decoded into
signed distance function values (SDF) and color information using the
corresponding geometry MLP decoder $f_{g}$ and texture MLP decoder $f_{t}$.
For any arbitrary query point $x\in\mathbb{R}^{3}$, we get the SDF $s$ and
color $c$ as

$s(x)\,=\,f_{g}(E(x),V(x)),$ (2)
$c(x)=f_{t}(\\{P(F_{i},x)\\}_{i=0}^{N-1},V(x),\\{\Delta
d_{i}\\}_{i=0}^{N-1}),$ (3)

where $E$ denotes position encoding, $V(p)$ denotes tri-linearly interpolated
feature from cost volume at query point $x$, $\Delta d_{i}=d-d_{i}$ is the
viewing direction of the query ray relative to the viewing direction of the
$i{th}$ multi-view image.

The final rendered image $I{{}^{\prime}}$ is achieved via SDF-based
differentiable volume rendering $R$. In this work, we get the pre-trained
parameters of the $f_{g}$, $f_{t}$, and $f_{3D}$ networks from the One-2-3-45
[23], which is trained on ground truth images $I$ rendered from the Objaverse
dataset with a loss

$\mathcal{L}_{rgb}=||I-I{{}^{\prime}}||_{2},$ (4)

where $I{{}^{\prime}}=R(\\{s(x_{j}),c(x_{j}))\\}_{j=0}^{M-1})$, $M$ denotes
sampling $M$ query points along the ray of viewing direction.

###  3.2 Priors Refinement

We present how to further finetune the geometric priors obtained from 3D
priors training stage, i.e., optimizable cost volume $V$ and the fixed pre-
trained geometric decoder $f_{g}$, using the 2D diffusion priors, as shown in
Fig.2 (b). During priors refinement stage, we replace the $N$ ground truth
rendered images with multi-view diffusion model predictions. In contrast to
One-2-3-45, GeoDream is not limited to the Zero123 [24] predictions. We
conduct extensive experiments with various multi-view diffusion models, such
as MVDream [45] and Zero123++ [44]. We also introduce a critical viewpoint
sampling strategy to ensure GeoDream robustly adapts to various multi-view
diffusion models, rather than being limited to just one. Overall, we justify
that by decoupling 3D and 2D diffusion priors, GeoDream unlocks the immense
potential of 3D awareness in the 2D diffusion model, avoiding the tendency to
produce canonical views, resulting in 3D assets featuring multiple faces and
collapsed geometry. Thanks to the decoupling, GeoDream maintains the
generalization and imaginativeness of 2D diffusion priors, while also
exploring the significant role that geometric priors play in improving
appearance modeling.

Multi-View Images Generation. The rapid advancement of 3D generation has
provided a wide range of methods available for generating multi-view images,
such as Zero123[24], MVDream[45], and Zero123++ [44]. Given a set of
predefined camera poses $\\{(R_{i},T_{i})^{N-1}_{i=0}\\}$ and a user-provided
condition $c$, we utilize a fixed multi-view diffusion $f_{mv}$ to predict
posed images $I_{p}=\\{(I^{p}_{i})^{N-1}_{i=0}\\}$ and extract 2D feature maps
$F_{p}=\\{(F^{p}_{i})^{N-1}_{i=0}\\}$,

$F^{p}_{i}=f_{2D}(f_{mv}(c,R_{i},T_{i})),$ (5)

where $R\in\mathbb{R}^{3\times 3}$, $T\in\mathbb{R}^{3\times 3}$ respectively
denote relative camera rotation and translation of the default viewpoint.

3D Geometric Priors. By replacing $F_{i}$ in Eq.1 into $F^{p}_{i}$, we obtain
the value of SDF at an arbitrary query point $x$ defined in Eq.2,

$V_{p}=f_{3D}(\mathrm{~{}Var}\\{P(F^{p}_{i},h)\\}_{i=0}^{N-1}),$ (6)
$s_{p}(x)\,=\,f_{g}(E(x),V_{p}(x)),$ (7)

where $s_{p}(x)$ is treated as a geometric prior since it encodes the hidden
geometric clues in the predicted multiple views.

Texture Decoder. We propose to drop the pre-trained texture priors $f_{t}$
defined in Eq.3 because we empirically find that texture priors tend to
generate 3D assets with lighting and texture styles similar to the rendered
dataset. We choose Instant NGP [33] for efficient high-resolution texture
encoding. Specifically, for any arbitrary query point $x\in\mathbb{R}^{3}$, a
learnable hash encoding $h_{\Omega}$ is decoded into a color $c$ using
initialized texture decoder $f_{t}^{\prime}$, as formulated by,

$c_{p}(x)=f_{t}^{\prime}(h_{\Omega}(x),x),$ (8)

where $h_{\Omega}(x)$ denotes the looked-up feature vector from $h_{\Omega}$
at query point $x$.

Texture and Geometry Refinement. To incorporate 3D geometric priors with 2D
diffusion priors, we minimize the VSD loss introduced in ProlificDreamer [53]
to optimize the parameters of ${\theta}_{1}$ in cost volume $V$,
${\theta}_{2}$ in hash encoding $h_{\Omega}$ and ${\theta}_{3}$ in texture
decoder $f_{t}^{\prime}$. At each iteration, we sample a camera pose $o$ from
a pre-defined distribution. We render 2D image $\hat{x}$ at pose $o$ by
combining Eq.7 and Eq.8 via differential rendering $R$. Our objective function
during priors refinement is to minimize the VSD loss $\mathcal{L}_{VSD}$, the
corresponding gradient
$\nabla_{{\theta}_{1},{\theta}_{2},{\theta}_{3}}\mathcal{L}_{VSD}$ is

$\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial({\theta}_{1},{\theta}_{2},{\theta}_{3})}],$
(9)

where $\hat{x_{t}}$ denotes a noisy rendered image in timestep $t$, $w(t)$
denotes a weighting function, $\epsilon_{pretrain}$ is a 2D pretrained
diffusion model and $\epsilon_{l}$ is a trainable LoRA [14] diffusion model
with parameters of $l$. We propose to fix the geometry decoder $f_{g}$
conjointly with a learning rate decay strategy for the cost volume, aiming to
maintain geometric priori cues as well as tuning to achieve better details in
the early stage of optimization. More details on viewpoint sampling and
learning rate decay strategy are provided in Sec.4.2.

Mesh Fine-tuning. For high-resolution rendering, we use DMTet [43] to extract
textured 3D mesh representation from optimized NeuS [50]. By minimizing the
loss in Eq.9, we follow ProlificDreamer [53] first to optimize the geometry
using the normal map and then optimize the texture. We empirically find that
we can increase the rendering resolution from 512 to 1024. But unlike previous
work [35, 53], attempting to increase the rendering resolution suffers from
over-saturation issues. We successfully increase the rendering resolution from
512 to 1024. We hypothesize that well-optimized results are aided and abetted
by 3D priors that provide more plausible geometry and realistic texture,
making the optimization easier, because the rendered image $\hat{x}$ is closer
to diffused distributions at each iteration.

##  4 Experiment

###  4.1 Results of GeoDream

Baselines. We report our performance with the latest 3D generation methods,
including DreamFusion [35], ProlificDreamer [53], MVDream [45], GSGEN [4],
Fantasia3D [3] and Wonder3D [27]. Specifically, DreamFusion [35], Fantasia3D
[3] and ProlificDreamer [53] adopt a similar approach to optimize 3D
representation through the score function of a 2D diffusion model, without
intervening in 3D priors. We compare our results with these three methods,
highlighting the distinct advantages of inferring 3D-consistent geometry and
reducing artifacts by incorporating explicit 3D priors. Meanwhile, MVDream
[45] and Wonder3D [27] are very recent proposals to use multi-view consistency
priors, which derived from finetuned multi-view diffusion models trained on
synthetic multi-view rendering image data. GSGEN [4], on the other hand,
addresses 3D inconsistency by initializing geometry with Point-E [34]
generated shapes. By comparing these three methods, we demonstrate that our
introduced 3D priors offer greater generality in challenging and uncommon
cases and effectively prevent the generation of 3D assets with lighting and
texture styles similar to the synthetic rendered dataset. For DreamFusion
[35], ProlificDreamer [53] and Fantasia3D [3], we utilize their
implementations in the ThreeStudio [9] library for comparison. For MVDream
[45], GSGEN [4] and Wonder3D [27], we use their official implementation.

Figure 3: Qualitative comparison with baselines. Back views are highlighted
with red rectangles for distinct observation of multiple faces.

Experiment Setup. We collected 35 prompts from various sources, including
prompts from previous work [45, 27] and real user inputs in the wild. To
comprehensively assess 3D consistency and semantic coherence, we intentionally
selected more prompts indicating asymmetric geometric structures (80% of the
collected prompts) and fewer prompts indicating symmetric geometric structures
(20%). For a fair comparison, we render 3D assets generated by our method and
baselines by circling around the object at a default elevation and camera
distance [9], resulting in 120 images. We then evaluate the gap between the
rendered images and reference images generated by Stable Diffusion [40] based
on the collected prompts. We sample 10$k$ points on the generated meshes to
calculate 3D metric. To demonstrate that our method is trivially adaptable to
various multi-view diffusion models, we randomly use either Zero123 [24] or
MVDream [45] and Zero123++ [44] for subsequent experiments. For the effect of
different diffusions on the results, please refer to supplementary for detail.

2D Metrics. $\rm{FID_{CLIP}}$ [18] for image fidelity measurement, which is
calculated by the disparity in distribution between the rendered image and
reference image features, both encoded by CLIP ViT-B-32 [38]. CLIP R-score for
semantic coherence measurement is calculated by the probability of rendered
images retrieving the correct caption among collected prompts. We average the
metric over 120 rendered images for the quantitative comparison.

3D Metric. These metrics mentioned above are for measuring 2D images. Limited
by rendering angles and geometric self-occlusion, 2D metrics often struggle to
assess 3D objects in 360 degrees fully. To the best of our knowledge, no
metrics have yet been introduced in text-to-3D tasks for evaluating the
semantic consistency of 3D assets. Therefore, we propose using Uni3D [60], the
largest 3D presentation model with one billion parameters under text-image-
pointcloud alignment learning objective, to lift semantic coherence
measurement from 2D to 3D. We adopt a similar strategy to the CLIP R-score,
except that we replace the image and text encoders in the CLIP with the point
cloud and text encoders from the Uni3D, referred to as “$\rm{Uni3D_{score}}$”.

Subjective Metric. 3D reconstruction tasks are typically evaluated of the
error reconstructed shape compared to the ground truth [29]. However, these
metrics are difficult to apply to text-to-3D tasks, as there is no ground
truth. We further manually check the number of examples with 3D or semantic
inconsistency problems, and then report the rate of success as an auxiliary
metric, referred to as “Cons. Rate”.

Quantitative Comparison. In Tab.2, we conduct a quantitative comparison over
generation quality, text-image consistency and 3D consistency. Overall, the
results indicate that our method significantly outperforms the baselines
across all metrics, demonstrating that we achieve high-fidelity, text-image
and text-3D consistency in the generated quality while ensuring 3D spatial
consistency.

Table 2: Quantitative comparison with baselines.

Model |  ${\rm{FID}}_{\rm{CLIP}}$ $\downarrow$ | CLIP R-score$\uparrow$ |  ${\rm{Uni3D}}_{\rm{score}}$ $\uparrow$ | Cons. Rate$\uparrow$
---|---|---|---|---
|  | B/16 | L/14 |  |
DreaFusion [35] | 59.6 | 0.844 | 0.870 | 0.514 | 0.429
ProlificDreamer [53] | 48.8 | 0.866 | 0.892 | 0.629 | 0.257
MVDream [45] | 50.6 | 0.852 | 0.886 | 0.771 | 0.829
Ours | 47.9 | 0.935 | 0.962 | 0.800 | 0.914

Qualitative Comparison. Fig.3 compares our method with the baselines. We
present four visual examples: the first three rows depict non-symmetric
geometries, while the last row is for symmetric geometry. Notably, we display
the front, side, and back views, where the back views are highlighted with red
rectangles to enhance the observation of potential multiple faces issues. We
highlight our improvements in visual comparison in Fig.3. Dreamfusion and
ProlificDreamer produce high-quality frontal views but fail to form a
plausible 3D object. In particular, ProlificDreamer delivers photorealistic 3D
assets with semantic coherence, where every view resembles canonical views,
i.e., the back views that are shown in red rectangles, are mistakenly
optimized as front views, resulting in Janus problems. GSGEN mitigates some of
the 3D inconsistencies by introducing 3D priors from the pre-trained Point-E.
However, the fidelity of the textures it generates is still insufficient for
complete satisfaction. Compared to the three methods mentioned above, MVDream
stands out as the most effective solution for addressing multi-view
inconsistency issues. This is achieved by fine-tuning pre-trained 2D diffusion
models using multi-view images rendered from 3D data. Nevertheless, due to the
rendering quality and sparsity of 3D training data, the generated results
often exhibit cartoon-style textures and semantically lost geometries,
particularly when dealing with uncommon and challenging given prompts. For
example, it struggles to generate a rocket as required in the second case, a
samurai style as required in the third case, and a log as required in the
fourth case. By incorporating explicit 3D priors with a 2D diffusion model
that is capable of imagination diversity, GeoDream significantly alleviates
the multifaceted nature of generated 3D assets, in terms of both meshes and
rendered images exhibiting impressive photorealistic textural details, while
maintaining semantic faithfulness, as shown in Fig.1 and Fig.3. More analysis
and comparisons with other baselines can be found in the supplementary.

Figure 4: Ablation study of proposed improvements for text-to-3D generation.

###  4.2 Ablation Study

We then conduct ablation studies to justify the effectiveness of each design
in GeoDream. We activate all the modules and training strategies mentioned in
the Sec.3 during ablation studies, except for the modified part described in
each ablation experiment below.

The Effect of 3D Priors. We first visualize the initial cost volume obtained
through the volume construction model, as shown in Fig.4 (a). Fig. 4 (a)
combined with Fig.4 (e) demonstrate that relying solely on rough geometric
cues can significantly activate potential of 3D awareness in 2D diffusion,
alleviating the character’s tendency to exhibit multifaceted issues. In
contrast to fixed priors in Fig.4 (b), we propose using optimizable priors
that gradually evolve according to the optimization state, thus producing
progressively refined results, as shown in Fig.4 (e) and Fig.4 (j). To further
assess its impact, we also attempt to deactivate the cost volume, i.e.,
randomly initializing the 3D prior. The 3D inconsistency issue also arises, as
shown in Fig.4 (f). To assess the impact of the learning rate decay schedule,
an ablation study is conducted, where the learning rate of the cost volume is
set to a suitable constant value. The generated 3D assets still suffer severe
degeneration, resulting in a completely collapsed geometry in Fig.4 (c). The
reason is that, during the early stage of optimization, there may be a lot of
ambiguity and conflict in the appearance information across different views.
Hence, during the early optimization stage, we propose to set the learning
rate of the cost volume to a smaller value and gradually increase it for
geometric detail optimization. And vice versa for the learning rate of
texture, which can prevent content drift in the later stage of optimization,
please refer to supplementary for detail.

We further justify whether we should use texture priors. We report a visual
result using a pre-trained texture MLP in Sec.3.1, rather than reinitializing
the MLP network and hash encoding in Sec.3.2. Fig.4 (g) demonstrates that
introducing texture priors generally leads to a visual appearance that tends
toward non-photorealism and over-smoothing. This observation underlines the
necessity of introducing only 3D geometric priors, which only contribute to
the geometry modeling during the lifting, avoiding compromising the appearance
modeling due to texture priors.

The Effect of Mesh Fine-tuning. We convert NeuS to DMTet to improve geometric
and appearance details. We first show the NeuS-based visual results in Fig.4
(h). GeoDream produces better results with finer details, as evidenced in
Fig.4 (j). The reason is that the advantages of the 3D assets we generate,
which yield improved 3D consistency, lie in the ability to enhance the
accuracy of surfaces, thereby reducing the complexity of texture optimization
in the DMTet. Fig.4 (d) presents an ablation study on SDS and VSD loss. SDS is
observed to produce over-saturated textures, as opposed to the VSD loss that
we default to using.

The Effect of Rendering Resolution. Through empirical experimentation, we
deduce that collapsed geometry often results in textural distortions, thereby
increasing the difficulty of optimization. Hence, we conjecture that 3D
consistency is one of the main bottlenecks for increasing the rendering
resolution in prior work. Instead, by integrating 3D geometric priors, we
achieved better results closer to diffused distributions, making the
optimization becomes easier. Consequently, we successfully increase the
rendering resolution from 512 to 1024, as shown in Fig.4 (j). Additionally,
Fig.4 (i) demonstrates that GeoDream still provides competitive results at 512
$\times$ 512 resolution.

##  5 Conclusion

We significantly improve the rendering fidelity of images and the details of
texture meshes, while greatly alleviating the notorious Janus problems by the
awakened 3D-aware capability of 2D diffusion priors, which is unleashed by
geometric clues provided by 3D priors in a disentangled solution.
Additionally, the disentangled design offers a flexible way to optimize 3D
priors gradually. The visual and numerical comparisons with the state-of-the-
art methods justify our effectiveness and show our superiority over the latest
methods in 3D generation.

Supplementary Material

##  6 Video

Our supplementary material also includes a video, which shows more
visualizations, inviting readers to watch for a more intuitive visual
experience.

Figure 5: More visualization comparisons with baselines. For each row from up
to down, the given prompts are: (1) _3D render of a statue of an astronaut._
(2) _3D stylized game little building._ (3) _A brightly colored mushroom
growing on a log._ (4) _An ice-cream cone_

##  7 Source Code

To facilitate future research, our code and 3D metric are available at:
GeoDream

##  8 Definition of The Janus Problem

We explain in further detail the definition of the Janus problem (3D
inconsistency), which refers to a phenomenon that the learned 3D
representation, instead of presenting the 3D desired output, shows multiple
canonical views of an object in different directions [54, 1]. For instance,
when the given prompt indicates an asymmetric geometric structure, such as a
person or an animal, the generated 3D asset has multiple faces but lacks
complete and correct back views. In contrast, when the given prompt indicates
a symmetric structure, such as a cake or a hamburger, which does not have
strictly defined back views, issues of 3D inconsistency typically do not
arise. Therefore, when calculating the subjective metric, geometrically
symmetric 3D assets do not suffer from 3D inconsistency by default.

##  9 More Visualization Comparisons with Baselines

We report our performance with more 3D generation methods, including
Fantasia3D [3], Wonder3D [27], and Magic123 [22]. Fantasia3D employs DMTet
[43] initialized with a handcrafted 3D model or a predefined geometric shape
as the 3D representation, which is the same representation used in our mesh
fine-tuning phase. We compare our DMT-based results with Fantasia3D to show
the gains in rendering appearance from geometry initialization with 3D priors.
Wonder3D employs NeuS [50] as its 3D representation, which is subsequently
processed through the Marching Cube algorithm [28] to extract mesh. Magic123
adopts a coupled approach, optimizing the 3D representation by using both 3D
and 2D priors as losses. The comparisons with Magic123 justify that the
disentangling 3D and 2D priors allows for the simultaneous harnessing of the
generalization capabilities of 2D diffusion priors and the 3D consistency of
3D priors. In contrast, Magic123 requires careful design of the balance
weights between 3D and 2D loss to avoid compromising between the two types of
priors. Visual comparisons in Fig.5 reveal that we enhance the fidelity and
semantic coherence of the generated 3D assets, accompanied by an absence of
geometric and textural distortions, indicating excellent 3D spatial
consistency.

Figure 6: The detailed learning rate schedule.

##  10 Viewpoint Sampling Strategy

Figure 7: Ablation on the methods for obtaining reference views. We compare
the generated 3D assets based on reference views predicted by Stable Diffusion
and MVDream, driven by user-provided texts. GeoDream adapt to reference views
from various sources. For each row from up to down, the given prompts are: (1)
_A majestic giraffe with a long neck._ (2) _Viking axe, fantasy, weapon,
blender, 8k, HD._ Figure 8: Ablation on the viewpoint sampling strategy. We
demonstrate that using our proposed viewpoint sampling strategy contributes to
the more robust generation of a consistent cost volume, significantly avoiding
the outcomes of geometric collapse. For each row from up to down, the given
prompts are: (1) _A dinosaur toy._ (2) _A corgi._

We propose a critical viewpoint sampling strategy to enhance the stability of
constructing cost volumes. Cost volume-based methods [57, 59, 26, 23] rely on
the consistency and accuracy of multi-views to find local correspondences and
infer geometry. We empirically find that current multi-view diffusion models
[24, 56, 23, 20, 45, 58] can provide relatively accurate and consistent
predictions for small relative pose, when fed with front and side views as
reference views. Instead, when a back view is used as the reference view,
inconsistencies tend to worsen. Our analysis indicates that these multi-view
models are fine-tuned from 2D pre-trained diffusion models, which exhibit
weaker performance in predicting non-canonical view information. Additionally,
the information implied by back views is quite ambiguous, posing challenges
for predicting consistent information. Consequently, we propose a viewpoint
sampling strategy to mitigate the aforementioned problems.

Specifically, We obtain reference views driven by a user-provided text in one
of two methods: i) Obtaining a front view predicted by Stable Diffusion [40],
which is trivial as Stable Diffusion often biases towards generating canonical
views. ii) Utilizing MVDream [45] to output desired views based on our
predefined absolute camera positions. In our experiments, following the
default settings of MVDream, we set the absolute elevation angle at
$15^{\circ}$ and absolute azimuth angles at $0^{\circ}$, $90^{\circ}$,
$180^{\circ}$, and $270^{\circ}$. We sample four viewpoints on the sphere
surface with a default radius to obtain the front, left, back, and right views
as reference views.

When the reference view is predicted by Stable Diffusion, we require either
Zero123 [24] or Zero123++ [44] to randomly sample viewpoints within a range of
a relative azimuth angle less than $180^{\circ}$ and a relative elevation
angle less than $30^{\circ}$. Subsequently, we sample an image with a relative
azimuth angle of $180^{\circ}$ and a relative elevation angle of $0^{\circ}$
to serve as the back view, which is then added to the source views. In the
case of reference views are predicted by MVDream, we use Zero123 or Zero123++
to sample viewpoints relative to the front view, left side view, and right
side views, within a range of a relative azimuth angle less than $45^{\circ}$
and a relative elevation angle less than $30^{\circ}$. Subsequently, the back
view predicted by MVDream is supplemented to the source views. We show the
visualized comparison of the impact of reference views generated by Stable
Diffusion and MVDream on the generated 3D assets, as shown in Fig.7. We report
visualized results without viewpoint sampling strategy and the results with
viewpoint sampling strategy, as shown in Fig.8. The visualized results
indicate that our proposed sampling strategy can adapt to reference views
predicted by both Stable Diffusion and MVDream, significantly enhancing the
quality of the constructed cost volume and the consistency of the generated 3D
assets.

Finally, we observe that due to the inherent lack of perfect consistency
between source views, the constructed cost volume is quite rough, even with
the viewpoint sampling strategy, as shown in Fig.7 and Fig.8. However, the
ultimately generated 3D assets tend to produce rich details and more complete
and consistent geometry. This suggests that disentangling 3D and 2D priors is
a potentially exciting direction, as it provides a flexible way to further
refine 3D priors while maintaining the ability of 3D priors to unleash 2D
diffusion priors.

##  11 Learning Rate Decay Schedule

We propose to set the learning rate of the cost volume to a smaller value and
gradually increase it for geometric detail optimization, aiming to maintain
geometric priori cues in the early stage of optimization. And vice versa for
the learning rate of texture, which can prevent content drift in the later
stage of optimization. During the early optimization stage, we adopt an
initially high learning rate to fight early overfitting [21, 10]. The detailed
learning rate curves are depicted in Fig.6.

Figure 9: Ablation on negative prompting, rendering resolution, and corner
case. The given prompts are: (a) and (b) _A 3D printed white bust of a man
with curly hair._ (c) _An astronaut riding a horse._ (d) and (e) _A DSLR photo
of a squirrel playing guitar._

##  12 Ablation on negative prompting, rendering resolution, and corner case

Prompting. Perp-Neg [1] introduces a negative prompt algorithm that transforms
2D Diffusion into 3D, addressing the Janus problem. We attempt to integrate
the negative prompt algorithm into both ProlificDreamer and GeoDream, as shown
in Fig.9 (a) and Fig.9 (b). The result shown in Fig.9 (a) demonstrates that
the negative prompt algorithm still fails to mitigate the Janus problem
stably. Fig.9 (b) illustrates that GeoDream is able to yield consistent 3D
assets both with and without the negative prompt algorithm. However, since we
did not observe a significant improvement in the results, we opt not to use
the negative prompt algorithm as a default in our experiments. Instead, we
employ view-dependent prompting as in previous works [35, 53].

Rendering Resolution. We attempt to increase the rendering resolution to 1024
in ProlificDreamer, which typically struggles with over-saturation issues, as
demonstrated in Fig.9 (c). Our analysis suggests that the absence of 3D priors
often leads to collapsed geometry, resulting in textural distortions and
thereby increasing the complexity of the optimization.

Corner Case. We further explore the robustness of GeoDream when faced with
failures of multi-view diffusion in predicting multiple views. For instance,
when the given prompt is “ _A DSLR photo of a squirrel playing guitar_ ”,
multi-view diffusion struggles to accurately predict the correct spatial
relationship between the guitar and the squirrel, due to the sparsity of 3D
training data. However, GeoDream excels in preserving the generalizability and
creativity of 2D diffusion priors, enabling more effective compatibility with
imperfect multi-view predictions, and thus generating semantically correct 3D
assets, as shown in Fig.9 (e).

##  13 Training Stability and Diversity

Stability. Prior text-to-3D studies are notoriously brittle. The same
hyperparameter settings often lead to vastly different results in terms of
complete failure or success, depending on the random seed, making them hard to
control. To assess the training stability of GeoDream, we conduct several
experiments on the same prompt, as shown in Fig.10. GeoDream exhibits
exceptional training stability. The reason lies in the 3D priors we
introduced, which significantly reduce the randomness caused by the random
seeds.

Diversity Additionally, we can generate diverse 3D models by controlling and
leveraging the diversity capabilities of Stable Diffusion or MVDream to
predict various reference views, as mentioned in Sec.10 and Fig.7. In summary,
GeoDream provides a balanced solution between diversity and stability.

Figure 10: Ablation on training stability. We conduct several experiments on
the same prompt to verify the training stability of GeoDream. The given prompt
is: _An astronaut riding a horse._

##  14 Licenses

We provide the URL, citations, and licenses of the open-sourced assets we used
in this work, as shown in Tab.3.

##  15 Algorithm

We provide a summarized algorithm of priors refinement in Algorithm 1.

##  16 Training Details

We construct a cost volume with 150 $\times$ 150 $\times$ 150 voxels in 2
minutes on an NVIDIA-V100-32GB GPU. During the priors refinement stage, we
employ a network modified based on ProlificDreamer [53]. We replace the
learnable hash encoding used in ProlificDreamer by cost volume. We choose a
single-layer MLP to decode the color from texture hash encoding as Instant-NGP
[33]. Following ProlificDreamer, we set the particle to 1 and utilize
v-prediction [42] to train the LoRA [14] based on Stable Diffusion v2.1 model
for VSD loss. Notably, even when the rendering resolution increased from 512
to 1024, the training time does not show a significant difference compared to
ProlificDreamer. The reason is that 3D assets generated by GeoDream, which
exhibit fewer artifacts and thus enhanced rendering efficiency. Specifically,
training the NeuS representation [50] with the batch size set to 1 typically
requires approximately 3 hours on a single NVIDIA-V100-32GB GPU. Mesh
finetuning with a batch size of 2 usually requires around 8 hours on a single
NVIDIA-V100-32GB GPU. Utilizing larger batch sizes and parallel multi-GPUs
training could potentially reduce training times and we leave this exploration
in future work.

##  17 Ablation on Source Views Predicted by Different Multi-View Diffusion
Models

To demonstrate that GeoDream is trivially adaptable to various multi-view
diffusion models, we conduct the visual comparison with our generated 3D
assets based on either Zero123 or Zero123++. Specifically, for a fair
comparison, the reference views are generated by MVDream driven by user-
provided texts. Then, employing the viewpoint sampling strategy proposed in
Sec.10, we obtain source views predicted by Zero123 or Zero123++. Fig.11 and
Fig.12 show the comparison of our generated 3D assets based on source views
predicted by Zero123 and Zero123++. Fig.11 and Fig.12 illustrate that GeoDream
can adapt to different multi-view diffusion models, producing 3D assets with
plausible geometry and intricate rendering details in visual appearance. The
adaptability and seamless integration of GeoDream with various multi-view
diffusion models highlight the evolutionary potential of GeoDream, alongside
the future advancements of multi-view diffusion models.

Table 3: URL, citations and licenses of the open-sourced assets we used in this work. URL | Citation | License
---|---|---
https://github.com/threestudio-project/threestudio | [9] | Apache License 2.0
https://github.com/bytedance/MVDream | [45] | Apache License 2.0
https://github.com/One-2-3-45/One-2-3-45 | [23] | Apache License 2.0
https://github.com/cvlab-columbia/zero123 | [24] | MIT License
https://github.com/SUDO-AI-3D/zero123plus | [44] | Apache License 2.0
https://github.com/huggingface/diffusers | [40] | Apache License 2.0
https://github.com/allenai/objaverse-xl | [5, 6] | Apache License 2.0

Input: A condition $c$, rotation and translation matrix
$\\{(R_{i},T_{i})^{N-1}_{i=0}\\}$, voxel location $h$, the variance operation
$\mathrm{~{}Var}\\{\cdot\\}$, the projection procedure $P(\cdot,\cdot)$,
multi-view diffusion $f_{mv}$, a 2D feature network $f_{2D}$, a 3D feature
network $f_{3D}$, a geometric decoder $f_{g}$, texture decoder
$f_{t}^{\prime}$ , position encoding $E(\cdot)$, 2D diffusion model
$\epsilon_{pretrain}$. Learning rate $\eta_{1}$, $\eta_{2}$,
$\eta_{3}$,$\eta_{4}$ and $\eta_{5}$ for cost volume $V$, hash texture
encoding $h_{\Omega}$, texture decoder $f_{t}^{\prime}$, a LoRA diffusion
model $\epsilon_{l}$ and DMTet parameters, respectively.

1

2Initialize 2D feature network $f_{2D}$, 3D feature network $f_{3D}$, and
geometry MLP decoder $f_{g}$ with pretrained parameters obtained from 3D
priors training stage. Initialize texture hash encoding and texture decoder
$f_{t}^{\prime}$ parameterized by $({\theta}_{2},{\theta}_{3})$. Initialize a
LoRA diffusion model parameterized by $l$.

3for  _i=0 to N-1_ do

4 $F^{p}_{i}\leftarrow f_{2D}(f_{mv}(c,R_{i},T_{i}))$

5 end for

6

7$V_{p}=f_{3D}(\mathrm{~{}Var}\\{P(F^{p}_{i},h)\\}_{i=0}^{N-1})$

8Cost volume $V_{p}$ parameterized by ${\theta}_{1}$.

9while  _not converged_ do

10 Ramdomly sample a camera pose $o$. Sample $M$ query points $x_{j}$ along
the view ray based on camera pose $o$.

11 for  _j=0 to M-1_ do

12 $s_{j}\leftarrow\,f_{g}(E(x_{j}),V_{P}(x_{j}))$

13 $c_{j}\leftarrow f_{t}^{\prime}(h_{\Omega}(x_{j}),x_{j})$

14 end for

15

16 $\hat{x}\leftarrow R(\\{s_{j}\\}_{j=0}^{M-1},\\{c_{j}\\}_{j=0}^{M-1})$

17
${\theta}_{1}\leftarrow{\theta}_{1}-\eta_{1}\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial{\theta}_{1}}]$

18
${\theta}_{2}\leftarrow{\theta}_{2}-\eta_{2}\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial{\theta}_{2}}]$

19
${\theta}_{3}\leftarrow{\theta}_{3}-\eta_{3}\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial{\theta}_{3}}]$

20 $l\leftarrow
l-\eta_{4}\nabla_{l}\mathrm{E}_{t,\epsilon}||\epsilon_{l}(\hat{x_{t}},t,c,o))-\epsilon||^{2}_{2}$

21 end while

22

23Mesh fine-tuning, we use DMTet to extract textured mesh from optimized 3D
representation parameterized by $({\theta}_{1},{\theta}_{2},{\theta}_{3})$ and
geometry MLP decoder $f_{g}$. The extracted DMTet parameterized by
${\theta}_{4}$. Initialize a LoRA diffusion model parameters $l^{\prime}$.

24while  _not converged_ do

25 Ramdomly sample a camera pose $o$. Render 2D image $\hat{x}$ at pose $o$.

26
${\theta}_{5}\leftarrow{\theta}_{5}-\eta_{5}\mathrm{E}_{t,\epsilon,o}[w(t)(\epsilon_{pretrain}(\hat{x_{t}},t,c)-\epsilon_{l^{\prime}}(\hat{x_{t}},t,c,o))\frac{\partial\hat{x}}{\partial{\theta}_{5}}]$

27 $l^{\prime}\leftarrow
l^{\prime}-\eta_{4}\nabla_{l^{\prime}}\mathrm{E}_{t,\epsilon}||\epsilon_{l^{\prime}}(\hat{x_{t}},t,c,o))-\epsilon||^{2}_{2}$

28 end while

29

return __

Algorithm 1 Priors Refinement Figure 11: Ablation on source views predicted by
different multi-view diffusion models. We compare our generated 3D assets
based on source views predicted by Zero123 and Zero123++. For a fair
comparison, the reference views are generated by MVDream driven by user-
provided texts. GeoDream adapt to source views predicted by various multi-view
diffusion models. For each row from up to down, the given prompts are: (1) _A
brightly colored mushroom growing on a log._ (2) _Mech robot with large
weapons on top with hexagonal bases._ (3) _A small kitten._ Figure 12:
Ablation on source views predicted by different multi-view diffusion models.
We compare our generated 3D assets based on source views predicted by Zero123
and Zero123++. For a fair comparison, the reference views are generated by
MVDream driven by user-provided texts. GeoDream adapt to source views
predicted by various multi-view diffusion models. For each row from up to
down, the given prompts are: (1) _3D render of a statue of an astronaut._ (2)
_A high quality photo of a dragon._ (3) _A cute rabbit in a stunning, detailed
Chinese coat._

## References

  * Armandpour et al. [2023] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou.  Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond.  _arXiv preprint arXiv:2304.04968_ , 2023.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16123–16133, 2022.
  * Chen et al. [2023a] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  _arXiv preprint arXiv:2303.13873_ , 2023a.
  * Chen et al. [2023b] Zilong Chen, Feng Wang, and Huaping Liu.  Text-to-3d using gaussian splatting.  _arXiv preprint arXiv:2309.16585_ , 2023b.
  * Deitke et al. [2023a] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  _arXiv preprint arXiv:2307.05663_ , 2023a.
  * Deitke et al. [2023b] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13142–13153, 2023b.
  * Deng et al. [2022] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong.  Gram: Generative radiance manifolds for 3d-aware image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10673–10683, 2022.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Guo et al. [2023] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang.  threestudio: A unified framework for 3d content generation.  https://github.com/threestudio-project/threestudio, 2023.
  * He et al. [2019] Fengxiang He, Tongliang Liu, and Dacheng Tao.  Control batch size and learning rate to generalize well: Theoretical and empirical evidence.  _Advances in neural information processing systems_ , 32, 2019.
  * Henderson and Ferrari [2020] Paul Henderson and Vittorio Ferrari.  Learning single-image 3d reconstruction by generative modelling of shape, pose and shading.  _International Journal of Computer Vision_ , 128(4):835–854, 2020.
  * Henderson et al. [2020] Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert.  Leveraging 2d data to learn textured 3d mesh generation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7498–7507, 2020.
  * Hong et al. [2023] Susung Hong, Donghoon Ahn, and Seungryong Kim.  Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation.  _arXiv preprint arXiv:2303.15413_ , 2023.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Jun and Nichol [2023] Heewoo Jun and Alex Nichol.  Shap-e: Generating conditional 3d implicit functions.  _arXiv preprint arXiv:2305.02463_ , 2023.
  * Karnewar et al. [2023] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra.  Holodiffusion: Training a 3d diffusion model using 2d images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18423–18433, 2023.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kynkäänniemi et al. [2022] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen.  The role of imagenet classes in fr$\backslash$’echet inception distance.  _arXiv preprint arXiv:2203.06026_ , 2022.
  * Li et al. [2023a] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu.  Diffusion-sdf: Text-to-shape via voxelized diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12642–12651, 2023a.
  * Li et al. [2023b] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan.  Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d.  _arXiv preprint arXiv:2310.02596_ , 2023b.
  * Li et al. [2019] Yuanzhi Li, Colin Wei, and Tengyu Ma.  Towards explaining the regularization effect of initial large learning rate in training neural networks.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lin et al. [2023] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 300–309, 2023.
  * Liu et al. [2023a] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  _arXiv preprint arXiv:2306.16928_ , 2023a.
  * Liu et al. [2023b] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9298–9309, 2023b.
  * Liu et al. [2023c] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  _arXiv preprint arXiv:2309.03453_ , 2023c.
  * Long et al. [2022] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang.  Sparseneus: Fast generalizable neural surface reconstruction from sparse views.  In _European Conference on Computer Vision_ , pages 210–227. Springer, 2022.
  * Long et al. [2023] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al.  Wonder3d: Single image to 3d using cross-domain diffusion.  _arXiv preprint arXiv:2310.15008_ , 2023.
  * Lorensen and Cline [1998] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_ , pages 347–353. 1998.
  * Ma et al. [2021] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker.  Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8446–8455, 2023.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Mo et al. [2023] Shentong Mo, Enze Xie, Ruihang Chu, Lewei Yao, Lanqing Hong, Matthias Nießner, and Zhenguo Li.  Dit-3d: Exploring plain diffusion transformers for 3d shape generation.  _arXiv preprint arXiv:2307.01831_ , 2023.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Nichol et al. [2022] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  _arXiv preprint arXiv:2212.08751_ , 2022.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Purushwalkam and Naik [2023] Senthil Purushwalkam and Nikhil Naik.  Conrad: Image constrained radiance fields for 3d generation from a single image.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Qian et al. [2023] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  _arXiv preprint arXiv:2306.17843_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.  _URL https://arxiv. org/abs/2204.06125_ , 7, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Shen et al. [2021] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  _Advances in Neural Information Processing Systems_ , 34:6087–6101, 2021.
  * Shi et al. [2023a] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model.  _arXiv preprint arXiv:2310.15110_ , 2023a.
  * Shi et al. [2023b] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  _arXiv preprint arXiv:2308.16512_ , 2023b.
  * Shue et al. [2023] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.  3d neural field generation using triplane diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20875–20886, 2023.
  * Sun et al. [2023] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu.  Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior.  _arXiv preprint arXiv:2310.16818_ , 2023.
  * Tang et al. [2023] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng.  Dreamgaussian: Generative gaussian splatting for efficient 3d content creation.  _arXiv preprint arXiv:2309.16653_ , 2023.
  * Wang et al. [2023a] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12619–12629, 2023a.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021.
  * Wang et al. [2023b] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4563–4573, 2023b.
  * Wang et al. [2023c] Yu Wang, Xuelin Qian, Jingyang Huo, Tiejun Huang, Bo Zhao, and Yanwei Fu.  Pushing the limits of 3d shape generation at scale, 2023c.
  * Wang et al. [2023d] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _arXiv preprint arXiv:2305.16213_ , 2023d.
  * Wikipedia [2023] Wikipedia.  Janus — wikipedia, the free encyclopedia, 2023.  [Online; accessed 17-November-2023].
  * Xu et al. [2023] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang.  Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360deg views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4479–4489, 2023.
  * Yang et al. [2023] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li.  Consistnet: Enforcing 3d consistency for multi-view images diffusion.  _arXiv preprint arXiv:2310.10343_ , 2023.
  * Yao et al. [2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 767–783, 2018.
  * Ye et al. [2023] Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang.  Consistent-1-to-3: Consistent image to 3d view synthesis via geometry-aware diffusion models.  _arXiv preprint arXiv:2310.03020_ , 2023.
  * Zhang et al. [2022] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu.  Nerfusion: Fusing radiance fields for large-scale scene reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5449–5458, 2022.
  * Zhou et al. [2023] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang.  Uni3d: Exploring unified 3d representation at scale.  _arXiv preprint arXiv:2310.06773_ , 2023.