# Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and
3D Diffusion Priors

Guocheng Qian1,2, Jinjie Mai1, Abdullah Hamdi3, Jian Ren2, Aliaksandr
Siarohin2, Bing Li1,
Hsin-Ying Lee2, Ivan Skorokhodov1, Peter Wonka1, Sergey Tulyakov2, Bernard
Ghanem1
1King Abdullah University of Science and Technology (KAUST), 2Snap Inc.
3Visual Geometry Group, University of Oxford
{guocheng.qian<EMAIL_ADDRESS>
###### Abstract

We present “Magic123”, a two-stage coarse-to-fine approach for high-quality,
textured 3D meshes generation from a single unposed image in the wild using
both 2D and 3D priors. In the first stage, we optimize a neural radiance field
to produce a coarse geometry. In the second stage, we adopt a memory-efficient
differentiable mesh representation to yield a high-resolution mesh with a
visually appealing texture. In both stages, the 3D content is learned through
reference view supervision and novel views guided by a combination of 2D and
3D diffusion priors. We introduce a single trade-off parameter between the 2D
and 3D priors to control exploration (more imaginative) and exploitation (more
precise) of the generated geometry. Additionally, we employ textual inversion
and monocular depth regularization to encourage consistent appearances across
views and to prevent degenerate solutions, respectively. Magic123 demonstrates
a significant improvement over previous image-to-3D techniques, as validated
through extensive experiments on synthetic benchmarks and diverse real-world
images. Our code, models, and generated 3D assets are available at
https://github.com/guochengqian/Magic123.

Figure 1: Magic123 for image-to-3D generation. Magic123 can reconstruct high-
fidelity 3D content with detailed 3D geometry and high rendering resolution
($1024\times 1024$) from a _single_ unposed image in the wild. Visit
https://guochengqian.github.io/project/magic123 for an immersive
visualization.

##  1 Introduction

Despite observing the world in 2D, human beings have a remarkable capability
to navigate, reason, and engage with their 3D surroundings. This points
towards a deep-seated cognitive understanding of the characteristics and
behaviors of the 3D world - a truly impressive facet of human nature. This
ability is taken to another level by artists who can produce detailed 3D
replicas from a single image. Contrarily, from the perspective of computer
vision, the task of 3D reconstruction from an unposed image - which
encompasses the creation of geometry and textures - remains an unresolved,
ill-posed problem, despite decades of exploration and development [89, 59, 69,
25].

The recent advances in deep learning [23, 37, 55, 71] have allowed an
increasing number of 3D generation tasks to become learning-based. Even though
deep learning has accomplished significant strides in image recognition [27,
15] and generation [23, 37, 71], the particular task of single-image 3D
reconstruction in the wild is still lagging. We attribute this considerable
discrepancy in 3D reconstruction abilities between humans and machines to two
primary factors: (i) a deficiency in large-scale 3D datasets that impedes
large-scale learning of 3D geometry, and (ii) the trade-off between the level
of detail and computational resources when working on 3D data.

One possible approach to tackle the problem is to employ 2D priors. The pool
of realistic 2D image data available online is voluminous. LAION [75], one of
the most extensive text-image pair datasets, aids in training modern image
understanding and generation models like CLIP [63] and Stable Diffusion [71].
With the increasing generalization capabilities of 2D generation models, there
has been a notable rise in approaches that use 2D models as priors for
generating 3D content. DreamFusion [62] serves as a trailblazer for this 2D
prior-based methodology for text-to-3D generation. The technique demonstrates
an exceptional capacity to guide novel views and optimize a neural radiance
field (NeRF) [55] in a zero-shot setting. Drawing upon DreamFusion, recent
work such as RealFusion [51] and NeuralLift [94], have endeavored to adapt
these 2D priors for single image 3D reconstructions.

Figure 2:  Trade-off between 2D and 3D priors in Magic123. We compare single
image reconstructions for three cases: a teddy bear (common object), two
stacked donuts (less common object), and a dragon statue (uncommon object). As
shown on the right, Magic123 with only a 2D prior favors geometry exploration,
generating 3D content with more imagination while potentially lacking 3D
consistency. Magic123 with only 3D prior (on the left) prioritizes geometry
exploitation, resulting in precise yet potentially simplified geometry with
reduced details. Magic123 thus proposes to use both 2D and 3D prior and
introduces a trade-off parameter $\lambda_{2D/3D}$ to control the geometry
exploration and exploitation (see Fig. 8). We provide a balanced point
$\lambda_{2D/3D}$=1, with which Magic123 consistently offers identity-
preserving 3D content with fine-grained geometry and visually appealing
texture.

Another approach is to employ 3D priors. Earlier attempts at 3D reconstruction
leveraged 3D priors like topology constraints to assist in 3D generation [89,
53, 59, 24]. However, these manually-crafted 3D priors fall short of
generating high-quality 3D content. Recently, approaches like Zero-1-to-3 [46]
and 3Dim [92] adapted a 2D diffusion model [71] to become view-dependent and
utilized this view-dependent diffusion as a 3D prior.

We analyzed the behavior of both 2D and 3D priors and found that they both
have advantages and disadvantages. 2D priors exhibit impressive generalization
for 3D generation that is unattainable with 3D priors (_e.g._ , the dragon
statue example in Fig.2). However, methods relying on 2D priors alone
inevitably compromise on 3D fidelity and consistency due to their restricted
3D knowledge. This leads to unrealistic geometry like multiple faces (Janus
problems), mismatched sizes, inconsistent texture, and so on. An instance of a
failure case can be observed in the teddy bear example in Fig.2. On the other
hand, a strict reliance on 3D priors alone is unsuitable for in-the-wild
reconstruction due to the limited 3D training data. Consequently, as
illustrated in Fig.2, while 3D prior-based solution effectively processes
common objects (for instance, the teddy bear example in the top row), it
struggles with less common ones, yielding oversimplified, sometimes even flat
3D geometries (_e.g._ , dragon statue at bottom left).

In this paper, rather than solely relying on a 2D or 3D prior, we advocate for
the simultaneous use of both priors to guide novel views in image-to-3D
generation. By modulating the simple yet effective tradeoff parameter between
the potency of the 2D and 3D priors, we can manage the balance between
exploration and exploitation in the generated 3D geometry. Prioritizing the 2D
prior can enhance imaginative 3D capabilities to compensate for the incomplete
3D information inherent in a single 2D image, but this may result in less
accurate 3D geometry due to a lack of 3D knowledge. In contrast, prioritizing
the 3D prior can lead to more 3D-constrained solutions, generating more
accurate 3D geometry, albeit with reduced imaginative capabilities and
diminished ability to discover plausible solutions for challenging and
uncommon cases. We introduce Magic123, a novel image-to-3D pipeline that
yields high-quality 3D outputs through a two-stage coarse-to-fine optimization
process utilizing both 2D and 3D priors. In the coarse stage, we optimize a
neural radiance field (NeRF) [55]. NeRF learns an implicit volume
representation, which is highly effective for complex geometry learning.
However, NeRF demands significant memory, resulting in low-resolution rendered
images passed to the diffusion models, making the output for the image-to-3D
task low-quality. Even the more resource-efficient NeRF alternative, Instant-
NGP [57], can only reach a resolution of $128\times 128$ in the image-to-3D
pipeline on a 16GB memory GPU. Hence, to improve the quality of the 3D
content, we introduce a second stage, employing a memory-efficient and
texture-decomposed SDF-Mesh hybrid representation known as Deep Marching
Tetrahedra (DMTet) [78]. This approach enables us to increase the resolution
up to 1K and refine the geometry and texture of the NeRF separately. In both
stages, we leverage a combination of 2D and 3D priors to guide the novel
views.

We summarize our contributions as follows:

  * •

We introduce Magic123, a novel image-to-3D pipeline that uses a two-stage
coarse-to-fine optimization process to produce _high-quality high-resolution_
3D geometry and textures.

  * •

We propose to use 2D and 3D priors simultaneously to generate faithful 3D
content from any given image. The strength parameter of priors allows for the
trade-off between geometry exploration and exploitation. Users therefore can
play with this trade-off parameter to generate desired 3D content.

  * •

Moreover, we find a balanced trade-off between 2D and 3D priors, leading to
reasonably realistic and detailed 3D reconstructions. Using the exact _same_
set of parameters for all examples without any additional reconfiguration,
Magic123 achieves state-of-the-art results in 3D reconstruction from single
unposed images in both real-world and synthetic scenarios.

##  2 Methodology

Figure 3: The pipeline of Magic123. Magic123 is a two-stage coarse-to-fine
framework for high-quality 3D generation from a reference image. Magic123 is
guided by the reference image, constrained by the monocular depth estimation
from the image, and driven by a joint 2D and 3D diffusion prior to dream up
novel views. At the coarse stage, we optimize an Instant-NGP neural radiance
field (NeRF) to reconstruct a coarse geometry. At the fine stage, we
initialize a DMTet mesh from the NeRF output and optimize a high-resolution
mesh and texture. Textural inversion is used in both stages to generate
object-preserving geometry and view-consistent textures.

We propose a two-stage framework, Magic123, that generates 3D content from a
single reference image in a coarse to fine fashion, as shown in Fig. 3. In the
coarse stage, Magic123 learns a coarse geometry and texture by optimizing a
NeRF. In the fine stage, Magic123 improves the quality of 3D content by
directly optimizing a memory-efficient differentiable mesh representation with
high-resolution renderings. In both stages, Magic123 uses joint 2D and 3D
diffusion priors to trade off geometry exploration and geometry exploitation,
yielding reliable 3D content with high generalizability.

###  2.1 Magic123 pipeline

Image preprocessing. Magic123 is a pipeline for object-level image-to-3D
generation. Given an image with a background, Magic123 requires a
preprocessing step to extract the foreground object. We leverage an off-the-
shelf segmentation model, Dense Prediction Transformer [67], to segment the
object. The extracted mask, denoted as $\mathbf{M}$ is a binary segmentation
mask and will be used in the optimization. To prevent flat geometry collapse,
the model generates textures that only appear on the surface without capturing
the actual geometric details, we further extract the depth map from the
reference view by the pretrained MiDaS [68]. The foreground image is used as
the input, while the mask and the depth map are used in the optimization as
regularization priors. These reference images are assigned fixed camera poses,
assumed to the front view. More details in camera settings can be found in
Sec.3.2.

####  2.1.1 Coarse stage

The coarse stage of our Magic123 is aimed at learning underlying geometry that
respects the reference image. Due to its strong ability in handling complex
topological changes in a smooth and continuous fashion, we adopt NeRF in this
stage.

Instant-NGP and its optimization. We leverage Instant-NGP [57] as our NeRF
implementation because of its fast inference and ability to recover complex
geometry. To reconstruct 3D faithfully from a single image, the optimization
of NeRF requires at least two loss functions: (i) the reference view
reconstruction supervision; and (ii) the novel view guidance.

Reference view reconstruction loss $\mathcal{L}_{rec}$ is imposed in our
pipeline as one of the major loss functions to ensure the rendered image from
the reference viewpoint ($\mathbf{v}^{r}$, assumed to be front view) is as
close to the reference image $\mathbf{I}^{r}$ as possible. We adopt the mean
squared error (MSE) loss on both the reference image and its mask as follows:

$\mathcal{L}_{rec}=\lambda_{rgb}\|\mathbf{M}\odot(\mathbf{I}^{r}-G_{\theta}(\mathbf{v}^{r}))\|_{2}^{2}+\lambda_{mask}\|\mathbf{M}-M(G_{\theta}(\mathbf{v}^{r})))\|_{2}^{2},$
(1)

where $\theta$ is the NeRF parameters to be optimized, $\odot$ is Hadamard
product, $G_{\theta}(\mathbf{v}^{r})$ is NeRF rendered view from
$\mathbf{v}^{r}$ viewpoint, $M()$ is the foreground mask acquired by
integrating the volume density along the ray of each pixel. Since the
foreground object is extracted as input, we do not model any background and
simply use pure white for the background rendering for all experiments.
$\lambda_{rgb},\lambda_{mask}$ are the weights for the foreground RGB and the
mask.

Novel view guidance $\mathcal{L}_{g}$ is necessary since multiple views are
required to train a NeRF. We follow the pioneering work in text/image-to-3D
[62, 94] and use diffusion priors to guide the novel view generation. As a
significant difference from previous works, we do not rely solely on a 2D
prior or a 3D prior, but we use both of them to guide the optimization of the
NeRF. See §2.2 for details.

Depth prior $\mathcal{L}_{d}$ is exploited to avoid overly-flat or caved-in 3D
content. Using only the appearance reconstruction losses might yield poor
geometry due to the inherent ambiguity of reconstructing 3D content from 2D
images: the content of 3D may lie at any distance and still be rendered as the
same image. This ambiguity might result in flat or curved-in geometry as noted
in previous works [94]. We alleviate this issue by leveraging a depth
regularization. A pretrained monocular depth estimator [68] is leveraged to
acquire the pseudo depth $d^{r}$ on the reference image. The depth output $d$
from the NeRF model from the reference viewpoint should be close to the depth
prior. However, due to the value mismatch of two different sources of depth
estimation, an MSE loss is not an ideal loss function. We use the normalized
negative Pearson correlation as the depth regularization:

$\mathcal{L}_{d}=\frac{1}{2}\left[1-\frac{\text{cov}(\mathbf{M}\odot
d^{r},\mathbf{M}\odot d)}{\sigma(\mathbf{M}\odot d^{r})\sigma(\mathbf{M}\odot
d)}\right],$ (2)

where $\text{cov}(\cdot)$ denotes covariance and $\sigma(\cdot)$ measures
standard deviation.

Normal smoothness $\mathcal{L}_{n}$. One of the NeRF limitations is the
tendency to produce high-frequency artifacts on the surface of the object. To
this end, we enforce the smoothness of the normal maps of geometry for the
generated 3D model following [51]. We use the finite differences of the depth
to estimate the normal vector of each point, render a 2D normal map
$\mathbf{n}$ from the normal vector, and impose a loss as follows:

$\mathcal{L}_{n}=\|\mathbf{n}-\tau(g(\mathbf{n},k))\|,$ (3)

where $\tau(\cdot)$ denotes the stopgradient operation and $g(\cdot)$ is a
Gaussian blur. The kernel size of the blurring $k$ is set to $9\times 9$.

Overall, the coarse stage is optimized by a combination of losses:

$\mathcal{L}_{c}=\mathcal{L}_{rec}+\mathcal{L}_{g}+\lambda_{d}\mathcal{L}_{d}+\lambda_{n}\mathcal{L}_{n},$
(4)

where $\lambda_{d},\lambda_{n}$ are the weights of depth and normal
regularizations.

####  2.1.2 Fine stage

The coarse stage offers a low-resolution 3D model, possibly with noise due to
the tendency of NeRF to create high-frequency artifacts. Our fine stage aims
to refine the 3D model and obtain a high-resolution and disentangled geometry
and texture. To this end, we adopt DMTet [78], which is a hybrid SDF-Mesh
representation and is capable of generating high-resolution 3D shapes due to
its high memory efficiency. Note the fine stage is identical to the coarse
stage except for the 3D representation and rendering.

DMTet represents the 3D shape in terms of a deformable tetrahedral grid
$(V_{T},T)$, where $T$ denotes the tetrahedral grid and $V_{T}$ are its
vertexes. Given a vertex $v_{i}\in V_{T}$, a Signed Distance Function (SDF)
$s_{i}\in\mathbb{R}$ and a triangle deformation vector $\triangle
v_{i}\in\mathbb{R}^{3}$ are the parameters to be learned during optimization
to extract a differentiable mesh [78]. The SDF is initialized by converting
the density field of the coarse stage, while the triangle deformation is
initialized as zero. For the textures, we follow Magic3D [43] to use a neural
color field that is initialized from the color field of the coarse stage.
Since differentiable rasterization can be performed efficiently at very high
resolution, we always use $8\times$ resolution of the coarse stage, which is
found to have a similar memory consumption to the coarse stage.

###  2.2 Joint 2D and 3D priors for image-to-3D generation

2D priors. Using a single reference image is insufficient to train a complete
NeRF model without any priors [100, 45]. To address this issue, DreamFusion
[62] proposes to use a 2D diffusion model as the prior to guide the novel
views via the proposed score distillation sampling (SDS) loss. SDS exploits a
2D text-to-image diffusion model [72], encodes the rendered view as latent,
adds noise to it, and guesses the clean novel view guided by the input text
prompt. Roughly speaking, SDS translates the rendered view into an image that
respects both the content from the rendered view and the prompt. The SDS loss
is illustrated in the upper part of Fig. 4 and is formulated as:

$\displaystyle\mathcal{L}_{2D}=\mathbb{E}_{t,\mathbf{\epsilon}}\left[w(t)(\mathbf{\epsilon}_{\phi}(\mathbf{z}_{t};\mathbf{e},t)-\mathbf{\epsilon})\frac{\partial\mathbf{z}}{\partial\mathbf{I}}\frac{\partial\mathbf{I}}{\partial\theta}\right],$
(5)

where $\mathbf{I}$ is a rendered view, and $\mathbf{z}_{t}$ is the noisy
latent by adding a random Gaussian noise of a time step $t$ to the latent of
$\mathbf{I}$. $\epsilon,\epsilon_{\phi}$, $\phi$, $\theta$ are the added
noise, predicted noise, parameters of the 2D diffusion prior, and the
parameters of the 3D model. $\theta$ can be MLPs of NeRF for the coarse stage,
or SDF, triangular deformations, and color field for the fine stage.
DreamFusion [62] further points out that the Jacobian term of the image
encoder $\frac{\partial\mathbf{z}}{\partial\mathbf{I}}$ in Eq. (5) can be
further eliminated, making the SDS loss much more efficient in terms of both
speed and memory. In our experiments, we utilize the SDS loss with Stable
Diffusion [71] v1.5 as our 2D prior. The rendered images are interpolated to
$512\times 512$ as required by the image encoder in [71].

Textural inversion. Note the prompt $\mathbf{e}$ we use for each reference
image is not a pure text chosen from tedious prompt engineering. Using pure
text for image-to-3D generation most likely results in inconsistent texture
and geometry due to the limited expressiveness of the human language. For
example, using “A high-resolution DSLR image of a colorful teapot” will
generate different geometry and colors that do not respect the reference
image. We thus follow RealFusion [51] to leverage the same textual inversion
[20] technique to acquire a special token <e> to represent the object in the
reference image. We use the same prompt for all examples: “A high-resolution
DSLR image of <e>”. We find that Stable Diffusion can generate the teapot with
a more similar texture and style to the reference image with the textural
inversion technique compared to the results without it.

Overall, the 2D diffusion priors [62, 52, 43] exhibit a remarkable capacity
for exploring the space of geometry, thereby facilitating the generation of
diverse geometric representations with a heightened sense of imagination. This
exceptional imaginative capability compensates for the inherent limitations
associated with the availability of incomplete 3D information in a single 2D
image. Moreover, the utilization of 2D prior-based techniques for 3D
reconstruction reduces the likelihood of overfitting in certain scenarios,
owing to their training on an extensive dataset comprising over a billion
images. However, it is crucial to acknowledge that the reliance on 2D priors
may introduce inaccuracies in the generated 3D representations, thereby
potentially deviating from true fidelity. This low-fidelity generation happens
because 2D priors lack 3D knowledge. For instance, the utilization of 2D
priors may yield imprecise geometries, such as Janus problems and mismatched
sizes as depicted in Fig. 2 and Fig. 8.

Figure 4: 2D 3D Diffusion priors. Magic123 uses Stable Diffusion [71] as the
2D prior and viewpoint-conditioned diffusion model Zero-1-to-3 [46] as the 3D
prior. Stable Diffusion takes the noisy rendered view and a text prompt as
input, while Zero-1-to-3 uses additionally the novel view camera pose as
input, creating a 3D-aware prior for Magic123.

####  2.2.1 3D prior

Using only the 2D prior is not sufficient to capture detailed and consistent
3D geometry. Zero-1-to-3 [46] thus proposes a 3D prior solution. Zero-1-to-3
finetunes Stable Diffusion into a view-dependent version on Objaverse [14],
the largest open-source 3D dataset that consists of 818K models. Zero-1-to-3
takes a reference image and a viewpoint as input and can generate a novel view
from the given viewpoint. Zero-1-to-3 thereby can be used as a strong 3D prior
for 3D reconstruction. The usage of Zero-1-to-3 in an image-to-3D generation
pipeline using SDS loss [62] is formulated as:

$\displaystyle\mathcal{L}_{3D}=\mathbb{E}_{t,\mathbf{\epsilon}}\left[w(t)(\mathbf{\epsilon}_{\phi}(\mathbf{z}_{t};\mathbf{I}^{r},t,R,T)-\mathbf{\epsilon})\frac{\partial\mathbf{I}}{\partial\theta}\right],$
(6)

where $R,T$ are the camera poses passed to Zero-1-to-3, the view-dependent
diffusion model. The difference between using the 3D prior and the 2D prior is
illustrated in Fig. 4, where we show that the 2D prior uses text embedding as
guidance while the 3D prior uses the reference view $\mathbf{I}^{r}$ with the
novel view camera poses as guidance. The 3D prior utilizes camera poses to
encourage 3D consistency and enable the usage of more 3D information compared
to the 2D prior.

Overall, the utilization of 3D priors demonstrates a commendable capacity for
effectively harnessing the expansive realm of geometry, resulting in the
generation of significantly more accurate geometric representations compared
to their 2D counterparts. This heightened precision particularly applies when
dealing with objects that are commonly encountered within the pre-trained 3D
dataset. However, it is essential to acknowledge that the generalization
capability of 3D priors is comparatively lower than that of 2D priors, thereby
potentially leading to the production of geometric structures that may appear
implausible. This low generalization results from the limited scale of
available 3D datasets, especially in the case of high-quality real-scanned
objects. For instance, in the case of uncommon objects, the employment of
Zero-1-to-3 often tends to yield overly simplified geometries, flat surfaces
without details in the back view (see Fig. 2 and Fig. 8).

####  2.2.2 Joint 2D and 3D priors

We find that the 2D and 3D priors are complementary to each other. Instead of
relying solely on 2D or 3D prior, we propose to use both priors in 3D
generation. The 2D prior is used to explore the geometry space, favoring high
imagination but might lead to inaccurate geometry. We name this characteristic
of the 2D prior as geometry exploration. On the other hand, the 3D prior is
used to exploit the geometry space, constraining the generated 3D content to
fulfill the implicit requirement of the underlying geometry, favoring precise
geometry but with less generalizability. In the case of uncommon objects, the
3D prior might result in over-simplified geometry. We name this feature of
using the 3D prior as geometry exploitation. In our image-to-3D pipeline, we
propose a new prior loss for the novel view supervision to combine both 2D and
3D priors:

$\displaystyle\mathcal{L}_{g}=\mathbb{E}_{t_{1},t_{2},\mathbf{\epsilon}_{1},\mathbf{\epsilon}_{2}}\left[w(t)\left[\lambda_{2D/3D}(\mathbf{\epsilon}_{\phi_{2D}}(\mathbf{z}_{t_{1}};\mathbf{e},t_{1})-\mathbf{\epsilon}_{1})+\lambda_{3D}(\mathbf{\epsilon}_{\phi_{3D}}(\mathbf{z}_{t_{2}};\mathbf{I}^{r},t_{2},R,T)-\mathbf{\epsilon}_{2})\right]\frac{\partial\mathbf{I}}{\partial\theta}\right],$
(7)

where $\lambda_{2D/3D}$ and $\lambda_{3D}$ determine the strength of 2D and 3D
prior, respectively. Weighting more on $\lambda_{2D/3D}$ leads to more
geometry exploration, while weighting more on $\lambda_{3D}$ results in more
geometry exploitation. However, tuning two parameters at the same time is not
user-friendly. Interestingly, through both qualitative and quantitative
experiments, we find that Zero-1-to-3, the 3D prior we use, is much more
tolerant to $\lambda_{3D}$ than Stable Diffusion to $\lambda_{2D}$. When only
the 3D prior is used, $\lambda_{2D}=0$, Zero-1-to-3 generates consistent
results for $\lambda_{3D}$ ranging from 10 to 60. On the contrary, Stable
Diffusion is rather sensitive to $\lambda_{2D}$. When setting $\lambda_{3D}$
to $0$ and using the 2D prior only, the generated geometry varies a lot when
$\lambda_{2D}$ is changed from $1$ to $2$. This observation leads us to fix
$\lambda_{3D}=40$ and to rely on tuning the $\lambda_{2D}$ to trade off the
geometry exploration and exploitation. We set $\lambda_{2D/3D}=1.0$ for all
results throughout the paper, but this value can be tuned according to the
user’s preference. More details and discussions on the choice of 2D and 3D
priors weights are available in Sec.3.4.

##  3 Experiments

###  3.1 Datasets

NeRF4. We introduce a NeRF4 dataset that we collect from 4 scenarios, chair,
drums, ficus, and microphone, out of the 8 test examples from the synthetic
NeRF dataset [55]. These four scenarios cover complex objects (drums and
ficus), a hard case (the back view of the chair), and a simple case (the
microphone). The other four examples are removed since they are not subject to
the front view assumption, requiring further camera pose estimation or a
manual tuning of the camera pose, which is out of the scope of this work.

RealFusion15. We further use the dataset collected and released by RealFusion
[51], consisting of 15 natural images that include bananas, birds, cacti,
barbie cakes, cat statues, teapots, microphones, dragon statues, fishes,
cherries, and watercolor paintings .

Figure 5:  Qualitative comparisons on image-to-3D generation. We compare
Magic123 to recent methods (Point-E [58], ShapeE [34], 3DFuse [77], RealFusion
[51], and Zero-1-to-3 [46]) for generating 3D objects from a single unposed
image (the leftmost column). On top, we show results on the RealFusion15
dataset, and on the bottom, we show results on the NeRF4 dataset.

###  3.2 Implementation details

Optimizing the pipeline. We use _exactly the same_ set of hyperparameters for
all experiments and do not perform any per-object hyperparameter optimization.
Both coarse and fine stages are optimized using Adam with $0.001$ learning
rate and no weight decay for $5,000$ iterations.
$\lambda_{rgb},\lambda_{mask},\lambda_{d}$ are set to $5,0.5,0.001$ for both
stages. $\lambda_{2D}$ and $\lambda_{3D}$ are set to $1$ and $40$ for the
first stage and are lowered to $0.001$ and $0.01$ in the second stage for
refinement to alleviate oversaturated textures. We adopt the Stable Diffusion
[80] model of V1.5 as the 2D prior. The guidance scale of the 2D prior is set
to $100$ following [62]. For the 3D prior, Zero-1-to-3 [46] ($105,000$
iterations finetuned version) is leveraged. The guidance scale of Zero-1-to-3
is set to $5$ following [46]. The NeRF backbone is implemented by three layers
of multi-layer perceptrons with $64$ hidden dims. Regarding lighting and
shading, we keep nearly the same as [62]. The difference is we set the first
$3,000$ iterations in the first stage to normals’ shading to focus on learning
geometry. For other iterations as well as the fine stage, we use diffuse
shading with a probability $0.75$ and textureless shading with a probability
$0.25$. The rendering resolutions are set to $128\times 128$ and $1024\times
1024$ for the coarse and the fine stage, respectively.

Camera setting. Since the reference image is unposed, we assume its camera
parameters are as follows. First, the reference image is assumed to be shot
from the front view, polar angle $90\degree$, azimuth angle $0\degree$.
Second, the camera is placed $1.8$ meters from the coordinate origin, the
radial distance is $1.8$. Third, the field of view (FOV) of the camera is
40$\degree$. We highlight that the 3D reconstruction performance is not
sensitive to camera parameters, as long as they are reasonable, FOV between
$20$ and $60$, and radial distance between $1$ to $4$ meters. Note this camera
setting works for images subject to the front-view assumption. For images
taken deviating from the front view, a manual change of polar angle or a
camera estimation is required.

###  3.3 Results

Evaluation metrics. For a comprehensive evaluation, we adhere to the metrics
employed in prior studies [94, 51], namely PSNR, LPIPS [104], and CLIP-
similarity [63]. PSNR and LPIPS are gauged in the reference view to measure
reconstruction quality and perceptual similarity. CLIP-similarity calculates
an average CLIP distance between rendered image and the reference image to
measure 3D consistency through appearance similarity across novel views and
the reference view.

Quantitative and qualitative comparisons. We compare Magic123 against the
state-of-the-art PointE [58], Shap-E [34], 3DFuse [77], NeuralLift [94],
RealFusion [51] and Zero-1-to-3 [46] in both NeRF4 and RealFusion15 datasets.
For Zero-1-to-3, we adopt the implementation here [84], which yields better
performance than the original implementation. For other works, we use their
officially released code. As shown in Table 1, Magic123 achieves Top-1
performance across all the metrics in both datasets when compared to previous
approaches. It is worth noting that the PSNR and LPIPS results demonstrate
significant improvements over the baselines, highlighting the exceptional
reconstruction performance of Magic123. The improvement of CLIP-Similarity
reflects the great 3D coherency regards to the reference view. Qualitative
comparisons are available in Fig. 5. Magic123 achieves the best results in
terms of both geometry and texture. Note how Magic123 greatly outperforms the
3D-based zero-1-to-3 [46] especially in complex objects like the dragon statue
and the colorful teapot in the first two rows, while at the same time greatly
outperforming 2D-based RealFusion [51] in all examples. This performance
demonstrates the superiority of Magic123 over the state-of-the-art and its
ability to generate high-quality 3D content.

Table 1: Magic123 results. We show quantitative results in terms of CLIP-
Similarity$\uparrow$ / PSNR$\uparrow$ / LPIPS$\downarrow$. The results are
shown on the NeRF4 and Realfusion datasets, while bold reflects the best.

Dataset | Metrics\Methods | Point-E [58] | Shap-E [34] | 3DFuse [77] | NeuralLift [94] | RealFusion [51] | Zero-1-to-3 [46] |  Magic123 (Ours)
---|---|---|---|---|---|---|---|---
NeRF4 | CLIP-Similarity$\uparrow$ | 0.48 | 0.60 | 0.60 | 0.52 | 0.38 | 0.62 | 0.80
PSNR$\uparrow$ | 0.70 | 0.99 | 5.86 | 12.55 | 15.37 | 23.96 | 24.62
LPIPS$\downarrow$ | 0.80 | 0.76 | 0.76 | 0.50 | 0.20 | 0.05 | 0.03
RealFusion15 | CLIP-Similarity$\uparrow$ | 0.53 | 0.59 | 6.28 | 0.65 | 0.67 | 0.75 | 0.82
PSNR$\uparrow$ | 0.98 | 1.23 | 18.87 | 11.08 | 0.67 | 19.49 | 19.50
LPIPS$\downarrow$ | 0.78 | 0.74 | 0.80 | 0.53 | 0.14 | 0.11 | 0.10

###  3.4 Ablation and analysis

Magic123 introduces a coarse-to-fine pipeline for single image reconstruction
and a joint 2D and 3D prior for novel view guidance. We provide analysis and
ablation studies to show their effectiveness.

Figure 6: Ablation study (quantitative). We quantitatively compare using the
coarse and fine stages in Magic123. In both setups, we ablate utilizing only
2D prior ($\lambda_{2D}$=$1$,$\lambda_{3D}$=$0$), utilizing only 3D prior
($\lambda_{2D}$=$0$,$\lambda_{3D}$=$40$), and utilizing both 2D and 3D priors
($\lambda_{2D}$=$1$,$\lambda_{3D}$=$40$).

The effect of two stages. We study in Fig. 6 and Fig. 7 the effect of using
the fine stage of our pipeline on the performance of Magic123. We note that a
consistent improvement in terms of both qualitative and quantitative
performance is observed throughout different setups when the fine stage is
combined with the coarse stage. The use of a textured mesh DMTet
representation enables higher quality 3D content that fits the objective and
produces more compelling and higher resolution 3D consistent visuals.

3D priors only. We first turn off the guidance of the 2D prior by setting
$\lambda_{2D}=0$, such that we only use the 3D prior Zero-1-to-3 [46] as the
guidance. We study the effects of $\lambda_{3D}$ by setting it to
$10,20,40,60$. Interestingly, we find that Zero-1-to-3 is very robust to the
change of $\lambda_{3D}$. Tab. 2 demonstrates that different $\lambda_{3D}$
lead to a consistent quantitative result. We thus simply set $\lambda_{3D}=40$
throughout the experiments since it achieves a slightly better CLIP-similarity
score than other values.

Table 2: Effects of $\lambda_{3D}$ and $\lambda_{2D}$ in Magic123 using only
2D or 3D prior on NeRF4 dataset.

| varying $\lambda_{3D}$ when $\lambda_{2D}$=$0$ | varying $\lambda_{2D}$ when $\lambda_{3D}$=$0$
---|---|---
| 10 | 20 | 40 | 60 | 80 | 0.1 | 1 | 2
CLIP-similarity$\uparrow$ | 0.58 | 0.61 | 0.62 | 0.61 | 0.58 | 0.54 | 0.60 | 0.72
PSNR$\uparrow$ | 23.96 | 24.05 | 23.96 | 23.75 | 23.34 | 23.62 | 24.11 | 22.42
LPIPS$\downarrow$ | 0.04 | 0.04 | 0.05 | 0.06 | 0.08 | 0.04 | 0.04 | 0.07

2D priors only. We then turn off the 3D prior and study the effect of
$\lambda_{2D}$ in the image-to-3D task. As shown in Tab. 2, with the increase
of $\lambda_{2D}$, an increase in CLIP-similarity is observed. This is due to
the fact that a larger 2D prior weight leads to more imagination but
unfortunately might result in the Janus problem.

Combining both 2D and 3D priors and the trade off factor $\lambda_{2D/3D}$. In
Magic123, we propose to use both 2D and 3D priors. Fig. 6 demonstrates the
effectiveness of combining the 2D and 3D priors on the quantitative
performance of image-to-3D generation. In Fig. 8, we further analyze the
tradeoff hyperparameter $\lambda_{2D/3D}$ from Eq. (7). We start from
$\lambda_{2D/3D}$=$0$ to use only the 3D prior and gradually increase
$\lambda_{2D/3D}$ to $0.1,0.5,1.0,2,5$, and finally $\infty$ to use only the
2D prior with $\lambda_{2D}$=$1$ and $\lambda_{3D}$=$0$. The key observations
include: (1) Relying solely on the 3D prior results in precise geometry (as
observed in the teddy bear) but falters in generating complex and uncommon
objects, often rendering oversimplified geometry with minimal details (as seen
in the dragon statue); (2) Relying solely on the 2D prior significantly
improves performance in conjuring complex scenes like the dragon statue but
simultaneously triggers the Janus problem in simple examples such as the bear;
(3) As $\lambda_{2D/3D}$ escalates, the imaginative prowess of Magic123 is
enhanced and more details become evident, but there is a tendency to
compromise 3D consistency. We assign $\lambda_{2D/3D}$=$1$ as the default
value for all examples. However, this parameter could also be fine-tuned for
even better results on certain inputs.

Figure 7: Ablation study (qualitative). We qualitatively compare the novel
view renderings from the coarse and fine stages in Magic123. We ablate
utilizing only 2D prior ($\lambda_{2D}$=$1$,$\lambda_{3D}$=$0$), utilizing
only 3D prior ($\lambda_{2D}$=$0$,$\lambda_{3D}$=$40$), and utilizing both 2D
and 3D priors ($\lambda_{2D}$=$1$,$\lambda_{3D}$=$40$).  Figure 8: Setting
$\lambda_{2D/3D}$. We study the effects of $\lambda_{2D/3D}$ on Magic123.
Increasing $\lambda_{2D/3D}$ leads to a 3D geometry with higher imagination
and less precision and vice versa. $\lambda_{2D/3D}$=$1$ provides a good
balance and thus is used as the default.

##  4 Related work

Multi-view 3D reconstruction. Multi-view 3D reconstruction aims to recover the
3D structure of a scene from its 2D RGB images captured from different camera
positions [18, 1]. Classical approaches usually recover a scene’s geometry as
a point cloud using SIFT-based [49] point matching [73, 74]. More recent
methods enhance them by relying on neural networks for feature extraction
([96, 30, 97, 101]). The development of Neural Radiance Fields (NeRF) [55, 47]
has prompted a shift towards reconstructing 3D as volume radiance [83],
enabling the synthesis of photo-realistic novel views [86, 3, 4]. Subsequent
works have also explored the optimization of NeRF in few-shot ([32, 39, 16])
and one-shot ([100, 9]) settings. NeRF does not store any 3D geometry
explicitly (only the density field), and several works propose to use a signed
distance function to recover a scene’s surface [99, 90, 98, 91, 13], including
in the few-shot setting as well ([102, 103]).

In-domain single-view 3D reconstruction. 3D reconstruction from a single view
requires strong priors on the object geometry since even epipolar constraints
[26] cannot be imposed in such a setup. Direct supervision in the form of 3D
shapes or keypoints is a robust way to impose such constraints for a
particular domain, like human faces [5, 6], heads [42, 88], hands [61] or full
bodies [48, 50]. Such supervision requires expensive 3D annotations and manual
3D prior creation. Thus several works explore unsupervised learning of 3D
geometry from object-centric datasets ([35, 17, 38, 82, 22, 41, 79]). These
methods are typically structured as auto-encoders [93, 36, 44] or generators
[7, 81] with explicit 3D decomposition under the hood. Due to the lack of
large-scale 3D data, these methods are limited to simple shapes (chairs, cars)
and cannot generalize to more complex or uncommon objects (dragons, statues).

Zero-shot single-view 3D reconstruction. Foundational multi-modal networks
[63, 8, 71] have enabled various zero-shot 3D synthesis tasks. Earlier works
employed CLIP [63] guidance for 3D generation [31, 29, 56, 95] and
manipulation [60, 40, 21] from text prompts. Modern zero-shot text-to-image
generators [66, 71, 65, 72, 2, 19] allowed to improve these results by
providing stronger synthesis priors [62, 87, 52, 10, 12]. DreamFusion [62] is
a seminal work that proposed to distill an off-the-shelf diffusion model [72]
into a NeRF [55, 4] for a given text query. It sparked numerous follow-up
approaches for text-to-3D synthesis ([43, 11]) and image-to-3D reconstruction
([76, 51, 46, 85]). The latter is achieved via additional reconstruction
losses on the frontal camera position [46] and/or subject-driven diffusion
guidance [64, 43]. The developed methods improved the underlying 3D
representation [43, 11, 84] and 3D consistency of the supervision [46, 77];
explored task-specific priors [28, 33, 70] and additional controls [54].
Similar to the recent image-to-3D generators [46, 51], we also follow the
DreamFusion [62] pipeline, but focus on reconstructing a high-resolution,
textured 3D mesh using a joint 2D and 3D priors.

##  5 Conclusion and discussion

This work presents Magic123, a two-stage coarse-to-fine solution for
generating high-quality, textured 3D meshes from a _single_ unposed image. By
leveraging both 2D and 3D priors, our approach overcomes the limitations of
existing studies and achieves state-of-the-art results in image-to-3D
reconstruction. The trade-off parameter between the 2D and 3D priors allows
for control over the balance between exploration and exploitation of the
generated geometry. Our method outperforms previous techniques in terms of
both realism and level of detail, as demonstrated through extensive
experiments on real-world images and synthetic benchmarks. Our findings
contribute to narrowing the gap between human abilities in 3D reasoning and
those of machines, and pave the way for future advancements in single image 3D
reconstruction. The availability of our code, models, and generated 3D assets
will further facilitate research and applications in this field.

Limitation. One of the limitations is that we assume the reference image is
taken from the front view. This assumption leads to poor geometry when the
reference image does not conform to the front-view assumption, a photo of a
dish on the table taken from the up view. Our method will instead focus on
generating the bottom of the dish and table instead of the dish geometry
itself. This limitation can be alleviated by a manual reference camera pose
tuning or camera estimation. Another limitation of our work is the dependency
on the preprocessed segmentation [67] and the monocular depth estimation model
[68]. Any error on these modules will creep into the later stages and affect
the overall generation quality. Similar to previous work, Magic123 also tends
to generate over-saturated textures due to the usage of the SDS loss. The
over-saturation issue becomes more severe for the second stage because of the
higher resolution.

Acknowledgement. The authors would like to thank Xiaoyu Xiang for the
insightful discussion and Dai-Jie Wu for sharing Point-E and Shap-E results.
This work was supported by the KAUST Office of Sponsored Research through the
Visual Computing Center funding, as well as, the SDAIA-KAUST Center of
Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI). Part
of the support is also coming from KAUST Ibn Rushd Postdoc Fellowship program.

## References

  * [1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski.  Building rome in a day.  Communications of the ACM, 54(10):105–112, 2011.
  * [2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [3] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5855–5864, 2021.
  * [4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5470–5479, 2022.
  * [5] Volker Blanz and Thomas Vetter.  Face recognition based on fitting a 3d morphable model.  IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 25(9):1063–1074, 2003.
  * [6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5543–5552, 2016.
  * [7] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3981–3990, June 2022.
  * [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9650–9660, 2021.
  * [9] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16123–16133, 2022.
  * [10] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Text2tex: Text-driven texture synthesis via diffusion models.  arXiv preprint arXiv:2303.11396, 2023.
  * [11] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  arXiv preprint arXiv:2303.13873, 2023.
  * [12] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [13] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry.  Improving neural implicit surfaces geometry with patch warping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6260–6269, 2022.
  * [14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142–13153, 2023.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [16] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann.  Learning to render novel views from wide-baseline stereo pairs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [17] Shivam Duggal and Deepak Pathak.  Topologically-aware deformation fields for single-view 3d reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1536–1546, 2022.
  * [18] Olivier D Faugeras.  What can be seen in three dimensions with an uncalibrated stereo rig?  In Computer Vision—ECCV’92: Second European Conference on Computer Vision Santa Margherita Ligure, Italy, May 19–22, 1992 Proceedings 2, pages 563–578. Springer, 1992.
  * [19] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 89–106, 2022.
  * [20] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In International Conference on Learning Representations (ICLR), 2023\.
  * [21] William Gao, Noam Aigerman, Groueix Thibault, Vladimir Kim, and Rana Hanocka.  Textdeformer: Geometry manipulation using text guidance.  In ACM Transactions on Graphics (SIGGRAPH), 2023.
  * [22] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoint without keypoints.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16, pages 88–104. Springer, 2020.
  * [23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems (NIPS), 2014\.
  * [24] Abdullah Hamdi, Bernard Ghanem, and Matthias Nießner.  Sparf: Large-scale learning of 3d sparse radiance fields from few input images.  arxiv, 2022.
  * [25] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or.  Point2mesh: A self-prior for deformable meshes.  In ACM Transactions on Graphics (SIGGRAPH), 2020.
  * [26] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [28] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  arXiv preprint arXiv:2303.11989, 2023.
  * [29] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  arXiv preprint arXiv:2205.08535, 2022.
  * [30] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  Deepmvs: Learning multi-view stereopsis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2821–2830, 2018.
  * [31] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 867–876, 2022.
  * [32] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5885–5894, 2021.
  * [33] Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Farm3d: Learning articulated 3d animals by distilling 2d diffusion.  arXiv preprint arXiv:2304.10535, 2023.
  * [34] Heewoo Jun and Alex Nichol.  Shap-e: Generating conditional 3d implicit functions.  arXiv preprint arXiv:2305.02463, 2023.
  * [35] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 371–386, 2018.
  * [36] Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik.  Category-specific object reconstruction from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1966–1974, 2015.
  * [37] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401–4410, 2019.
  * [38] Ira Kemelmacher-Shlizerman.  Internet based morphable model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3256–3263, 2013.
  * [39] Mijeong Kim, Seonguk Seo, and Bohyung Han.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12912–12921, 2022.
  * [40] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing nerf for editing via feature field distillation.  In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  * [41] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani.  Canonical surface mapping via geometric cycle consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2202–2211, 2019.
  * [42] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [43] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [44] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey.  Sdf-srn: Learning signed distance 3d object reconstruction from static images.  Advances in Neural Information Processing Systems (NeurIPS), 33:11453–11464, 2020.
  * [45] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In WACV, 2023.
  * [46] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  arXiv preprint arXiv:2303.11328, 2023.
  * [47] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  arXiv preprint arXiv:1906.07751, 2019.
  * [48] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM Transactions on Graphics (TOG), 34(6):1–16, 2015.
  * [49] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International Journal of Computer Vision (IJCV), 60:91–110, 2004\.
  * [50] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little.  A simple yet effective baseline for 3d human pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2640–2649, 2017.
  * [51] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360$\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [52] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  arXiv preprint arXiv:2211.07600, 2022.
  * [53] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [54] Aryan Mikaeili, Or Perel, Daniel Cohen-Or, and Ali Mahdavi-Amiri.  Sked: Sketch-guided text-based 3d editing.  arXiv preprint arXiv:2303.10735, 2023.
  * [55] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 405–421. Springer, 2020.
  * [56] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In SIGGRAPH Asia 2022 Conference Papers, pages 1–8, 2022.
  * [57] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In ACM Transactions on Graphics (SIGGRAPH), 2022.
  * [58] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  arXiv preprint arXiv:2212.08751, 2022.
  * [59] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 165–174, 2019.
  * [60] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2085–2094, 2021.
  * [61] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019.
  * [62] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  International Conference on Learning Representations (ICLR), 2022\.
  * [63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In Proceedings of the International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021.
  * [64] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  arXiv preprint arXiv:2303.13508, 2023.
  * [65] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [66] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 8821–8831. PMLR, 2021.
  * [67] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12159–12168, 2021.
  * [68] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 44(3):1623–1637, 2020.
  * [69] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 704–720, 2018.
  * [70] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  arXiv preprint arXiv:2302.01721, 2023.
  * [71] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022.
  * [72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems (NeurIPS), 35:36479–36494, 2022.
  * [73] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [74] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [75] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [76] Hoigi Seo, Hayeon Kim, Gwanghyun Kim, and Se Young Chun.  Ditto-nerf: Diffusion-based iterative text to omni-directional 3d model.  arXiv preprint arXiv:2304.02827, 2023.
  * [77] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [78] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 6087–6101, 2021.
  * [79] Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Hsin-Ying Lee, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Unsupervised volumetric animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [80] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Proceedings of the International Conference on Machine Learning (ICML), pages 2256–2265. PMLR, 2015.
  * [81] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  ACM Transactions on Graphics (TOG), 41(6):1–10, 2022.
  * [82] Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, and Steven M Seitz.  Total moving face reconstruction.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pages 796–812. Springer, 2014.
  * [83] Andrea Tagliasacchi and Ben Mildenhall.  Volume rendering digest (for nerf).  arXiv preprint arXiv:2209.02417, 2022.
  * [84] Jiaxiang Tang.  Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022.  https://github.com/ashawkey/stable-dreamfusion.
  * [85] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  arXiv preprint arXiv:2303.14184, 2023.
  * [86] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE, 2022.
  * [87] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [88] Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, and Yebin Liu.  Faceverse: a fine-grained and detail-controllable 3d face morphable model from a hybrid dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20333–20342, 2022.
  * [89] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  * [90] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [91] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka.  Hf-neus: Improved surface reconstruction using high-frequency details.  Advances in Neural Information Processing Systems (NeurIPS), 35:1966–1978, 2022.
  * [92] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  In International Conference on Learning Representations (ICLR), 2023\.
  * [93] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [94] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang.  Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360$\\{$$\backslash$deg$\\}$ views.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [95] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  arXiv preprint arXiv:2212.14704, 2022.
  * [96] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018.
  * [97] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5525–5534, 2019.
  * [98] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems (NeurIPS), 34:4805–4815, 2021.
  * [99] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Advances in Neural Information Processing Systems (NeurIPS), 33:2492–2502, 2020.
  * [100] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4578–4587, 2021.
  * [101] Zehao Yu and Shenghua Gao.  Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1949–1958, 2020.
  * [102] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [103] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan.  Ners: neural reflectance surfaces for sparse-view 3d reconstruction in the wild.  Advances in Neural Information Processing Systems (NeurIPS), 34:29835–29847, 2021.
  * [104] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.