11institutetext: Tsinghua University
11email<EMAIL_ADDRESS>22institutetext: ShengShu

# V3D: Video Diffusion Models are Effective 3D Generators

Zilong Chen 1122 Yikai Wang† 11 Feng Wang 11 Zhengyi Wang 1122 Huaping Liu† 11

###### Abstract

Automatic 3D generation has recently attracted widespread attention. Recent
methods have greatly accelerated the generation speed, but usually produce
less-detailed objects due to limited model capacity or 3D data. Motivated by
recent advancements in video diffusion models, we introduce V3D, which
leverages the world simulation capacity of pre-trained video diffusion models
to facilitate 3D generation. To fully unleash the potential of video diffusion
to perceive the 3D world, we further introduce geometrical consistency prior
and extend the video diffusion model to a multi-view consistent 3D generator.
Benefiting from this, the state-of-the-art video diffusion model could be
fine-tuned to generate $360\degree$ orbit frames surrounding an object given a
single image. With our tailored reconstruction pipelines, we can generate
high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method
can be extended to scene-level novel view synthesis, achieving precise control
over the camera path with sparse input views. Extensive experiments
demonstrate the superior performance of the proposed approach, especially in
terms of generation quality and multi-view consistency. Our code is available
at https://github.com/heheyas/V3D.

###### Keywords:

Video Diffusion Models Single Image to 3D Novel View Synthesis

Figure 1: Delicate 3D assets generated by V3D. Our approach can generate high-
fidelity 3D objects within 3 minutes. ††footnotetext: †Corresponding Authors.

##  1 Introduction

With the advancement of diffusion-based image generation, automatic 3D
generation has also made remarkable progress recently. Approaches based on
Score Distillation Sampling [67] have achieved tremendous success in the field
of 3D generation by extracting priors directly from text-to-image diffusion
models [99, 44, 14, 58, 18]. However, such optimization-based methods are
relatively slow and suffer from mode collapse [93, 99] and the Janus problem
[1, 78].

Subsequent efforts have proposed various alternative pathways to address these
issues, mainly including directly mapping images into 3D representations [117,
85, 30, 40, 32, 63] (such as Triplane NeRF [8] or 3DGS [35]), or first
generating consistent multi-view images then conducting 3D reconstruction to
obtain the underlying 3D model [52, 50, 48]. With the increase in 3D data size
and model capacity, these methods have reduced the generation time from hours
to minutes. However, the training process of these models still exhibits
certain limitations; Methods that directly map to 3D representations require
training on 3D datasets, making it impractical to leverage higher-quality
image data and large-scale pre-trained image diffusion models. Meanwhile,
approaches based on multi-view generation face difficulties in directly
generating a sufficient number of images for reconstruction due to the
requirement of memory-consuming cross-attention or intermediate 3D structure
to ensure consistency across views.

Recently, video diffusion models have attracted significant attention due to
their remarkable ability to generate intricate scenes and complex dynamics
with great spatio-temporal consistency [4, 5, 7, 25, 13, 109, 2, 24]. Many
videos incorporate changes in perspectives which provide natural observations
of 3D objects from different views [43], enabling the diffusion models to
perceive the physical 3D world [7]. Current multi-view generation methods are
based on image diffusion models, which can only generate a few views (less
than 6). In contrast, video diffusion models can generate hundreds of frames,
which is adequate for downstream 3D tasks. These advancements give us the
possibility to generate consistent multi-view images using video diffusion
models and then reconstruct underlying 3D assets to achieve high-quality 3D
generation. However, reconstruction from multi-views imposes significant
demands on consistency which is difficult for current video diffusion models.
In this paper, we aim to fully unleash the potential of video diffusion models
to perceive the 3D world and inject the geometrical consistency prior to make
the diffusion model an effective multi-view generator. In light of this, we
propose V3D, a novel approach for 3D generation with video diffusion models.
Specifically, we propose to fine-tune video diffusion models on 3D datasets
with additional conditions to generate novel views in both object-centric and
scene-level scenarios. For object-centric 3D generation, we fine-tune the base
video diffusion model on $360\degree$ orbit videos of synthetic 3D objects
with the front view as a condition. We find that our fine-tuned model can
generate reliable multi-views for 3D reconstruction. However, there still
exists inconsistency in some details which may diminish the reconstruction
quality. To address this, we adopt perceptual loss instead of pixel-wise loss
as the reconstruction objective. We employ Gaussian Splatting as the 3D
representation due to its reconstruction and rendering efficiency. To further
accelerate the reconstruction, we propose a space-carving initialization
method for Gaussians, which involves unprojecting multi-view object masks back
to 3D space to roughly position the Gaussians on the object’s surface,
eliminating the invalid optimizations on empty spaces. Considering the demands
of real-world applications, we also propose a novel mesh extraction method
that initially extracts the surface using SDF and subsequently refines the
appearance on generated views with image-level loss. Furthermore, we
demonstrate the effectiveness of our approach in scene-level 3D generation. To
achieve precise camera path control and accommodate multi-view input, we
integrate a PixelNeRF encoder to enhance the video diffusion model with robust
3D signals. Combined with auxiliary loss and fine-tuning on real-world posed
videos, we extend the video diffusion model to generate novel views on an
arbitrary camera path given sparse view input. Our contributions are:

  * •

We propose V3D, a framework that first employs a video diffusion model to
generate consistent multi-view frames, then reconstructs the underlying 3D
content. By fine-tuning on 3D datasets, we successfully boost the ability of
geometrical consistency of video diffusion models. Our framework is generally
applicable to both object and scene generation.

  * •

We design a reconstruction pipeline tailored for video diffusion outputs to
generate high-quality 3D Gaussians or textured meshes. With our designed
efficient initialization and refinement, V3D can reconstruct delicate 3D
assets within 3 minutes.

  * •

We conduct experiments to validate the effectiveness of V3D. Both object-
centric and scene-level experiments show that our method achieves state-of-
the-art performance in reconstruction quality and multi-view consistency.

##  2 Related Work

###  2.1 3D Generation

Early efforts in 3D generation focus on per-scene optimization methods based
on CLIP [70, 73, 31, 59, 17, 92, 36]. DreamFusion [67], the pioneer work,
leverages stronger diffusion prior by introducing Score Distillation Sampling
that minimizes the difference between rendered images from the underlying 3D
assets and the diffusion prior. Subsequent methods have further achieved
substantial improvements of this paradigm in quality [99, 14, 44, 114],
optimization speed [86, 58], alleviating the Janus problem [1, 78, 18, 80,
42], and have expanded their applications to generative tasks such as editing
[16, 29, 116, 65, 98], texture generation [108, 14, 58], and single image-
to-3D [95, 69, 87, 82]. Despite great success has been achieved, optimization-
based methods still suffer from slow generation speed and low success rates.
To overcome these challenges, researchers have explored some non-optimization
paradigms. One stream involves first generating consistent multi-view images,
and then utilizing reconstruction methods to obtain the corresponding 3D model
[48, 47, 89, 88]. SyncDreamer [50] combines explicit voxels and 3D convolution
with a diffusion model to enhance the consistency of generated multi-views.
Wonder3D [52] fine-tunes image diffusion models to generate sparse views with
corresponding normal maps and utilizes a NeuS [94] to reconstruct the
underlying geometry. Direct2.5 [55] adopts a remeshing-based technique to
recover geometry from generated sparse normal maps. Another line of research
involves directly mapping sparse-views into 3D representations [83, 51, 84,
104, 11, 105, 97, 103, 68, 81]. LRM [30], PF-LRM [40], and Instant3D [41]
adopt a huge transformer to directly predict triplanes from single or sparse
views. TriplaneGaussian [117] and LGM [85] instead map sparse views into more
memory-efficient 3D Gaussian Splatting which supports much higher resolution
for supervision. Our concurrent work IM-3D [56] also explores the capability
of video diffusion models in object-centric multi-view generation, we further
extend this paradigm in scene-level novel view synthesis and achieve better
performance.

###  2.2 Generative Models for Novel View Synthesis

While NeRF [60] and 3D Gaussian Splatting [35] have shown impressive
performance in novel view synthesis with a sufficient number of inputs,
reconstruction from sparse views requires additional priors due to the
incomplete information provided. Early work mainly focuses on using
regression-based [96, 106, 72, 12, 21, 28, 90, 91, 111] or GAN-based [8, 9,
22, 23, 27, 62, 64, 77, 113] methods for generalizable scene reconstruction.
However, due to the lack of high-quality data and the limited model capacity,
the generated novel views are often blurry and exhibit poor generalization
ability. Subsequent work further integrates diffusion priors to achieve better
scene-level novel view synthesis [100, 33]. SparseFusion [112] proposes view-
conditioned diffusion with epipolar feature transformer to synthesize novel
views with sparse inputs. GeNVS [10] utilizes a PixelNeRF encoder to
incorporate geometry information and exploits an auto-aggressive generation
strategy to enhance multi-view consistency. ZeroNVS [74] extends Zero-1-to-3
[49] to unbounded scenes and resolves the scale ambiguity with better camera
pose conditioning. ReconFusion [102] proposes to fine-tune image diffusion
models to condition on multi-view input and utilizes a novel sample loss to
reconstruct the 3D scene from sparse posed inputs. We draw inspiration from
ReconFusion and GeNVS, adopting PixelNeRF-based conditioner to accommodate any
number of input images and provide precise camera path control in scene-level
generation.

##  3 Approach

###  3.1 Overall Structure

Figure 2: Overview of the proposed V3D.

As demonstrated in Fig. 2, the core insight of V3D lies in conceptualizing
dense multi-view synthesis as video generation, thus leveraging the structure
and powerful priors of the large-scale pre-trained video diffusion model to
facilitate consistent multi-view generation. For object-centric image-to-3D,
we fine-tune the base video diffusion model on $360\degree$ orbit videos of
synthetic 3D objects rendered on fixed circular camera poses (Sec. 3.2) and
design a reconstruction and mesh extraction pipeline that is tailored for
generated multi-views (Sec. 3.3). For scene-level novel view synthesis, we
enhance the base video diffusion model by incorporating a PixelNeRF encoder,
enabling it to precisely control camera poses of generated frames and
seamlessly adapt to any number of input images (Sec. 3.4). The detailed V3D
methodology is presented as follows.

###  3.2 Dense View Prediction as Video Generation

Previous multi-view generation models [15, 101, 89] often extend from image
diffusion models, incorporating multi-view cross attention (e.g. Wonder3D [52]
and Zero123-XL [79]) or intermediate 3D representations (e.g. SyncDreamer [50]
with 3D voxels and Viewset Diffusion [84] and DMV3D [104] with triplane NeRF)
to enhance the consistency of generated images across different views. The
former tends to involve a limited number of views due to the use of cross-
attention with quadratic complexity, whereas the latter often results in low
image resolution due to memory-consuming 3D representations and volume
rendering. To overcome these drawbacks, V3D adopts a fundamentally distinct
approach to generate dense multi-view images given a single view. Inspired by
Stable Video Diffusion (SVD) [4], we interpret continuous multi-view images
rotating around an object as a video, thereby treating multi-view generation
as a form of image-to-video generation with the front view as the condition.
This approach capitalizes on the comprehensive understanding of the 3D world
offered by large-scale pre-trained video diffusion models to address the
scarcity of 3D data and harnesses the inherent network architecture of video
diffusion models to effectively produce an adequate number of multi-view
images.

####  3.2.1 Conditioning.

Similar to SVD, we provide high-level semantic information of the front view
by injecting the corresponding CLIP embedding into the diffusion U-Net via
cross-attention and prompt the diffusion model low-level information by
concatenating its latent along the channel dimension of the input. To better
adapt to image-to-3D, we removed irrelevant conditions including motion bucket
ID and FPS ID, and opted not to condition on elevation angle. Instead, we
randomly rotate the object in elevation to make the generation model
compatible with inputs with non-zero elevation. Our modification to the
conditioning arises from experimental evidence suggesting that it is not
feasible to perfectly constrain the elevation of generated multi-view images
to a specified angle. The misalignment between the camera poses of the video
and those used for reconstruction could lead to significant damage to the
final outcomes.

####  3.2.2 Data.

We fine-tune the base video diffusion model on the Objaverse dataset [19] for
object-centric image-to-3D. It is well-established that the Objaverse dataset
contains quite many low-quality 3D objects [52, 47] which may significantly
degenerate the model performance [49]. We, therefore, start by manually
curating a subset with higher quality, comprising approximately 290k synthetic
3D triangle meshes. We then construct a $360\degree$ orbit video dataset by
rendering synthetic 3D objects in this filtered subset on $N=18$ fixed camera
poses. Specifically, for each object, we normalize the mesh to unit length,
fix the camera distance to 2, set the elevation to 0, and uniformly move the
camera on the azimuth $N$ times (from 0 to $2\pi$) to generate a smooth orbit
video at 512 resolution.

####  3.2.3 Training.

We follow SVD to adopt the commonly used EDM [34] framework with the
simplified diffusion loss for fine-tuning. For the noise distribution used in
training, we inspired by [46, 79], adopt a relatively large noise distribution
with $P_{\text{mean}}=1.5$ and $P_{\text{std}}=2.0$. To enable classifier-free
guidance, we randomly set the latent and the CLIP embedding of the front view
to zero independently with a probability of 20%. To speed up training and save
GPU VRAM, we preprocess and store the latent and CLIP embedding of all video
frames beforehand. During training, we load these tensors directly from the
disk instead of computing them on the fly. We fine-tuned the base diffusion
model for 22.5k steps, taking about one day. More detailed training settings
are provided in the supplemental material.

###  3.3 Robust 3D Reconstruction and Mesh Extraction

After obtaining dense views of an object using the fine-tuned video diffusion
model, our next step is to reconstruct the underlying 3D object from these
views. While the obtained multi-view images are of high resolution and
satisfactory consistency, it’s challenging to ensure strict pixel-to-pixel
alignment between different views. In this case, directly applying pixel-wise
loss for 3D reconstruction often leads to artifacts such as floaters or blurry
textures [3, 66, 45]. In response, we propose using image-level perceptual
loss to resolve inconsistencies among multiple input images. To support image-
level loss, we opt for 3D Gaussian Splatting due to its fast training speed
and great memory efficiency that enables rendering the full image with minimal
cost. We adopt LPIPS [110] as the perceptual loss and combine it with the
original loss for 3DGS, i.e.

$\mathcal{L}_{\text{recon}}(I,I^{\text{gt}})=\text{MSE}(I,I^{\text{gt}})+\lambda_{s}\text{D-SSIM}(I,I^{\text{gt}})+\lambda_{l}\text{LPIPS}(I,I^{\text{gt}})$
(1)

where $I$ and $I^{\text{gt}}$ represent the rendered images and the ground
truth images (our generated views), $\lambda_{s}$ and $\lambda_{l}$ refer to
the loss weights.

####  3.3.1 Initialization.

Initialization is crucial for 3D Gaussian Splatting to achieve promising
results [35, 115]. Due to no sparse point cloud available in object-centric
reconstruction, the original 3DGS initializes the Gaussians with Gaussian-blob
and removes unnecessary Gaussians by resetting opacities. This requires many
optimization iterations to eliminate the floaters, and artifacts may occur
when the input images are not perfectly consistent. To accelerate the
reconstruction, we propose to initialize the Gaussians by space carving [38,
55]. Concretely, we first segment the foreground of the generated frames with
an off-the-shelf background removal model to obtain an object mask for each
view. An occupancy grid is then established by projecting voxels onto image
planes to determine whether the projected pixel belongs to the object and
aggregating the results of all projections under all views. Finally, we
exploit marching cubes [54] to obtain a surface according to this occupancy
grid and uniformly sample $N_{\text{init}}$ points on the surface as the
initializing positions of the Gaussians.

####  3.3.2 Mesh Extraction.

For the demand of real-world applications, we also propose a mesh extraction
pipeline for generated views. Similar to [52, 88, 50], we adopt NeuS [94] with
multi-resolution hash grid [61] for fast surface reconstruction. While V3D can
generate a considerable number of views, it remains modest compared to the
typical use cases of NeuS, which commonly involve more than 40 posed images
[94, 53]. Therefore we adopt a normal smooth loss and a sparse regularization
loss to ensure a reasonable geometry. Besides, since our generated images are
not perfectly consistent and it is not feasible for NeuS to exploit image-
level loss due to the high render cost, the textures of the mesh extracted by
NeuS are often blurry. To address this, we further refine the texture of the
NeuS extracted mesh with generated multi-views using LPIPS loss while keeping
the geometry unchanged, which greatly reduces the rendering cost and enhances
the quality of final outputs. Owing to the great efficiency of differentiable
mesh rendering [39], this refinement process can be completed within 15
seconds.

Figure 3: Qualitative comparison between the proposed V3D and state-of-the-art
3DGS-based image-to-3D methods (LGM [85] and TriplaneGaussian [117]) and
optimization-based methods (Magic123 [69] and ImageDream [95]). The
corresponding $360\degree$ videos are demonstrated in the supplemental video.

###  3.4 Scene-level Novel View Synthesis

In addition to object-centric generation, we further extend video diffusion to
scene-level novel view synthesis. Unlike generating views of objects, novel
view synthesis entails generating images along a given camera path, which
requires precise control over camera poses and accommodating multiple input
images.

####  3.4.1 Conditioning.

Previous approaches, such as Zero-1-to-3 [49], incorporate camera poses into
image diffusion by introducing camera embeddings. However, this method did not
explicitly utilize 3D information, thus making it challenging to ensure
consistency across multiple views and precise control over camera pose [48,
102]. To better incorporate 3D prior into video diffusion and precisely
control the poses of generated frames, we draw inspiration from GeNVS [10] and
ReconFusion [102], integrating a PixelNeRF [106] feature encoder into video
diffusion models. Specifically, given a set of posed images as condition
$\pi_{\text{cond}},I_{\text{cond}}=\\{(I_{i},\pi_{i})\\}_{i}$, we adopt a
PixelNeRF to render the feature map of the target camera poses $\\{\pi\\}$ by:

$f=\text{PixelNeRF}(\\{\pi\\}|\pi_{\text{cond}},I_{\text{cond}}),$ (2)

where $\\{\pi\\}$ refers to the camera poses of the frames we want to
generate. We then concatenate this feature map to the input of the U-Net to
explicitly encode relative camera pose information. This approach seamlessly
supports an arbitrary number of images as input. Other conditionings are
similar to those of object-centric generation, except that the number of CLIP
image embedding has been increased from one to multiple.

####  3.4.2 Data.

For scene-level novel view synthesis, we fine-tune the base video diffusion
model on MVImgNet [107] which consists of 219k posed real-world video clips.
To make the videos from MVImgNet compatible with the input size of the video
diffusion, we obtain the object masks with an off-the-shelf background removal
tool, then crop, recenter the image, and adjust the principal points and focal
lengths according to the foreground mask.

####  3.4.3 Training.

Except for the simplified diffusion loss used in video diffusion models, we
follow ReconFusion [102] to regulate the parameters of the PixelNeRF encoder
with a down-sampled photometric loss:

$\mathcal{L}_{\text{PixelNeRF}}=\mathbb{E}\lVert
f_{\text{RGB}}(\pi|x_{\text{cond}},\pi_{\text{cond}})-x_{\downarrow}\rVert$
(3)

where $f_{\text{RGB}}$ refers to another color branch of the PixelNeRF encoder
that predicts a downsampled image and $x_{\downarrow}$ refers to the
downsampled target frames. We experimentally found this loss crucial to avoid
local minima, as illustrated in [102]. We follow InstructPix2Pix [6],
initializing the newly added input channels for the PixelNeRF feature in the
first layer of the diffusion U-Net with zero to best keep the pre-trained
priors. For input views conditioning, we randomly select 1 to 4 images as
input and set the conditions to zeros independently with a probability of 20%.
We present more training details of the scene-level model in the supplemental
material.

##  4 Experiments

In this section, we present our experiments on validating the effectiveness of
the proposed approach in both object-centric and scene-level 3D generation.
Specifically, we quantitatively and qualitatively compare V3D with previous
state-of-the-art methods in image-to-3D and novel view synthesis.
Additionally, we conduct several ablation studies to evaluate the importance
of model designs, including noise distribution, number of fine-tuning steps,
pre-training prior, and camera conditioning. The detailed results are shown as
follows.

Figure 4: Qualitative comparison of generated views between the proposed V3D
and state-of-the-art multi-view generation methods, including SyncDreamer [50]
and Wonder3D [52]. The corresponding $360\degree$ videos are shown in the
supplemental video.

###  4.1 Object-centric 3D Generation

####  4.1.1 Qualitative Comparisons.

We evaluate the performance of the proposed V3D on image-to-3D and compare the
results with state-of-the-art methods. The upper part of Fig. 3 demonstrates
qualitative comparisons between our approach with 3DGS-based methods
TriplaneGaussian [117] and LGM [85]. Our approach demonstrates impressive
quality improvement, while TriplaneGaussian and LGM tend to generate a much
more blurry appearance due to the limited number of Gaussians they can
produce. The lower part of Fig. 3 presents a comparison between state-of-the-
art SDS-based methods, including Magic123 [69] and ImageDream [95]. V3D
outperforms in both front-view alignment and fidelity. In contrast, objects
generated by Magic123 exhibit inaccurate geometry and blurry backface, while
ImageDream tends to generate assets with over-saturated textures. Importantly,
our method achieves these results within 3 minutes, making significant
acceleration compared to optimization-based methods which require more than 30
minutes. Fig. 4 illustrates comparisons on multi-view generation with
SyncDreamer [50] and Wonder3D [52], where our approach consistently achieves
superior results with a higher resolution of 512 and much finer details.
SyncDreamer tends to generate multi-view images that are simple in geometry
and with over-saturated texture. We attribute this to the significant
difference in structure between SyncDreamer and the original Stable Diffusion,
thus failing to fully leverage the information from image pre-training. As for
Wonder3D, our model triples the number of generated multi-views and doubles
the resolution, achieving significant improvements in both texture quality and
multi-view consistency.

####  4.1.2 Quantitative Comparisons.

For qualitative comparisons, we conduct a user study to evaluate the generated
3D assets with human ratings. Specifically, we ask 58 volunteers to evaluate
objects generated by V3D and other methods under 30 condition images by
watching the rendered $360\degree$ spiral videos, and choose the best method
in (a) Alignment: how well does the 3D asset align with the condition image;
(b) Fidelity: how realistic is the generated object. Tab. 2 demonstrates the
win rate of each method in the two criteria. V3D is regarded as the most
convincing model of all, and it significantly outperforms other competitors in
both image alignment and fidelity.

Table 1: User study results. We collect 58 anonymous preference results on
image alignment and fidelity.

Approach | Alignment | Fidelity
---|---|---
SyncDreamer [50] | 10.34 | 5.17
Wonder3D [52] | 17.24 | 12.07
TriplaneGaussian [117] | 6.90 | 3.45
LGM [85] | 13.79 | 24.14
Ours | 51.72 | 55.17

Table 2: Quantitative comparison of dense multi-view synthesis on Google
Scanned Objects dataset [20].

Approach | PSNR$\shortuparrow$ | SSIM$\shortuparrow$ | LPIPS$\shortdownarrow$
---|---|---|---
RealFusion [57] | 15.26 | 0.722 | 0.283
Zero-1-to-3 [49] | 18.93 | 0.779 | 0.166
SyncDreamer [50] | 20.05 | 0.798 | 0.146
MVDiffusion++ [88] | 21.45 | 0.844 | 0.129
Ours | 22.08 | 0.882 | 0.102

###  4.2 Novel View Synthesis

For scene-level novel view synthesis, we test the performance of the proposed
V3D on the 10-category subset of the CO3D dataset. To align with the settings
of previous methods, we fine-tune V3D on videos in each category for only one
epoch (denoted as “fine-tuned”). The results are shown in Tab. 3. Our approach
achieves consistently better performance in terms of image metrics against
previous state-of-the-art novel view synthesizers by a clear margin,
demonstrating the effectiveness of utilizing pre-trained video diffusion
models in scene-level novel view synthesis. Besides, V3D shows impressive
generalization ability as the zero-shot version of V3D (only trained on
MVImgNet) beats most of the competitors.

####  4.2.1 Qualitative Comparisons.

We show in Fig. 5 qualitative comparisons on generated multi-views between
SparseFusion [112] on the hydrant subset of the CO3D dataset [71]. For better
comparison, we use COLMAP [75, 76] to perform multi-view stereo reconstruction
given camera poses and showcased, in Fig. 5, the number of points in the
obtained point cloud and the Chamfer distance between the one reconstructed
from ground truth images. The point clouds reconstructed from V3D generated
images contain more points and are closer to the point clouds reconstructed
from real images. This indicates a significant advantage of our method in
terms of both reconstruction quality and multi-view consistency.

Table 3: Quantitative comparison on 10-category subset of CO3D dataset [71].
We follow SparseFusion [112], benchmark view synthesis results with 2, 3, and
6 input views. The best result is highlighted in bold, while the second-best
result is underscored.

| 2 Views | 3 Views | 6 Views
---|---|---|---
|  PSNR$\shortuparrow$ |  SSIM $\shortuparrow$ |  LPIPS $\shortdownarrow$ |  PSNR $\shortuparrow$ |  SSIM $\shortuparrow$ |  LPIPS $\shortdownarrow$ |  PSNR $\shortuparrow$ |  SSIM $\shortuparrow$ |  LPIPS $\shortdownarrow$
PixelNeRF [106] | 19.52 | 0.667 | 0.327 | 20.67 | 0.712 | 0.293 | 22.47 | 0.776 | 0.241
NerFormer [71] | 17.88 | 0.598 | 0.382 | 18.54 | 0.618 | 0.367 | 19.99 | 0.661 | 0.332
ViewFormer [37] | 18.37 | 0.697 | 0.282 | 18.91 | 0.704 | 0.275 | 19.72 | 0.717 | 0.266
EFT [26, 112] | 20.85 | 0.680 | 0.289 | 22.71 | 0.747 | 0.262 | 24.57 | 0.804 | 0.210
VLDM [112] | 19.55 | 0.711 | 0.247 | 20.85 | 0.737 | 0.225 | 22.35 | 0.768 | 0.201
SparseFusion [112] | 21.34 | 0.752 | 0.225 | 22.35 | 0.766 | 0.216 | 23.74 | 0.791 | 0.200
Ours (w. camera emb.) | 18.27 | 0.598 | 0.303 | 18.83 | 0.605 | 0.273 | - | - | -
Ours (w. Plücker emb.) | 20.11 | 0.714 | 0.277 | 20.39 | 0.727 | 0.223 | - | - | -
Ours (w/o. pre-training) | 16.20 | 0.544 | 0.333 | 16.76 | 0.576 | 0.303 | - | - | -
Ours (zero-shot) | 20.64 | 0.734 | 0.213 | 22.04 | 0.805 | 0.141 | 23.94 | 0.851 | 0.126
Ours (fine-tuned) | 22.19 | 0.790 | 0.093 | 23.31 | 0.860 | 0.082 | 24.70 | 0.890 | 0.074

Figure 5: Qualitative comparison on scene-level novel view synthesis with
SparseFusion on hydrant subset of CO3D dataset. We reconstruct a sparse point
cloud using COLMAP and report the number of points and Chamfer distance
against the point cloud extracted with ground truth images.

###  4.3 Ablations

Figure 6: Ablations study. We show that the number of fine-tuning steps,
large-scale pre-training, and stronger noise distribution are crucial for
achieving promising results.

####  4.3.1 Pre-training.

To demonstrate the impact of large-scale video pre-training, we trained a
variant of the model from scratch on the constructed Objaverse $360\degree$
orbit video for 45k steps, which, apart from the initialized weights of the
diffusion U-Net, is identical to the full model. As shown in Fig. 6, this
from-scratched variant completely failed to generate multi-view images even
with doubled training steps compared to the full model, demonstrating the
significant importance of adopting large-scale video pre-training.

####  4.3.2 Number of Fine-tuning Steps.

To illustrate the impact of the Objaverse dataset on the pre-trained model, we
fine-tune the base model with different training steps, as shown in the top
part of Fig. 6. Due to the considerably lower complexity of renders from
Objaverse objects, excessive fine-tuning steps can yield overly simplistic
textures in the generated frames, while an inadequate number of fine-tuning
steps may lead to strange and inaccurate geometries.

####  4.3.3 Noise Distribution.

To validate the necessity of adopting a large noise distribution during
training, we fine-tune the base diffusion model with a smaller noise schedule
($P_{\text{mean}}=0.7$ and $P_{\text{std}}=1.6$), which is used in the image-
to-video pre-training in Stable Video Diffusion. As demonstrated in the lower
part of Fig. 6, the model trained with a smaller noise distribution tends to
generate geometric structures that are unreasonable (the pistol) or
degenerated (the cartoon-style astronaut). We attribute this phenomenon to the
strong image signal we exploited and the relatively simple nature of the
synthetic Objaverse dataset, indicating the need to shift the noise
distribution towards stronger noise.

####  4.3.4 Camera Conditioning.

To assess the impact of the approach of camera conditioning, we introduce two
variants that condition on camera poses by commonly used learnable camera
embedding or the Plücker ray embedding. For a fair comparison, for both
variants, we additionally concatenate the multi-view images used as conditions
into the input of the diffusion U-Net and similarly set the weights of added
channels to zero to minimize the gap. As shown in Tab. 3, either camera
embedding or Plücker embedding provides accurate 3D information and results in
a significant degeneration in model performance.

##  5 Limitations and Conclusion

####  5.0.1 Limitations.

Although achieves state-of-the-art performance in 3D generation, V3D would
produce unsatisfying results for some complex objects or scenes, such as
inconsistency among multiple views or unreasonable geometries. Concrete
failure cases and analyses are discussed in the supplemental material.

####  5.0.2 Conclusion.

In this paper, we propose V3D, a novel method for generating consistent multi-
view images with image-to-video diffusion models. By fine-tuning the base
video diffusion model on 3D datasets, we extend video diffusion models to
effective 3D generators. Specifically, for object-centric 3D generation, we
fine-tune the video diffusion on synthesizing $360\degree$ videos of 3D
objects to predict dense views given a single image. To obtain the underlying
3D asset from generated views, we propose a tailored reconstruction pipeline
with designed initialization and texture refinement, enabling the
reconstruction of high-quality 3D Gaussians or delicate textured meshes within
3 minutes. We further extend our framework to scene-level novel view
synthesis, achieving precise control over the camera path with great multi-
view consistency. We conduct extensive experiments to validate the
effectiveness of the proposed approach, illustrating its remarkable
performance in generating consistent multi-views and generalization ability.
We hope our method can serve as an efficient and powerful approach for high-
quality 3D generation and could pave the way for more extensive applications
of video diffusion models in 3D tasks.

## References

  * [1] Armandpour, M., Zheng, H., Sadeghian, A., Sadeghian, A., Zhou, M.: Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968 (2023)
  * [2] Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., Wang, O., Sun, D., Dekel, T., Mosseri, I.: Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv: 2401.12945 (2024)
  * [3] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855–5864 (2021)
  * [4] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D.: Stable video diffusion: Scaling latent video diffusion models to large datasets. ArXiv abs/2311.15127 (2023), https://api.semanticscholar.org/CorpusID:265312551
  * [5] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [6] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
  * [7] Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., Ramesh, A.: Video generation models as world simulators (2024), https://openai.com/research/video-generation-models-as-world-simulators
  * [8] Chan, E., Lin, C.Z., Chan, M., Nagano, K., Pan, B., Mello, S.D., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3d generative adversarial networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 16102–16112 (2021), https://api.semanticscholar.org/CorpusID:245144673
  * [9] Chan, E., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In: arXiv (2020)
  * [10] Chan, E.R., Nagano, K., Chan, M.A., Bergman, A.W., Park, J.J., Levy, A., Aittala, M., Mello, S.D., Karras, T., Wetzstein, G.: GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In: arXiv (2023)
  * [11] Charatan, D., Li, S., Tagliasacchi, A., Sitzmann, V.: pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv: 2312.12337 (2023)
  * [12] Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., Su, H.: Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 14104–14113 (2021), https://api.semanticscholar.org/CorpusID:232404617
  * [13] Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: Videocrafter2: Overcoming data limitations for high-quality video diffusion models (2024)
  * [14] Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2023)
  * [15] Chen, Y., Fang, J., Huang, Y., Yi, T., Zhang, X., Xie, L., Wang, X., Dai, W., Xiong, H., Tian, Q.: Cascade-zero123: One image to highly consistent 3d with self-prompted nearby views. ArXiv abs/2312.04424 (2023), https://api.semanticscholar.org/CorpusID:266051982
  * [16] Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L., Liu, H., Lin, G.: Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. CoRR abs/2311.14521 (2023). https://doi.org/10.48550/ARXIV.2311.14521, https://doi.org/10.48550/arXiv.2311.14521
  * [17] Chen, Y., Chen, R., Lei, J., Zhang, Y., Jia, K.: TANGO: text-driven photorealistic and robust 3d stylization via lighting decomposition. In: NeurIPS (2022), http://papers.nips.cc/paper_files/paper/2022/hash/c7b925e600ae4880f5c5d7557f70a72b-Abstract-Conference.html
  * [18] Chen, Z., Wang, F., Wang, Y., Liu, H.: Text-to-3d using gaussian splatting. In: CVPR (2024), https://arxiv.org/abs/2309.16585
  * [19] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051 (2022)
  * [20] Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R.M., Reymann, K., McHugh, T.B., Vanhoucke, V.: Google scanned objects: A high-quality dataset of 3d scanned household items. 2022 International Conference on Robotics and Automation (ICRA) pp. 2553–2560 (2022), https://api.semanticscholar.org/CorpusID:248392390
  * [21] Flynn, J., Neulander, I., Philbin, J., Snavely, N.: Deep stereo: Learning to predict new views from the world’s imagery. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5515–5524 (2015), https://api.semanticscholar.org/CorpusID:14517241
  * [22] Gadelha, M., Maji, S., Wang, R.: 3d shape induction from 2d views of multiple objects. 2017 International Conference on 3D Vision (3DV) pp. 402–411 (2016), https://api.semanticscholar.org/CorpusID:2516262
  * [23] Gao, J., Shen, T., Wang, Z., Chen, W., Yin, K., Li, D., Litany, O., Gojcic, Z., Fidler, S.: Get3d: A generative model of high quality 3d textured shapes learned from images. ArXiv abs/2209.11163 (2022), https://api.semanticscholar.org/CorpusID:252438648
  * [24] Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S.S., Shah, A., Yin, X., Parikh, D., Misra, I.: Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv: 2311.10709 (2023)
  * [25] Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning (2023)
  * [26] He, Y., Yan, R., Fragkiadaki, K., Yu, S.I.: Epipolar transformers. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 7776–7785 (2020), https://api.semanticscholar.org/CorpusID:218581558
  * [27] Henzler, P., Mitra, N.J., Ritschel, T.: Escaping plato’s cave: 3d shape from adversarial rendering. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 9983–9992 (2018), https://api.semanticscholar.org/CorpusID:207976631
  * [28] Henzler, P., Reizenstein, J., Labatut, P., Shapovalov, R., Ritschel, T., Vedaldi, A., Novotný, D.: Unsupervised learning of 3d object categories from videos in the wild. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4698–4707 (2021), https://api.semanticscholar.org/CorpusID:232417771
  * [29] Hertz, A., Aberman, K., Cohen-Or, D.: Delta denoising score. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 2328–2337. IEEE (2023). https://doi.org/10.1109/ICCV51070.2023.00221, https://doi.org/10.1109/ICCV51070.2023.00221
  * [30] Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: LRM: large reconstruction model for single image to 3d. CoRR abs/2311.04400 (2023). https://doi.org/10.48550/ARXIV.2311.04400, https://doi.org/10.48550/arXiv.2311.04400
  * [31] Jain, A., Mildenhall, B., Barron, J.T., Abbeel, P., Poole, B.: Zero-shot text-guided object generation with dream fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 857–866. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.00094, https://doi.org/10.1109/CVPR52688.2022.00094
  * [32] Jun, H., Nichol, A.: Shap-e: Generating conditional 3d implicit functions. CoRR abs/2305.02463 (2023). https://doi.org/10.48550/arXiv.2305.02463, https://doi.org/10.48550/arXiv.2305.02463
  * [33] Karnewar, A., Vedaldi, A., Novotny, D., Mitra, N.: Holodiffusion: Training a 3D diffusion model using 2D images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2023)
  * [34] Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion-based generative models. In: Proc. NeurIPS (2022)
  * [35] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023), https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [36] Khalid, N.M., Xie, T., Belilovsky, E., Tiberiu, P.: Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers (December 2022)
  * [37] Kulhánek, J., Derner, E., Sattler, T., Babuška, R.: Viewformer: Nerf-free neural rendering from few images using transformers. In: European Conference on Computer Vision (ECCV) (2022)
  * [38] Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. International Journal of Computer Vision 38, 199–218 (1999), https://api.semanticscholar.org/CorpusID:1538200
  * [39] Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics 39(6) (2020)
  * [40] Li, J., Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong, Y., Sunkavalli, K., Shakhnarovich, G., Bi, S.: Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. CoRR abs/2311.06214 (2023). https://doi.org/10.48550/ARXIV.2311.06214, https://doi.org/10.48550/arXiv.2311.06214
  * [41] Li, J., Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong, Y., Sunkavalli, K., Shakhnarovich, G., Bi, S.: Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. CoRR abs/2311.06214 (2023). https://doi.org/10.48550/ARXIV.2311.06214, https://doi.org/10.48550/arXiv.2311.06214
  * [42] Li, W., Chen, R., Chen, X., Tan, P.: Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arxiv:2310.02596 (2023)
  * [43] Li, X., Zhou, D., Zhang, C., Wei, S., Hou, Q., Cheng, M.M.: Sora generates videos with stunning geometrical consistency. arXiv preprint arXiv: 2402.17403 (2024)
  * [44] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [45] Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: Barf: Bundle-adjusting neural radiance fields. In: IEEE International Conference on Computer Vision (ICCV) (2021)
  * [46] Lin, S., Liu, B., Li, J., Yang, X.: Common diffusion noise schedules and sample steps are flawed. ArXiv abs/2305.08891 (2023), https://api.semanticscholar.org/CorpusID:258714883
  * [47] Liu, M., Shi, R., Chen, L., Zhang, Z., Xu, C., Wei, X., Chen, H., Zeng, C., Gu, J., Su, H.: One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885 (2023)
  * [48] Liu, M., Xu, C., Jin, H., Chen, L., Xu, Z., Su, H., et al.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16928 (2023)
  * [49] Liu, R., Wu, R., Hoorick, B.V., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. https://arxiv.org/abs/2303.11328 (2023)
  * [50] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [51] Liu, Z., Li, Y., Lin, Y., Yu, X., Peng, S., Cao, Y.P., Qi, X., Huang, X., Liang, D., Ouyang, W.: Unidream: Unifying diffusion priors for relightable text-to-3d generation (2023)
  * [52] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
  * [53] Long, X., Lin, C.H., Wang, P., Komura, T., Wang, W.: Sparseneus: Fast generalizable neural surface reconstruction from sparse views. ArXiv abs/2206.05737 (2022), https://api.semanticscholar.org/CorpusID:249625516
  * [54] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. Proceedings of the 14th annual conference on Computer graphics and interactive techniques (1987), https://api.semanticscholar.org/CorpusID:15545924
  * [55] Lu, Y., Zhang, J., Li, S., Fang, T., McKinnon, D., Tsin, Y., Quan, L., Cao, X., Yao, Y.: Direct2.5: Diverse text-to-3d generation via multi-view 2.5d diffusion. ArXiv abs/2311.15980 (2023), https://api.semanticscholar.org/CorpusID:265456029
  * [56] Melas-Kyriazi, L., Laina, I., Rupprecht, C., Neverova, N.V., Vedaldi, A., Gafni, O., Kokkinos, F.: Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. ArXiv abs/2402.08682 (2024), https://api.semanticscholar.org/CorpusID:267636572
  * [57] Melas-Kyriazi, L., Rupprecht, C., Laina, I., Vedaldi, A.: Realfusion: 360 reconstruction of any object from a single image. In: CVPR (2023), https://arxiv.org/abs/2302.10663
  * [58] Metzer, G., Richardson, E., Patashnik, O., Giryes, R., Cohen-Or, D.: Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 (2022)
  * [59] Michel, O., Bar-On, R., Liu, R., Benaim, S., Hanocka, R.: Text2mesh: Text-driven neural stylization for meshes. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 13482–13492. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.01313, https://doi.org/10.1109/CVPR52688.2022.01313
  * [60] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [61] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 102:1–102:15 (Jul 2022). https://doi.org/10.1145/3528223.3530127, https://doi.org/10.1145/3528223.3530127
  * [62] Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.: Hologan: Unsupervised learning of 3d representations from natural images. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) pp. 2037–2040 (2019), https://api.semanticscholar.org/CorpusID:91184364
  * [63] Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-e: A system for generating 3d point clouds from complex prompts. CoRR abs/2212.08751 (2022). https://doi.org/10.48550/arXiv.2212.08751, https://doi.org/10.48550/arXiv.2212.08751
  * [64] Niemeyer, M., Geiger, A.: Giraffe: Representing scenes as compositional generative neural feature fields. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 11448–11459 (2020), https://api.semanticscholar.org/CorpusID:227151657
  * [65] Park, J., Kwon, G., Ye, J.C.: Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. CoRR abs/2310.02712 (2023). https://doi.org/10.48550/ARXIV.2310.02712, https://doi.org/10.48550/arXiv.2310.02712
  * [66] Park, K., Henzler, P., Mildenhall, B., Barron, J.T., Martin-Brualla, R.: Camp: Camera preconditioning for neural radiance fields. ACM Transactions on Graphics (TOG) 42, 1 – 11 (2023), https://api.semanticscholar.org/CorpusID:261064934
  * [67] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/pdf?id=FjNys5c7VyY
  * [68] Qian, G., Cao, J., Siarohin, A., Kant, Y., Wang, C., Vasilkovsky, M., Lee, H.Y., Fang, Y., Skorokhodov, I., Zhuang, P., Gilitschenski, I., Ren, J., Ghanem, B., Aberman, K., Tulyakov, S.: Atom: Amortized text-to-mesh using 2d diffusion. arXiv preprint arXiv: 2402.00867 (2024)
  * [69] Qian, G., Mai, J., Hamdi, A., Ren, J., Siarohin, A., Li, B., Lee, H.Y., Skorokhodov, I., Wonka, P., Tulyakov, S., Ghanem, B.: Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In: The Twelfth International Conference on Learning Representations (ICLR) (2024), https://openreview.net/forum?id=0jHkUDyEO9
  * [70] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021), http://proceedings.mlr.press/v139/radford21a.html
  * [71] Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., Novotny, D.: Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In: International Conference on Computer Vision (2021)
  * [72] Sajjadi, M.S.M., Meyer, H., Pot, E., Bergmann, U., Greff, K., Radwan, N., Vora, S., Lucic, M., Duckworth, D., Dosovitskiy, A., Uszkoreit, J., Funkhouser, T.A., Tagliasacchi, A.: Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 6219–6228. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.00613, https://doi.org/10.1109/CVPR52688.2022.00613
  * [73] Sanghi, A., Chu, H., Lambourne, J.G., Wang, Y., Cheng, C.Y., Fumero, M.: Clip-forge: Towards zero-shot text-to-shape generation. arXiv preprint arXiv:2110.02624 (2021)
  * [74] Sargent, K., Li, Z., Shah, T., Herrmann, C., Yu, H.X., Zhang, Y., Chan, E.R., Lagun, D., Fei-Fei, L., Sun, D., Wu, J.: ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310.17994 (2023)
  * [75] Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [76] Schönberger, J.L., Zheng, E., Pollefeys, M., Frahm, J.M.: Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision (ECCV) (2016)
  * [77] Schwarz, K., Sauer, A., Niemeyer, M., Liao, Y., Geiger, A.: Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. ArXiv abs/2206.07695 (2022), https://api.semanticscholar.org/CorpusID:249674609
  * [78] Seo, J., Jang, W., Kwak, M.S., Ko, J., Kim, H., Kim, J., Kim, J.H., Lee, J., Kim, S.: Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07937 (2023)
  * [79] Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C., Wei, X., Chen, L., Zeng, C., Su, H.: Zero123++: a single image to consistent multi-view diffusion base model (2023)
  * [80] Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv:2308.16512 (2023)
  * [81] Siddiqui, Y., Alliegro, A., Artemov, A., Tommasi, T., Sirigatti, D., Rosov, V., Dai, A., Nießner, M.: Meshgpt: Generating triangle meshes with decoder-only transformers. CoRR abs/2311.15475 (2023). https://doi.org/10.48550/ARXIV.2311.15475, https://doi.org/10.48550/arXiv.2311.15475
  * [82] Sun, J., Zhang, B., Shao, R., Wang, L., Liu, W., Xie, Z., Liu, Y.: Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.16818 (2023)
  * [83] Szymanowicz, S., Rupprecht, C., Vedaldi, A.: Splatter image: Ultra-fast single-view 3d reconstruction. In: arXiv (2023)
  * [84] Szymanowicz, S., Rupprecht, C., Vedaldi, A.: Viewset diffusion: (0-)image-conditioned 3D generative models from 2D data. In: ICCV (2023)
  * [85] Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 (2024)
  * [86] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 (2023)
  * [87] Tang, J., Wang, T., Zhang, B., Zhang, T., Yi, R., Ma, L., Chen, D.: Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior (2023)
  * [88] Tang, S., Chen, J., Wang, D., Tang, C., Zhang, F., Fan, Y., Chandra, V., Furukawa, Y., Ranjan, R.: Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint arXiv:2402.12712 (2024)
  * [89] Tang, S., Zhang, F., Chen, J., Wang, P., Yasutaka, F.: Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint 2307.01097 (2023)
  * [90] Trevithick, A., Yang, B.: Grf: Learning a general radiance field for 3d representation and rendering. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 15162–15172 (2020), https://api.semanticscholar.org/CorpusID:236975860
  * [91] Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 548–557 (2020), https://api.semanticscholar.org/CorpusID:216080881
  * [92] Wang, C., Chai, M., He, M., Chen, D., Liao, J.: Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 3825–3834. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.00381, https://doi.org/10.1109/CVPR52688.2022.00381
  * [93] Wang, P., Xu, D., Fan, Z., Wang, D., Mohan, S., Iandola, F., Ranjan, R., Li, Y., Liu, Q., Wang, Z., Chandra, V.: Taming mode collapse in score distillation for text-to-3d generation. arXiv preprint: 2401.00909 (2023)
  * [94] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. ArXiv abs/2106.10689 (2021), https://api.semanticscholar.org/CorpusID:235490453
  * [95] Wang, P., Shi, Y.: Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 (2023)
  * [96] Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N., Funkhouser, T.A.: Ibrnet: Learning multi-view image-based rendering. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 4690–4699. Computer Vision Foundation / IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.00466, https://openaccess.thecvf.com/content/CVPR2021/html/Wang_IBRNet_Learning_Multi-View_Image-Based_Rendering_CVPR_2021_paper.html
  * [97] Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., Guo, B.: RODIN: A generative model for sculpting 3d digital avatars using diffusion. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 4563–4573. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.00443, https://doi.org/10.1109/CVPR52729.2023.00443
  * [98] Wang, X., Wang, Y., Ye, J., Wang, Z., Sun, F., Liu, P., Wang, L., Sun, K., Wang, X., He, B.: Animatabledreamer: Text-guided non-rigid 3d model generation and reconstruction with canonical score distillation (2023)
  * [99] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213 (2023)
  * [100] Watson, D., Chan, W., Martin-Brualla, R., Ho, J., Tagliasacchi, A., Norouzi, M.: Novel view synthesis with diffusion models. ArXiv abs/2210.04628 (2022), https://api.semanticscholar.org/CorpusID:252780361
  * [101] Weng, H., Yang, T., Wang, J., Li, Y., Zhang, T., Chen, C.L.P., Zhang, L.: Consistent123: Improve consistency for one image to 3d object synthesis (2023)
  * [102] Wu, R., Mildenhall, B., Henzler, P., Park, K., Gao, R., Watson, D., Srinivasan, P.P., Verbin, D., Barron, J.T., Poole, B., Holynski, A.: Reconfusion: 3d reconstruction with diffusion priors. arXiv (2023)
  * [103] Xu, D., Yuan, Y., Mardani, M., Liu, S., Song, J., Wang, Z., Vahdat, A.: Agg: Amortized generative 3d gaussians for single image to 3d. arXiv preprint arXiv: 2401.04099 (2024)
  * [104] Xu, Y., Tan, H., Luan, F., Bi, S., Wang, P., Li, J., Shi, Z., Sunkavalli, K., Wetzstein, G., Xu, Z., Zhang, K.: Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. ArXiv abs/2311.09217 (2023), https://api.semanticscholar.org/CorpusID:265213192
  * [105] Yariv, L., Puny, O., Neverova, N., Gafni, O., Lipman, Y.: Mosaic-sdf for 3d generative models. arXiv preprint arXiv: 2312.09222 (2023)
  * [106] Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 4578–4587. Computer Vision Foundation / IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.00455, https://openaccess.thecvf.com/content/CVPR2021/html/Yu_pixelNeRF_Neural_Radiance_Fields_From_One_or_Few_Images_CVPR_2021_paper.html
  * [107] Yu, X., Xu, M., Zhang, Y., Liu, H., Ye, C., Wu, Y., Yan, Z., Liang, T., Chen, G., Cui, S., Han, X.: Mvimgnet: A large-scale dataset of multi-view images. In: CVPR (2023)
  * [108] Zeng, X., Chen, X., Qi, Z., Liu, W., Zhao, Z., Wang, Z., FU, B., Liu, Y., Yu, G.: Paint3d: Paint anything 3d with lighting-less texture diffusion models (2023)
  * [109] Zeng, Y., Wei, G., Zheng, J., Zou, J., Wei, Y., Zhang, Y., Li, H.: Make pixels dance: High-dynamic video generation. arXiv:2311.10982 (2023)
  * [110] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 586–595. Computer Vision Foundation / IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00068, http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_The_Unreasonable_Effectiveness_CVPR_2018_paper.html
  * [111] Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification. ACM Transactions on Graphics (TOG) 37, 1 – 12 (2018), https://api.semanticscholar.org/CorpusID:219893035
  * [112] Zhou, Z., Tulsiani, S.: Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In: CVPR (2023)
  * [113] Zhu, J.Y., Zhang, Z., Zhang, C., Wu, J., Torralba, A., Tenenbaum, J.B., Freeman, B.: Visual object networks: Image generation with disentangled 3d representations. In: Neural Information Processing Systems (2018), https://api.semanticscholar.org/CorpusID:54008375
  * [114] Zhu, J., Zhuang, P.: Hifa: High-fidelity text-to-3d with advanced diffusion guidance. CoRR abs/2305.18766 (2023). https://doi.org/10.48550/arXiv.2305.18766, https://doi.org/10.48550/arXiv.2305.18766
  * [115] Zhu, Z., Fan, Z., Jiang, Y., Wang, Z.: Fsgs: Real-time few-shot view synthesis using gaussian splatting (2023)
  * [116] Zhuang, J., Wang, C., Lin, L., Liu, L., Li, G.: Dreameditor: Text-driven 3d scene editing with neural fields. In: Kim, J., Lin, M.C., Bickel, B. (eds.) SIGGRAPH Asia 2023 Conference Papers, SA 2023, Sydney, NSW, Australia, December 12-15, 2023. pp. 26:1–26:10. ACM (2023). https://doi.org/10.1145/3610548.3618190, https://doi.org/10.1145/3610548.3618190
  * [117] Zou, Z.X., Yu, Z., Guo, Y.C., Li, Y., Liang, D., Cao, Y.P., Zhang, S.H.: Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 (2023)