Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models
for 3D Generation

Haochen Wang^*1   Xiaodan Du^*1   Jiahao Li^*1   Raymond A. Yeh^2   Greg Shakhnarovich^1

^1TTI-Chicago  ^2Purdue University

1cA zoomed out high-quality photo of Temple of Heaven     1cA high quality photo of a delicious burger

A high quality photo of a Victorian style wooden chair
with velvet upholstery

1cA high quality photo of a classic silver muscle car

figureResults for text-driven 3D generation using Score Jacobian Chaining with Stable Diffusion as the pretrained model.

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score,
and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it.
We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION 5B dataset.

[* Equal contribution.]

§ INTRODUCTION

We introduce a method that converts a pretrained 2D diffusion generative model on images into a 3D generative model of radiance fields, without requiring access to any 3D data. The key insight is to interpret diffusion models as learned predictors of a gradient field, often referred to as the score function of the data log-likelihood. We apply the chain rule on the estimated score, hence the name Score Jacobian Chaining (SJC).

Following Hyvärinen and Dayan, 2005, the score is defined as the gradient of the log-density function with respect to the data. Diffusion models of various families [Song and Ermon, 2019, Sohl-Dickstein et al., 2015, Ho et al., 2020, Song et al., 2021] can all be interpreted [Song et al., 2021, Kingma et al., 2021, Karras et al., 2022] as modeling $\score{\vx}{\sigma}$ the denoising score
at noise level $\sigma$. For readability, we refer to the denoising score as the score.
Generating a sample from a diffusion model involves repeated evaluations of the score function
from large to small $\sigma$ level, so that a sample $\vx$
gradually moves closer to the data manifold.
It can be loosely interpreted as gradient descent, with
precise control on the step sizes so that data distribution evolves to match the annealed $\sigma$ level
(ancestral sampler [Ho et al., 2020], SDE and probability-flow ODE [Song et al., 2021], etc.).
While there are other perspectives to a diffusion model [Sohl-Dickstein et al., 2015, Ho et al., 2020], here we are primarily motivated from the viewpoint that diffusion models produce a gradient field.

A natural question to ask is whether the chain rule can be applied to the learned gradients. Consider a diffusion model on images. An image $\vx$ may be parameterized by some function $f$ with parameters $\vtheta$, , $\vx = f(\vtheta)$. Applying the chain rule through the Jacobian $\pdv{\vx}{\vtheta}$ converts a gradient on image $\vx$ into a gradient on the parameter $\vtheta$. There are many potential use cases for pairing a pretrained diffusion model with different choices of $f$. In this work we are interested in exploring the connection between 3D and multiview 2D by choosing $f$ to be a differentiable renderer, thus creating a 3D generative model using only pretrained 2D resources.

Many prior works [Yang et al., 2019, Cai et al., 2020, Wu et al., 2016] perform 3D generative modeling by training on 3D datasets [Chang et al., 2015, Sun et al., 2018, Wu et al., 2015, Koch et al., 2019]. This approach is often as challenging as it is format-ambiguous.
In addition to the high data acquisition cost of 3D assets [Deltombe, 2022], there is no universal data format: point clouds, meshes, volumetric radiance field, etc,
all have computational trade-offs. What is common to these 3D assets is that they can be rendered into 2D images.
An inverse rendering system, or a differentiable renderer [Loper and Black, 2014, Lassner and Zollhofer, 2021, Nimier-David et al., 2019, Li et al., 2018, Mildenhall et al., 2021], provides access to the Jacobian $\mJ_\pi \triangleq \pdv{\vx_{\pi}}{\vtheta}$ of a rendered image $\vx_{\pi}$ at camera viewpoint $\pi$ with respect to the underlying 3D parameterization $\vtheta$. Our method uses differentiable rendering to aggregate 2D image gradients over multiple viewpoints into a 3D asset gradient, and lifts a generative model from 2D to 3D. We parameterize a 3D asset $\vtheta$ as a radiance field stored on voxels and choose $f$ to be the volume rendering function.

A key technical challenge is that computing the 2D score by directly evaluating a diffusion model on a rendered image $\vx_\pi$ leads to an out-of-distribution (OOD) problem.
Generally, diffusion models are trained as denoisers and have only seen noisy inputs during training.
On the other hand, our method requires evaluating the denoiser on non-noisy rendered images from a 3D asset during optimization, and it leads to the OOD problem.
To address the issue, we propose , an approach to estimate the score for non-noisy images.

Empirically, we first validate the effectiveness of  at solving
the OOD problem and explore the hyperparameter choices on a simple 2D image canvas.
Here we identify open problems on using unconditioned diffusion models
trained on FFHQ and LSUN Bedroom.
Next, we use Stable Diffusion, a model pretrained on the web-scale LAION dataset to perform SJC for 3D generation, as shown in fig:teaser.
Our contributions are as follows:

* We propose a method for lifting a 2D diffusion model to 3D via an application of the chain rule.

* We illustrate the challenge of OOD when using a pretrained denoiser and propose  to resolve it.

* We point out the subtleties and open problems on applying  as gradient for optimization.

* We demonstrate the effectiveness of SJC for the task of 3D text-driven generation.

§ RELATED WORKS

Diffusion models have
recently advanced to image generation on Internet-scale datasets [Nichol et al., 2021, Ramesh et al., 2022, Saharia et al., 2022, Rombach et al., 2022, Schuhmann et al., 2022, Radford et al., 2021, Dhariwal and Nichol, 2021]. A diffusion model
could be interpreted as either a VAE [Ho et al., 2020, Sohl-Dickstein et al., 2015] or a denoising score-matcher [Vincent, 2011, Song and Ermon, 2019, Song et al., 2021]. Notably, models trained under one regime can be directly used
for inference and sampling by the other [Song et al., 2021, Karras et al., 2022]; they are in practice largely equivalent.

Neural radiance fields (NeRF) is a family of inverse rendering algorithms that
have excelled at multiview 3D reconstruction tasks including view synthesis and surface geometry estimation [Mildenhall et al., 2021, Martin-Brualla et al., 2021, Oechsle et al., 2021, Yariv et al., 2021, Wang et al., 2021]. Conceptually, a 3D asset is represented as a dense grid of $\mathrm{RGB}$ colors and spatial density $\tau$, and rendered into images in a way analogous to alpha compositing [Max, 1995].
NeRF parameterizes the $(\mathrm{RGB},\tau)$ volume with a neural network, but querying
the network densely in 3D incurs significant compute costs.
Alternatively, Voxel NeRFs [Liu et al., 2020, Sun et al., 2022, Yu et al., 2021, Chen et al., 2022] store the volume on voxels and observe no loss in end task performance [Sun et al., 2022, Yu et al., 2021]. Querying voxels is a simple
memory operation that is much faster than a feedforward pass of a neural network. Here we use a customized voxel radiance field with hyperparameters based on DvGO [Sun et al., 2022] and TensoRF [Chen et al., 2022].

2D-supervised 3D GANs pioneered [Nguyen-Phuoc et al., 2019, Rajeswar et al., 2018, Zhang et al., 2020] the approach of training 3D generative models using only unstructured 2D images, and promise greater scalability in terms of data. Rather than supervising directly on the 3D asset a model generates, these methods supervise the 2D renderings of the generated 3D asset, often using an adversarial loss [Schwarz et al., 2020, Chan et al., 2021, Niemeyer and Geiger, 2021, Chan et al., 2022, Zhao et al., 2022]. In other words, only images are needed as training data. However, training such a 3D generative model from scratch is still challenging [Niemeyer and Geiger, 2021]. Recent empirical evaluation remains mostly on human and animal faces [Chan et al., 2022]. Our method does the opposite: we take an image generative model that is already pretrained on large amounts of 2D data and use it to guide the iterative optimization of a 3D asset. Optimization-based generation makes it much slower compared to 3D GANs, but it becomes possible to harness powerful off-the-shelf 2D generative models such as Stable Diffusion [Rombach et al., 2022] for greater content diversity.

CLIP-guided, optimization-based 3D generative models share a similar philosophy
of optimizing 3D assets by guiding on 2D renderings [Jetchev, 2021, Michel et al., 2022, Khalid et al., 2022, Jain et al., 2022, Lee and Chang, 2022, Hong et al., 2022]. Among them, DreamFields [Jain et al., 2022] and PureClipNeRF [Lee and Chang, 2022] also use NeRF as their differentiable renderers.
In this case, the 2D guidance comes from CLIP [Radford et al., 2021],
a pretrained image-text matching model. These works optimize the 3D assets so that the
image renderings match a user-provided text prompt. Since CLIP is not a 2D generative
model per se, such a pipeline usually creates some abstract distilled content [Liu et al., 2021]
that looks very different from real images.
In contrast, we use diffusion models, which are proper 2D generative models, to create realistic looking 3D content.

DreamFusion. The recently arXived work by Poole et al., 2022, independent and concurrent to our work, proposes an algorithm that is similar to our approach at the pseudo-code level. Differently, their procedure uses the mathematical setup by Graikos et al., 2022 to
search for image parametrization that minimizes the training loss of a diffusion model.
In contrast, our work is motivated by applying the chain rule to the 2D score. The key differences have been summarized in sec:compare_dream_fusion. In terms of implementation, we do not have access to the close-sourced Imagen [Saharia et al., 2022] diffusion model. Instead, we use the pretrained Stable Diffusion model released by Rombach et al., 2022.
For a comparison with DreamFusion, we use with a third-party implementation based on the same diffusion model, namely Stable-DreamFusion[<github.com/ashawkey/stable-dreamfusion>].

§ PRELIMINARIES

To establish a common notation, we briefly review the score-based perspective of diffusion models. For readers familiar with VAE literature on diffusion models, we provide a concise score-based formula card and more details in Appendix sec:score_perspective to connect these ideas.

Denoising score matching.
Given a dataset of samples $\gY=\{\vy_i\}$ drawn from $p_{\text{data}}$,
a diffusion model revolves primarily around learning a denoiser $D$
by minimizing the difference between a noised sample $\vy+ \sigma\vn$ and $\vy$,

_∼p_data_∼(0, ) D(+σ; σ)-_2^2,
$D$ is denoising the input $\vy+\sigma\vn$, for a range of $\sigma$ values. For 2D images, $D$ is commonly chosen to be a ConvNet. Variants such as DDPM [Ho et al., 2020] parameterized the ConvNet to instead predict
a noise residual $\hat{\bm{\epsilon}}$, and these models can be converted back to the form of
a denoiser by [Song et al., 2021]
D(; σ) = - σϵ̂().
In this paper, we treat all pretrained diffusion models as denoisers, and perform the interface
conversion in our implementation when needed.

Score from denoiser.
Let $p_{\sigma}(\vx)$ denote the data distribution perturbed by Gaussian noise of standard deviation $\sigma$. It is shown in prior works [Song and Ermon, 2019, Hyvärinen and Dayan, 2005] that the denoiser $D$ trained according to eq:training provides a good approximation to the denoising score:

σ ≈D(; σ) -/σ^2.
A denoising diffusion model estimates the score function of the noised distribution $p_\sigma(\vx)$ at various $\sigma \in \{\sigma_i\}_{i=1}^T$. To perform sampling, the diffusion model gradually updates a sample through a sequence of noise levels of $\sigma_T > \dots > \sigma_0 = 0$. $\{\sigma_i\}$ are chosen empirically, with a typical range being $[0.01, 157]$ [Ho et al., 2020] in the case of DDPM.

Score as mean-shift. A helpful intuition is that the score behaves like mean-shift [Cheng, 1995, Comaniciu and Meer, 1999]. If we simplify $p_{\text{data}}$ to be an empirical data distribution over the i.i.d. samples $\{\vy_i\}$, then at noise level $\sigma$, $p_{\sigma}(\vx)$ takes the form of a mixture of Gaussians [Song and Ermon, 2020]

\begin{align}\label{eq:gmm}
p_{\sigma}(\vx) = \E_{\vy \sim p_\text{data}} ~ \mathcal{N}(\vx;\, \vy, \, \sigma^2 \rmI).
\end{align}

In this case there exists a closed-form expression [Song and Ermon, 2020, Karras et al., 2022] to the optimal denoiser

\begin{align}\label{eq:denoiser}
D(\vx; \sigma) = \frac{\sum_i \mathcal{N}(\vx;\, \vy_i,\, \sigma^2 \mathbf{I}) \, \vy_i}{\sum_i \mathcal{N}(\vx;\, \vy_i,\, \sigma^2 \mathbf{I})}.
\end{align}

In other words, $D(\vx;\, \sigma)$ is a locally weighted mean of data samples $\{\vy_i\}$ around $\vx$ under a Gaussian kernel with bandwidth $\sigma$. The denoising score function can be thought of as a
non-parametric guide on how to update $\vx$ in order to move it towards its weighted nearest neighbors.

§ SCORE JACOBIAN CHAINING FOR 3D GENERATION

Let $\vtheta$ denotes the parameters of a 3D asset, , voxel grid of $ (\mathrm{RGB}, \tau) $ as in Sec. <ref>. Our goal is to model and sample from the distribution $p(\vtheta)$ to generate a 3D scene. In our setting, only a pretrained 2D diffusion model on images $p(\vx)$ is given and we do not have access to 3D data. To relate the 2D and 3D distributions $p(\vx)$ and $p(\vtheta)$, we assume that the probability density of 3D asset $\vtheta$ is proportional to the expected probability densities of its multiview 2D image renderings $\vx_\pi$ over camera poses $\pi$, ,

\begin{align}\label{eq:p_theta}
p_\sigma(\vtheta) \propto \E_{\pi} \big[ p_\sigma(\vx_\pi(\vtheta)) \big],
\end{align}

up to a normalization constant $Z = \int \E_{\pi} \big[ p_\sigma(\vx_\pi(\vtheta)) \big] d\vtheta.$
That is, a 3D asset $\vtheta$ is as likely as its 2D renderings $\vx_\pi$.

Next, we establish a lower bound, $\log \tilde{p}_\sigma(\vtheta)$, on the distribution in eq:p_theta  using Jensen's inequality:
logp_σ() =    log[ _π (p_σ(_π)) ] - logZ

≥    _π [ logp_σ(_π) ] - logZ ≜logp̃_σ().
Recall that the score is the gradient of log probability density of data. By chain rule

\begin{align}
\grad_{\vtheta} \log\tilde{p}_\sigma(\vtheta) &= \E_{\pi} \left[ \grad_{\vtheta} \log p_\sigma(\vx_\pi) \right ] \\
\pdv{\log\tilde{p}_\sigma(\vtheta)}{\vtheta} &= \E_{\pi} \left[ \pdv{\log p_\sigma(\vx_\pi)}{\vx_\pi}  \cdot \pdv{\vx_\pi}{\vtheta} \right ] \\
\label{eq:3D_score}
\underbrace{\grad_{\vtheta}\log\tilde{p}_\sigma(\vtheta)}_{\text{3D score}}
&= \E_{\pi} [ ~
\underbrace{\grad_{\vx_\pi}\log p_\sigma(\vx_\pi)}_{\text{2D score; pretrained}}
\cdot
\underbrace{ \mJ_\pi \vphantom{\grad_{\vx_\pi}\log p(\vx_\pi)} }_{\text{renderer Jacobian}} ].
\end{align}

We will next discuss how to compute the 2D score in practice using a pretrained diffusion model.

§.§ Computing 2D Score on Non-Noisy Images

Computing the 3D score in eq:3D_score requires the 2D score on $\vx_\pi$. A first attempt would be to directly apply the score from the denoiser in  eq:score_approx, ,

score(_π, σ) ≜(D(_π; σ) - _π)/σ^2.
Unfortunately, evaluating the pretrained denoiser $D$ on $\vx_\pi$ causes an out-of-distribution (OOD) problem.
From the training objective in eq:training, at each noise level
$\sigma$, the denoiser $D$ has only seen noisy inputs of the distribution $\vy + \sigma \vn$ where $ \vy \sim p_\text{data}$ and $\vn \sim \gauss{0}{\mathbf{I}}$. However,
a rendered image $\vx_\pi$ from 3D asset $\vtheta$ is generally not consistent with such distribution.

We illustrate this OOD situation in fig:ood.
Given a denoiser pretrained on FFHQ [Karras et al., 2019] by Baranchuk et al., 2021, we visualize the output $D(\vx_{\text{blob}}; \sigma=6.5)$ where the input $\vx_{\text{blob}}$ is a non-noisy image showing an orange blob centered on a grey canvas. Under the intuition that $D$ predicts a weighted nearest neighbor as reviewed in (<ref>),
we expect the denoiser to blend the orange blob with the manifold of faces.
However in reality we observe sharp artifacts when updating with this score $(D(\vx_{\text{blob}}; \sigma) - \vx_{\text{blob}}) / \sigma^2 $ and the image becomes further away from the face manifold.

To address the OOD problem, we propose  ($\nescore{}$). It computes the score on non-noisy images $\vx_\pi$ with a denoiser $D$ by adding noise to the input, and then considering the expectation of the predicted scores the random noise,

\begin{align}
\label{eq:score_algorithm}
&\nescore{}(\vx_\pi, \sqrt{2}\sigma) \\
\triangleq & \E_{\vn \sim \gauss{0}{\rmI}}~ \left[\mathrm{score}(\vx_\pi + \sigma\vn, \sigma)\right] \\
=& \E_{\vn}~ \scalemath{0.9}{
\left[\frac{{D}(\vx_\pi + \sigma\vn, \sigma) - (\vx_\pi + \sigma\vn)}{\sigma^2}\right]
} \label{eq:before_ens} \\
=& \E_{\vn}~ \scalemath{0.9}{
\left[ \frac{{D}(\vx_\pi + \sigma\vn, \sigma) - \vx_\pi}{\sigma^2} \right]
\bcancel{
- \underbrace{\E_{\vn}\left[\frac{\vn}{\sigma}\right]}_{\text{=$0$}}
\end{align}

In practice, we use the Monte Carlo estimate of the expectation in <ref>.
The algorithm is illustrated in fig:score_x_pi. Given a set of sampled noises $\{\vn_i \}$, each $D(\vx_\pi + \sigma \vn_i)$ provides an update direction on the perturbed input $\vx_\pi + \sigma \vn_i$. By averaging over the noise perturbations $\{\vn_i \}$,
we obtain an update direction on $\vx_\pi$ itself.

Computing $\nescore{}$ on 2D renderings $\vx_\pi$. Directly evaluating $D(\vx_\pi; \sigma)$ leads to an OOD problem. Instead, we add noise to $\vx_\pi$, and evaluate $D(\vx_\pi + \sigma \vn; \sigma)$ (blue dots). The $\nescore{}$ is then computed by averaging over the brown dashed arrows, corresponding to multiple samples of $\vn$.
See sec:compute_render_score for details.

Justifying $\nescore{}$ in <ref>.
We show that  provides an approximation to the score on $\vx_\pi$ at
an inflated noise level of $\sqrt{2}\sigma$
(_π, √(2)σ) ≈_π√(2)σ.

Assuming an empirical data distribution $p_{\sigma}(\vx)$ in eq:gmm, for any $\vx \in \R^d$

logp_√(2)σ() ≥_∼0 logp_σ(+ σ).

Observe that the LHS of eq:convolve is a convolution of two Gaussians, therefore

_∼0  [𝒩(+ σ;  ,  σ^2 ) ]
= 𝒩(;  ,  2σ^2 )
Recall that $p_{\sigma}(\vx)$ is a mixture of Gaussians per eq:gmm;

\begin{align}
\mkern-18mu \scalemath{0.9}{p_{\sqrt{2}\sigma}(\vx)}  &= \E_{\vy \sim p_\text{data}}~ \mathcal{N}(\vx;\, \vy, \, 2\sigma^2 \rmI) \label{eq:left}  \\
&= \E_{\vy \sim p_\text{data}}~ \E_{\vn \sim \gauss{0}{\rmI}}~ \mathcal{N}(\vx+ \sigma\vn;\, \vy, \, \sigma^2 \rmI)  \\
&= \E_{\vn \sim \gauss{0}{\rmI}}~ \E_{\vy \sim p_\text{data}}~ \mathcal{N}(\vx+ \sigma\vn;\, \vy, \, \sigma^2 \rmI) \\
&= \E_{\vn \sim \gauss{0}{\rmI}}~ p_\sigma(\vx + \sigma\vn). \label{eq:right}
\end{align}

Taking the $\log$ on both sides of  eq:right and by Jensen's inequality, we arrive at eq:density_dilate.

Assuming a trained denoiser $D$ as in eq:score_approx, our $\nescore{}(\vx_\pi, \sqrt{2}\sigma)$ in <ref> computes the gradient a lower bound of $\log p_{\sqrt{2}\sigma}(\vx)$.

By taking the gradient of the RHS of <ref>,
_  _ logp(+ σ, σ) =    _ + σσ

=     _ [score(_π+σ, σ)].
which is the proposed $\nescore{}$ algorithm
in <ref>.

§.§ Inverse Rendering on Voxel Radiance Field

With the computation of the 2D score resolved, the other half of our setup in eq:3D_score requires access to the Jacobian of a differentiable renderer.

3D Representation. We represent a 3D asset $\vtheta$ as a voxel radiance field [Yu et al., 2021, Sun et al., 2022, Chen et al., 2022], which is much faster to access and update compared to a vanilla NeRF parameterized by a neural network [Mildenhall et al., 2021]. The parameters $\vtheta$ consist of a density voxel grid $ \mathbf{V}^{(\text{density})} \in \R^{1 \times N_x \times N_y \times N_z} $ and
a voxel grid of appearance features $ \mathbf{V}^{(\text{app})} \in \R^{c \times N_x \times N_y \times N_z} $. Conventionally the appearance features are simply the $\mathrm{RGB}$ colors
and $c = 3$. For simplicity, we do not model view dependencies in this work.

Inverse Volumetric Rendering. Image rendering is performed independently
along a camera ray through each pixel.
We cut a camera light ray into equally distanced segments of length $d$,
and at the spatial location corresponding to the beginning of the $i$-th segment
we sample a $(\mathrm{RGB}_i, \tau_i)$ tuple from the color and density grids using trilinear interpolation. These values are alpha-composited using volume rendering quadrature [Max, 1995] into the pixel color $C = \sum_i w_i \cdot \mathrm{RGB}_i$, where

\begin{align}
\label{eq:volrend}
w_i = \alpha_i \cdot \prod_{j=0}^{i-1} (1 - \alpha_j);\quad \alpha_i = 1 - \exp(-\tau_i d).
\end{align}

Volume rendering of $\vtheta$ is directly differentiable. At a rendered image $\vx_\pi$, the Vector-Jacobian product in eq:3D_score between $\nescore({\vx_\pi})$ and the Jacobian $J_\pi = \pdv{\vx_\pi}{\vtheta}$ is computed by back-propagating the score through eq:volrend. This Vector-Jacobian product provides us with the 3D gradient needed for generative modeling on the voxel radiance field.

Regularization Strategies.
The voxel grid is a very powerful 3D representation for volumetric rendering. Given noisy 2D guidance, the model may cheat by populating the entire grid with small densities such that the combined effect from one view hallucinates a plausible image. We propose several techniques to encourage the formation of a coherent 3D structure.

Emptiness Loss:  Ideally, the space should be sparse with near zero densities except at the object. We propose an emptiness loss to encourage sparsity on a ray $\mathbf{r}$:

_emptiness(𝐫) = 1/N ∑_i=1^N log(1+β·w_i ),
where $w_i$ are the alpha-composited weights shown in (<ref>).
The shape of the $\log$ function imposes severe penalties at the onset of small weights, but does not grow aggressively if the weights are large. It is consistent with our aim to eliminate
small densities. The hyperparameter $\beta$ controls the steepness of the loss function near 0. A larger $\beta$ will put more emphasis on eliminating low-density noise. We set $\beta = 10$.

Emptiness Loss Schedule:
We use a hyperparameter $\lambda$ to control the contribution of the emptiness loss. If we apply a large emptiness loss, it will hinder the learning of geometry in the early stage of training. But if the emptiness is too small, there will be floating density artifacts. We adopt a two-stage noise elimination schedule to deal with this problem. In the first $K$ iterations, we use a relatively small weighting factor $\lambda_1$. After the $K^{\text{th}}$ iteration, it is increased to a larger $\lambda_2$. In our experiments $\lambda_1=1\times 10^4$ and $\lambda_2=2\times 10^5$. We provide an ablation study of this technique in <ref> to show its effectiveness.

Center Depth Loss: Sometimes the optimization places the object away from the scene center. The object either becomes small or wander around the image boundary. For the few cases when this happens we apply a center depth loss

\begin{equation}\label{eq:center_depth_loss}
\gL_{\text{center}}(\mD) = -\log\left(\frac{1}{|\mathcal{B}|}\sum_{\mathbf{p}\in \mathcal{B}} \mD(\mathbf{p}) - \frac{1}{|\mathcal{B}^\complement|}\sum_{\mathbf{q}\notin \mathcal{B}} \mD(\mathbf{q})\right)
\end{equation}

where $\mD$ is the depth image, $\mathcal{B}$ is a box (set of pixel locations)
at the center of the image, and $\mathcal{B}^\complement$ is its complement.

§.§ SJC vs. DreamFusion

In this section, we describe the differences and the connections between our SJC and DreamFusion.

Differences from DreamFusion.
In terms of formulation, DreamFusion's computation of the gradient $\theta$ involves a U-Net Jacobian term (see Eq. 2 in their paper [Poole et al., 2022]). In practice, they “found that omitting the U-Net Jacobian term” to be more effective. On the other hand, this U-Net Jacobian term does not appear in our formulation. Their additional justification in the appendix actually leans more towards our viewpoint.
An additional contribution of ours beyond DreamFusion [Poole et al., 2022] is our analysis of the effect that the OOD problem has when using a denoiser on rendered images (Claim <ref>), and the method to address it.
For the variance reduction technique, namely the use of the Monte-Carlo estimate on <ref>, or $\hat{\bm{\epsilon}} - \bm{\epsilon}$ (in DreamFusion), vs. on <ref>, we observe comparable performance between the two methods empirically for 3D generation.

Influences by DreamFusion.
At the time of this submission, DreamFusion is a concurrent arXiv paper. However, as we have read the paper, our research was influenced by their reported observations. In particular, we adopted the idea of randomized scheduling of $\sigma$ during 3D optimization for easier hyperparameter tuning, and used view-augmented language prompting that improves the overall 3D quality. For future work we do hope to explore a more general solution than view-dependent prompts.

§ EXPERIMENTS

We conduct experiments on both unconditioned and conditioned diffusion models to have a more comprehensive understanding of the properties of SJC.

DDPMs trained on FFHQ and LSUN Bedroom are unconditioned diffusion
models with an architecture based on the implementation by Dhariwal and Nichol, 2021.
They are trained on an image resolution of $256 \times 256$.
FFHQ [Karras et al., 2019] is a dataset of aligned faces with diverse coverage of gender, age, race, facial appearance as well as head poses. LSUN Bedroom [Yu et al., 2015] includes bedroom images with varied furniture layout plans and rich interior design styles.

Stable Diffusion is an expanded work based on Latent Diffusion Model (LDM) developed by Rombach et al., 2022. It is trained on the
LAION5B dataset [Schuhmann et al., 2022]. We use the release version v1.5.
Diffusion is performed on a latent space of $4 \times 64 \times 64$, then upsampled to $3 \times 256 \times 256$ by a decoder. The model is natively trained for text-conditioned image generations, and exposes a guidance scale parameter that controls the strength of language conditioning [Ho and Salimans, 2022]. Intuitively a larger guidance scale makes the conditioned image distribution more faithful to the text prompt by trading off sample diversity.

§.§ Validating $\nescore{}$ on 2D images.

Before directly jumping to 3D generation, we first verify that $\nescore{}$ provides
effective guidance on a simple 2D image canvas. In other words, here $\vtheta$ is a grid of RGB values and $f$ is an identity function. The hope is that gradient descent on the vector
field produced by $\nescore$ creates high quality images. Here an important decision to make is the schedule of $\{\sigma_i\}$ at which we compute $\nescore{}$.

We experimented with an annealed schedule (Annealed $\sigma$) vs. a random schedule (Random $\sigma$) as proposed in DreamFusion. Under the Annealed $\sigma$ schedule, we start from a large
$\sigma$ and gradually decrease it as we update the image canvas $\vx$. $\nescore{}$ computed at larger $\sigma$ level attends to high level image structure while smaller $\sigma$ provides
stronger guidance on detailed features. The Random $\sigma$ schedule on the contrary uniformly
samples a $\sigma$ at every step. We show
qualitative comparisons
in <ref>.

For unconditioned diffusion models trained on FFHQ,
we observe that Annealed $\sigma$ performs better
than Random $\sigma$, and the image samples have better pose variation and quality. Particularly, the randomized $\sigma$ exhibits severe mode-seeking behavior converging to average faces. In the case of LSUN Bedroom, the mode-seeking behavior results in a blurry image
canvas with no content.

On the other hand, natural language prompting plays a critical role when sampling images
with Stable Diffusion. When the language guidance is set to a regular level of $3.0$, the observations are broadly consistent with sampling on FFHQ and LSUN. Random $\sigma$
schedule produces blurry outputs. However, when the guidance scale is elevated to $10.0$,
Random $\sigma$ schedule begins to generate crisp, clean images and outperforms Annealed $\sigma$
schedule. Despite various sophisticated strategies on Annealed $\sigma$ scheduling (see our code for details), at a high language guidance scale Randomized $\sigma$ remains the better option. We hypothesize that stronger language guidance forces the image distribution to be narrower and more beneficial
for a mode-seeking algorithm. We acknowledge none of the images in fig:2dcomp
can match the sample quality of a standard diffusion inference pipeline, and the right
way to apply $\nescore$ as gradient for optimization remains an open problem.

§.§ 3D Generation

In this paper, we focus on 3D Generation with the language-conditioned Stable Diffusion model.
We found that tuning the Annealed $\sigma$ schedule on FFHQ and LSUN Bedroom in 3D domain is difficult
in practice, and leave it as future work. Based on the insights from 2D experiments earlier,
we use Random $\sigma$ schedule coupled with a high language guidance scale.

Rendering with Latent 3D Features.
Stable Diffusion economizes compute by performing diffusion modeling on the latent features of a
pretrained AutoEncoder. We therefore choose to render a feature image in this latent space
from a features field [Niemeyer and Geiger, 2021, Chan et al., 2022] represented by a voxel grid in $\R^{4 \times N_x \times N_y \times N_z}$.

Qualitative Comparison.
In <ref>, we show text-prompted 3D generation results from SJC. It is capable of generating complex 3D models over a diverse set of prompts ranging from animals to the Sydney Opera House. Next, we compare SJC with Stable-DreamFusion, the third-party implementation based on the same pretrained Stable Diffusion model. In fig:comparison, we show qualitative comparisons of generated 3D assets given the same prompt. We observe that SJC generates 3D models with better image quality and more sensible structure than Stable-DreamFusion in a significant number of cases. We acknowledge that both systems exhibit quality fluctutations over different trials, and the point of this comparison is to show that our overall pipeline is competitive.

In <ref>, we conduct ablations to demonstrate the importance of the  proposed emptiness loss and scheduling of its weight $\lambda$ discussed in sec:voxels. We show results without the emptiness loss, with constant weight $\lambda$ vs our proposed scheduling of $\lambda$. We observe that our complete method (Ours) improves the quality of generated 3D models, , fewer floating artifacts and better geometry.

§ CONCLUSION

We propose an optimization-based approach to generate 3D assets from pretrained image (2D) diffusion models. The key technical contribution is the derivation of  method which bridges the gap between the denoising-trained diffusion models and the non-noisy images encountered in the process of optimizing a 3D model guided by the diffusion. We also propose a new regularization loss for improving the quality of generated 3D scene. Working with the large-scale Stable Diffusion model, we demonstrate that our approach can generate compelling 3D models, comparing favorably to available concurrent work. Finally, we investigate an interesting distinction between the  effect of noise scheduling regime in unconditional image diffusion models and a text-conditional model, and identify an avenue for future work.

§ ACKNOWLEDGEMENTS

The authors would like to thank David McAllester for feedbacks on an early pitch of the work, Shashank Srivastava and Madhur Tulsiani for discussing the $\sqrt{2}$ factor on synthetic experiments. We would like to thank friends at TRI
and 3DL lab at UChicago for suggestions on the manuscript. HC would like to thank Kavya Ravichandran for incredible officemate support, and Michael Maire for the discussion and encouragement while riding Metra.

[Baranchuk et al., 2021]
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem
Label-efficient semantic segmentation with diffusion models.
arXiv preprint arXiv:2112.03126, 2021.

[Cai et al., 2020]
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah
Snavely, and Bharath Hariharan.
Learning gradient fields for shape generation.
In Eur. Conf. Comput. Vis., 2020.

[Chan et al., 2022]
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini
De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis,
et al.
Efficient geometry-aware 3D generative adversarial networks.
In IEEE Conf. Comput. Vis. Pattern Recog., 2022.

[Chan et al., 2021]
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.
pi-GAN: Periodic implicit generative adversarial networks for
3d-aware image synthesis.
In IEEE Conf. Comput. Vis. Pattern Recog., 2021.

[Chang et al., 2015]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang,
Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.
ShapeNet: An information-rich 3D model repository.
arXiv preprint arXiv:1512.03012, 2015.

[Chen et al., 2022]
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.
Tensorf: Tensorial radiance fields.
arXiv preprint arXiv:2203.09517, 2022.

[Cheng, 1995]
Yizong Cheng.
Mean shift, mode seeking, and clustering.
IEEE Trans. Pattern Anal. Mach. Intell., 1995.

[Comaniciu and Meer, 1999]
Dorin Comaniciu and Peter Meer.
Mean shift analysis and applications.
In Int. Conf. Comput. Vis., 1999.

[Deltombe, 2022]
Amélie Deltombe.
How much does it cost to create 3D models?, Apr 2022.

[Dhariwal and Nichol, 2021]
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat GANs on image synthesis.
In Adv. Neural Inform. Process. Syst., 2021.

[Graikos et al., 2022]
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras.
Diffusion models as plug-and-play priors.
arXiv preprint arXiv:2206.09012, 2022.

[Ho et al., 2020]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In Adv. Neural Inform. Process. Syst., 2020.

[Ho and Salimans, 2022]
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint arXiv:2207.12598, 2022.

[Hong et al., 2022]
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei
Avatarclip: Zero-shot text-driven generation and animation of 3D
arXiv preprint arXiv:2205.08535, 2022.

[Hyvärinen and Dayan, 2005]
Aapo Hyvärinen and Peter Dayan.
Estimation of non-normalized statistical models by score matching.
J. Mach. Learn. Res, 2005.

[Jain et al., 2022]
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.
Zero-shot text-guided object generation with dream fields.
In IEEE Conf. Comput. Vis. Pattern Recog., 2022.

[Jetchev, 2021]
Nikolay Jetchev.
Clipmatrix: Text-controlled creation of 3D textured meshes.
arXiv preprint arXiv:2109.12922, 2021.

[Karras et al., 2022]
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
arXiv preprint arXiv:2206.00364, 2022.

[Karras et al., 2019]
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial
In IEEE Conf. Comput. Vis. Pattern Recog., 2019.

[Khalid et al., 2022]
Nasir Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.
Clip-mesh: Generating textured meshes from text using pretrained
image-text models.
ACM Trans. Graph., 2022.

[Kingma et al., 2021]
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.
Variational diffusion models.
Advances in neural information processing systems,
34:21696–21707, 2021.

[Kingma and Ba, 2014]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[Koch et al., 2019]
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey
Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo.
ABC: A big cad model dataset for geometric deep learning.
In IEEE Conf. Comput. Vis. Pattern Recog., 2019.

[Lassner and Zollhofer, 2021]
Christoph Lassner and Michael Zollhofer.
Pulsar: Efficient sphere-based neural rendering.
In IEEE Conf. Comput. Vis. Pattern Recog., 2021.

[Lee and Chang, 2022]
Han-Hung Lee and Angel X Chang.
Understanding pure clip guidance for voxel grid nerf models.
arXiv preprint arXiv:2209.15172, 2022.

[Li et al., 2018]
Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen.
Differentiable Monte Carlo ray tracing through edge sampling.
ACM Trans. Graph., 2018.

[Liu et al., 2020]
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.
Neural sparse voxel fields.
In Adv. Neural Inform. Process. Syst., 2020.

[Liu et al., 2021]
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu.
Fusedream: Training-free text-to-image generation with improved clip+
GAN space optimization.
arXiv preprint arXiv:2112.01573, 2021.

[Loper and Black, 2014]
Matthew M Loper and Michael J Black.
OpenDR: An approximate differentiable renderer.
In Eur. Conf. Comput. Vis., 2014.

[Maoutsa et al., 2020]
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper.
Interacting particle solutions of fokker–planck equations through
gradient–log–density estimation.
Entropy, 2020.

[Martin-Brualla et al., 2021]
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron,
Alexey Dosovitskiy, and Daniel Duckworth.
Nerf in the wild: Neural radiance fields for unconstrained photo
In IEEE Conf. Comput. Vis. Pattern Recog., 2021.

[Max, 1995]
Nelson Max.
Optical models for direct volume rendering.
IEEE Trans. Vis. Comput. Graph., 1995.

[Michel et al., 2022]
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.
Text2mesh: Text-driven neural stylization for meshes.
In IEEE Conf. Comput. Vis. Pattern Recog., 2022.

[Mildenhall et al., 2021]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi
Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view
Commun. ACM, 2021.

[Nguyen-Phuoc et al., 2019]
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang
HoloGAN: Unsupervised learning of 3D representations from natural
In Int. Conf. Comput. Vis., 2019.

[Nichol et al., 2021]
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin,
Bob McGrew, Ilya Sutskever, and Mark Chen.
Glide: Towards photorealistic image generation and editing with
text-guided diffusion models.
arXiv preprint arXiv:2112.10741, 2021.

[Niemeyer and Geiger, 2021]
Michael Niemeyer and Andreas Geiger.
Campari: Camera-aware decomposed generative neural radiance fields.
In Int. Conf. 3DV, 2021.

[Niemeyer and Geiger, 2021]
Michael Niemeyer and Andreas Geiger.
Giraffe: Representing scenes as compositional generative neural
feature fields.
In IEEE Conf. Comput. Vis. Pattern Recog., 2021.

[Nimier-David et al., 2019]
Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob.
Mitsuba 2: A retargetable forward and inverse renderer.
ACM Trans. Graph., 2019.

[Oechsle et al., 2021]
Michael Oechsle, Songyou Peng, and Andreas Geiger.
Unisurf: Unifying neural implicit surfaces and radiance fields for
multi-view reconstruction.
In Int. Conf. Comput. Vis., 2021.

[Poole et al., 2022]
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.
DreamFusion: Text-to-3D using 2D diffusion.
arXiv preprint arXiv:2209.14988, 2022.

[Radford et al., 2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language

[Rajeswar et al., 2018]
Sai Rajeswar, Fahim Mannan, Florian Golemo, David Vazquez, Derek
Nowrouzezahrai, and Aaron Courville.
Pix2scene: Learning implicit 3d representations from images.
openreview, 2018.

[Ramesh et al., 2022]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.

[Rombach et al., 2022]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
High-resolution image synthesis with latent diffusion models.
In IEEE Conf. Comput. Vis. Pattern Recog., 2022.

[Saharia et al., 2022]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily
Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi,
Rapha Gontijo Lopes, et al.
Photorealistic text-to-image diffusion models with deep language
arXiv preprint arXiv:2205.11487, 2022.

[Schuhmann et al., 2022]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et al.
LAION-5B: An open large-scale dataset for training next generation
image-text models.
arXiv preprint arXiv:2210.08402, 2022.

[Schwarz et al., 2020]
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.
GRAF: Generative radiance fields for 3D-aware image synthesis.
In Adv. Neural Inform. Process. Syst., 2020.

[Sohl-Dickstein et al., 2015]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In International Conference on Machine Learning, pages
2256–2265. PMLR, 2015.

[Song et al., 2021]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
In Int. Conf. Learn. Represent., 2021.

[Song and Ermon, 2019]
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
In Adv. Neural Inform. Process. Syst., 2019.

[Song and Ermon, 2020]
Yang Song and Stefano Ermon.
Improved techniques for training score-based generative models.
In Adv. Neural Inform. Process. Syst., 2020.

[Song et al., 2021]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential
In Int. Conf. Learn. Represent., 2021.

[Sun et al., 2022]
Cheng Sun, Min Sun, and Hwann-Tzong Chen.
Direct voxel grid optimization: Super-fast convergence for radiance
fields reconstruction.
In IEEE Conf. Comput. Vis. Pattern Recog., 2022.

[Sun et al., 2018]
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan
Xue, Joshua B Tenenbaum, and William T Freeman.
Pix3D: Dataset and methods for single-image 3D shape modeling.
In IEEE Conf. Comput. Vis. Pattern Recog., 2018.

[Vincent, 2011]
Pascal Vincent.
A connection between score matching and denoising autoencoders.
Neural computation, 2011.

[Wang et al., 2021]
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping
Neus: Learning neural implicit surfaces by volume rendering for
multi-view reconstruction.
arXiv preprint arXiv:2106.10689, 2021.

[Wu et al., 2016]
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.
Learning a probabilistic latent space of object shapes via 3D
generative-adversarial modeling.
In Adv. Neural Inform. Process. Syst., 2016.

[Wu et al., 2015]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang,
and Jianxiong Xiao.
3D ShapeNets: A deep representation for volumetric shapes.
In IEEE Conf. Comput. Vis. Pattern Recog., 2015.

[Yang et al., 2019]
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath
PointFlow: 3D point cloud generation with continuous normalizing
In Int. Conf. Comput. Vis., 2019.

[Yariv et al., 2021]
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.
Volume rendering of neural implicit surfaces.
In Adv. Neural Inform. Process. Syst., 2021.

[Yu et al., 2021]
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and
Angjoo Kanazawa.
Plenoxels: Radiance fields without neural networks.
arXiv preprint arXiv:2112.05131, 2021.

[Yu et al., 2015]
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong
LSUN: Construction of a large-scale image dataset using deep
learning with humans in the loop.
arXiv preprint arXiv:1506.03365, 2015.

[Zhang et al., 2020]
Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba,
and Sanja Fidler.
Image GANs meet differentiable rendering for inverse graphics and
interpretable 3D neural rendering.
arXiv preprint arXiv:2010.09125, 2020.

[Zhao et al., 2022]
Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G Schwing,
and Alex Colburn.
Generative multiplane images: Making a 2D GAN 3D-Aware.
In Eur. Conf. Comput. Vis., 2022.

* In sec:score_perspective, we provide diffusion models from a score-based perspective following Karras et al., 2022.

* In sec:more_results, we provide additional experiments on our approach, including additional ablation study, qualitative results, and video results.

* In sec:impl_details, we document implementation details.