# Implementing a Hybrid Quantum-Classical Neural Network by Utilizing a
Variational Quantum Circuit for Detection of Dementia

Ryan Kim  Thomas Jefferson High School for Science and Technology
Chantilly, USA
<EMAIL_ADDRESS>
###### Abstract

Magnetic resonance imaging (MRI) is a common technique to scan brains for
strokes, tumors, and other abnormalities that cause forms of dementia.
However, correctly diagnosing forms of dementia from MRIs is difficult, as
nearly 1 in 3 patients with Alzheimer’s were misdiagnosed in 2019, an issue
neural networks can rectify. The performance of these neural networks have
been shown to be improved by applying quantum algorithms. This proposed novel
neural network architecture uses a fully-connected (FC) layer, which reduces
the number of features to obtain an expectation value by implementing a
variational quantum circuit (VQC). This study found that the proposed hybrid
quantum-classical convolutional neural network (QCCNN) provided 97.5% and
95.1% testing and validation accuracies, respectively, which was considerably
higher than the classical neural network (CNN) testing and validation
accuracies of 91.5% and 89.2%. Additionally, using a testing set of 100 normal
and 100 dementia MRI images, the QCCNN detected normal and demented images
correctly 95% and 98% of the time, compared to the CNN accuracies of 89% and
91%. With hospitals like Massachusetts General Hospital beginning to adopt
machine learning applications for biomedical image detection, this proposed
architecture would approve accuracies and potentially save more lives.
Furthermore, the proposed architecture is generally flexible, and can be used
for transfer-learning tasks, saving time and resources.

###### Index Terms:

quantum, machine learning, dementia, variational quantum circuit

##  I Introduction

With the field of quantum physics growing rapidly, more and more applications
of quantum computing are entering the computer science field [1]. Quantum
computing offers several benefits over classical computing, which include
faster computations, a broader variety of problems to solve, and better
performance for machine learning tasks [2] [3]. The properties of encoding
information through superposition and quantum entanglement are not present in
classical computing and could provide new ways to solve specific tasks if
harnessed correctly. However, current available quantum computers have limited
qubits, which represent the superposition between two quantum states, and are
prone to noise, which is intrinsic quantum error. The current state of quantum
computing is often labeled the _Noisy Intermediate Scale Quantum_ (NISQ) era
[4].

Due to the advantages of quantum computing such as using significantly less
features than classical neural networks, technologies such as the variational
quantum circuit have been developed [5]. Variational quantum circuits are a
relatively new technology, invented by Peruzzo et al. 2014 [6]. More
fundamentally the variational quantum eigensolver, it has many applications in
quantum machine learning, quantum chemistry, optimization, and other fields
[7] [8]. In chemistry, it solves for the ground state energy of the
Hamiltonian, which gives lots of critical information about a system [9]. Many
have applied variational quantum circuits to machine learning, as it has been
shown to offer several benefits compared to its classical counterparts [10]
[11] [12] [13] [14] [15].

This novel hybrid quantum-classical neural network was tested utilizing an MRI
dataset of brains with and without dementia, which holds relevance to this
day. Dementia is a growing issue in the world, as over 55 million people in
the world suffer from it [16]. Dementia is the deterioration of cognitive
function and can be caused by a variety of factors, such as head trauma,
alcohol, Parkison’s, Alzheimer’s, and more. Magnetic resonance imaging (MRI)
is one of the most effective ways of diagnosing dementia, but identifying
whether a patient has a certain condition is difficult because the images are
complex and require subjective deduction [17][18]. One of the new ways to
detect a certain condition through MRI images is to use neural networks, which
can be much more accurate and objective than humans [19]. My research attempts
to enhance existing machine learning methods by applying a variational quantum
circuit layer to a pre-existing network, in hopes of improving the accuracy of
the network. While others have started to take advantage of quantum
applications in machine learning, an original, angle-encoded ansatz that can
be trained has not been applied to the binary classification of dementia in
MRIs.

##  II Materials and Methods

###  II-A Description of Dataset

Figure 1: Generalized variational quantum circuit, cost function, and
optimization

The image dataset used in this study was produced and published publicly by
Sarvesh Dubey on Kaggle [20]. The dataset consists of both demented and non-
demented MRI images of brains separated into four different classes
representing different severities of dementia: not demented, mildly demented,
moderately demented, and very demented. Overall, the total number of images
was approximately 6400.

Examples of images of the dataset can be seen in Figure 2, where a normal and
demented brain can be seen on the left and right, respectively. One of the
major differences between normal and demented brains is that the lateral
ventricle is expanded abnormally, distorting the surrounding parts of the
brain [21].

|
---|---
(a) | (b)
Figure 2: (a) Normal axial head of caudate and thalamus brain scan (b)
Demented axial head of caudate and thalamus brain scan

Since this study focuses on the binary classification between demented and
non-demented images, I chose to use the 3199 non-demented and 2239 very
demented images for my dataset [20].

###  II-B Preprocessing of Dataset

The two pools of images were separated into different folders, and labeled 1
through the total number of images in the folder. By keeping the labels
consistent within the folder, it becomes easy to iterate over the dataset to
use for the model.

Additionally, images were resized to 250 x 250 pixels and the dataset array
was converted into a Pytorch tensor.

###  II-C Practicality and Bias of Dataset

While the image dataset used in the study is sufficient in size, much
information is left out about the information of the human participants.
Potential biases could have arrived with a lack of racial or gender diversity,
both of which contribute to subtle differences within the brain matter [22]
[23]. Such biases cause the classification of the machine learning model to be
biased, which is a major issue in machine learning [24].

However, differences within the white brain matter, such as atrophies and
other abnormalities can detected with the given dataset. Additionally, the
main goal of study is to compare the advantage that hybrid quantum-classical
neural networks have other their classical counterparts.

###  II-D Classical Neural Network

A classical neural network consists of many different layers, used to extract
features with mathematical functions. In deep-learning, feed-forward neural
networks tend to be one of the most widely used models for object detection. A
generalized layer of one feed-forward neural network can be represented by
[25]:

$L_{i}:x_{i}\xrightarrow{}y_{i}=\psi(Wx_{i}+b)$ (1)

where $L_{i}$ is the specified layer, $W$ is the weights, $x_{i}$ is the input
feature vector, $y_{i}$ is the output feature vector, b is the bias, and
$\psi$ is a nonlinear activation function. The biases and weights are the
values that will be optimized during the training process.

The selected model used for feature extraction was the ResNet18, which is a
sequential convolutional neural network, that includes 18 layers in total.
ResNet18 consists of convolution, batch normalization, ReLU, and max pooling
layers.

###  II-E Variational Quantum Circuit

####  II-E1 Properties and Gates

A variational quantum circuit takes advantage of certain quantum properties
such as superposition, entanglement, and quantum gates to encode and
manipulate data. A generalized version of the variational quantum circuit can
be seen in Figure 1. Superposition is fundamental in quantum mechanics, as it
is a system in multiple states, with each state having a corresponding
probability, until a measurement occurs. Superposition of an arbitrary state
in the z-basis can be represented by:

$\ket{\psi}=a\ket{0}+b\ket{1}$ (2)

where $\ket{\psi}$ is the given state, {$\ket{0}$, $\ket{1}$} are the
orthogonal Z-basis vectors, a is the corresponding probability amplitude of
$\ket{0}$, and b is the corresponding probability ampltude of $\ket{1}$.

Entanglement is another foundational principle of quantum mechanics, which
states that once measuring a state of one particle of an entangled pair, the
state of the other particle is known, no matter how far the particles are from
each other. A simple representation of an entangled system in a Stern-Gerlach
experiment is:

$\ket{\psi}=\frac{1}{\sqrt{2}}(\ket{0}_{1}\ket{0}_{2}+\ket{1}_{1}\ket{1}_{2})$
(3)

where {$\ket{0}_{1}$, $\ket{1}_{1}$} represent basis state vectors of the
first particle and {$\ket{0}_{2}$ ,$\ket{1}_{2}$} represent basis state
vectors of the second particle.

In the Z-basis, the spin-up and down vectors, $\ket{0}$ and $\ket{1}$, can be
represented in matrix form as

$\ket{0}=\begin{pmatrix}1\\\ 0\end{pmatrix}$ (4)

$\ket{1}=\begin{pmatrix}0\\\ 1\end{pmatrix}$ (5)

In quantum computing, there are gates that perform operations on qubits, which
store information. A single qubit in the $\ket{0}$ state can be represented
visually as seen in Figure 3.

Figure 3: Bloch sphere represented of a single qubit in the $\ket{0}$ state

The most fundamental gate is the Hadamard (H) gate, which brings a single
qubit into a superposition state. It can be represented in the matrix form as
[26]

$H=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\\ 1&-1\end{pmatrix}$ (6)

A visual representation of the Hadamard after applying the gate to Figure 3
can be seen in Figure 4.

Figure 4: Bloch sphere represented after applying the Hadamard gate to the
$\ket{0}$ state

Figure 5: Variational quantum circuit with embedding layer, variational layer,
and measurement layer reducing 512 features to 4 features

Another important gate is the controlled-not (CNOT) gate, which checks if the
control qubit has the state $\ket{1}$ and then flips the target qubit from
$\ket{0}$ to $\ket{1}$ and vice versa. Its matrix representation is:

$CNOT=\begin{pmatrix}1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&1\\\ 0&0&1&0\end{pmatrix}$
(7)

The final gate used in this study is the rotation-y (RY) gate, which rotates a
given qubit state a given number of radians around the complex y-axis. It can
be represented as

$RY=\begin{pmatrix}cos(\frac{\theta}{2})&-sin(\frac{\theta}{2})\\\
sin(\frac{\theta}{2})&cos(\frac{\theta}{2})\\\ \end{pmatrix}$ (8)

where $\theta$ is the angle of rotation desired and Figure 6 is the Bloch
sphere representation.

Figure 6: Bloch sphere represented after applying the Rotation-Y gate to the
$\ket{0}$ state

In general, variational quantum circuits consist of three major layers:
embedding, variational, and measurement layer.

####  II-E2 Encoding

The embedding layer will map a vector from the classical space to the quantum
Hilbert space [27], and is parameterized by $\theta$, which is defined as

$\theta_{i,j}^{(k)}=tanh(I_{i,j})*\frac{\pi}{2}$ (9)

where $I_{i,j}$ is the intensity of a pixel specified at ($i$, $j$) of a given
convoluted input image, and k is the specified qubit number. In the four-qubit
variational circuit used in this study, $k\in{1,2,3,4}$ and the parameters at
a given pixel specified at ($i$, $j$) are $\theta_{i,j}^{(k)}$.

The embedding layer $E$ consists of a Hadamard gate, and a Rotation-Y gate,
which are applied to a qubit in the $\ket{0}$ state.

$E^{(k)}=R_{y}(\theta_{i,j}^{(k)})H$ (10)

$E:x_{k}\xrightarrow{}\ket{x}_{k}=E^{(k)}\ket{0}$ (11)

A variational layer, $L$, can be defined as [25]:

$L:\ket{\psi(x)}\xrightarrow{}\ket{\psi(y)}=U(w)\ket{\psi(x)}$ (12)
$\ket{\psi(x)}=\bigotimes_{k=1}^{4}\ket{x}_{k}$ (13)

where $\ket{\psi(x)}$ is the four-qubit input state, $\ket{\psi(y)}$ is the
output state, $w$ is a vector that consists of all the theta parameters, and U
is a unitary gate matrix that represents the matrix product of the H, CNOT,
and RY gates.

In machine learning, a common technique for classification is to non-linearly
map to a higher dimension to solve for the hyperplane, which separates the
data [10]. In terms of the variational quantum circuit, it can be represented
as [10] [25]

$U(w)=T\bigotimes_{k=1}^{4}R_{y}(w_{k})\ket{\psi(x)}$ (14)

where $k$ is the specified qubit, $w_{k}$ is the specified variational
parameter of vector $w$, and $\ket{x}$ is the state from the last layer.

T in equation 14 represents the unitary entanglement operation, represented by

$T=C_{1,2}C_{3,4}C_{2,3}$ (15)

where the first $C$ subscript represents the control qubit and the second
subscript represents the target qubit for CNOT gates.

The concatenation of these variational layers of depth d can be expressed as:

$Q=U_{d}\circ\cdots U_{2}\circ U_{1}$ (16)

The final layer produces four classical outputs by measuring the quantum
states $\ket{\psi_{k}(y)}$ from the four qubits and computes the expectation
value $y$. The measurement operator $M$ can be expressed as:

$M:\ket{\psi(x)}\xrightarrow{}y=|\braket{\psi(x)}{\psi(y)}|^{2}$ (17)

Overall, the variational quantum circuit V can be expressed as:

$V=M\circ Q\circ E$ (18)

where E is the embedding layer, Q is the variational circuit, and M is the
measurement layer in the Z-basis.

TABLE I:  Hyperparameters for the hybrid and classical models
---
Epochs | Learning Rate | Loss Function | Optimizer | Batch Size | Activation Function
20 | $10^{-4}$ | Cross-Entropy | Adam | 32 | RELu

###  II-F Creating a Hybrid Quantum-Classical Neural Network

The ResNet-18 neural network model was used the extract features from the
input images, and the variational quantum circuit significantly reduces the
number of features, which catalyzes the network towards classification.
According to Figure 7, 512 features are extracted from the initial ResNet-18
average pooling layer and then fed into the variational quantum circuit, which
is depicted in Figure 5.

Figure 7: Architecture of the Hybrid Quantum-Classical Neural Network

After the variational quantum circuit computes the given quantum logic gates
and applies a z-measurement on each qubit, the four dimensional vector is sent
to the classical fully-connected (FC) layer, to which it is converted into two
values: an expectation value for a brain with dementia and an expectation
value for a brain without dementia.

Representing the quantum-classical neural network mathematically as $QCNN$:

$QCNN=L_{4\xrightarrow{}2}\cdot Q_{512\xrightarrow{}4}\cdot L_{512}$ (19)

where $L_{512}$ represents the ResNet-18 Layers that extract 512 features,
$Q_{512\xrightarrow{}4}$ is the variational quantum circuit that reduces the
512 features to 4, and ${L_{4\xrightarrow{}2}}$ is the fully-connected layer.

###  II-G Defining Hyperparameters

In machine learning, hyperparameters define the learning process before
training the model. Hyperparameters cannot be changed during the training
process of the neural network model [28] [29].

The first hyperparameter to be defined as the number of epochs. Epochs are
defined as the number of times the whole dataset is passed through the model.
In this study, the number of epochs is set to 20 to ensure that the accuracies
and losses are able to converge.

The next hyperparameter to be defined as the learning rate. In machine
learning, the learning rate defines how much the model changes its weights
with respect to the loss gradient function. Generally, the higher the learning
rate, the faster the loss function converges and the lower the learning rate,
the more intricately and accurately the loss function converges [30].

Another hyperparameter to be defined is the batch size, which is the number of
images to be sent through the model in one iteration. The more images used in
a batch, the slower the model is trained [31]. However, like the learning
rate, time is exchanged for potentially higher accuracies and lower losses. In
this study, the chosen batch size was 32 images.

The layers of a neural network are made up of neurons, which can be activated,
from which information can be sent to the next layer. The activation function,
which determines whether or not a neuron activates, is another important
hyperparameter. In this study, the chosen nonlinear activation function is the
rectified linear unit (ReLU), which can be defined as [25]:

$ReLU(x)=max(0,x)$ (20)

The loss function is another important hyperparameter that quantifies how far
the model prediction is from the actual value. The chosen loss function for
this study is the cross-entropy loss function for binary classification, which
can be expressed as [25]:

$CL=-\sum_{n=1}^{2}y_{n}log(P_{n})$ (21)

where $y_{n}$ is the output value computed by the model and $P$ is the overall
probability of the $n$th class.

The last hyperparameter is the optimizer, which is an algorithm to minimize
the loss function. The selected optimizer for this study is the Adam optimizer
[32], which can compute large amounts of data effectively.

All of the hyperparameters are represented in Table I.

###  II-H Performance Metrics

To compare the ResNet-18 and the proposed model, the accuracy, recall,
precision, and F1-score metrics are used [33]:

$accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (22) $recall=\frac{TP}{TP+FN}$ (23)
$precision=\frac{TP}{TP+FP}$ (24)
${F_{1}}=\frac{2*recall*precision}{recall+precision}$ (25)

where $TP$ represents the true positives, which are data values correctly
given a positive label. $TN$ represents the true negatives, which are data
values correctly given a negative label. $FP$ represents the false positives,
which are data values incorrectly given a positive label. $FN$ represents the
false negatives, which are data values incorrectly given a negative label.

For calculating the testing metrics, the same 100 unused non-demented and 100
unused demented brain MRIs were used. Conditions were kept the same to prevent
confounding variables to influence the comparison of performance metrics
between the two models.

##  III Results And Discussion

###  III-A Training

During the training phase, the accuracy and loss graphs for the classical and
hybrid neural networks are shown in Figure 8.

Figure 8: Training accuracy and loss with respect to epoch number for both
hybrid and classical models

In the top graph of Figure 8, the hybrid model consistently has a higher
accuracy than the classical model, with the difference of accuracy during
convergence being approximately 0.065. Both models drastically improve in
accuracy between epochs 1-3. In the bottom graph of Figure 8, the hybrid model
consistently has less loss than the classical model, with the difference of
loss during convergence being approximately 0.2. Both models initially start
with large values of loss and then drastically decrease in loss between epochs
1-5.

After both models reached convergence, the training accuracy was approximately
0.976 for the hybrid model and 0.915 for the classical model. The
corresponding convergence training losses were 0.148 and 0.316 for the hybrid
and classical models, respectively.

TABLE II:  Comparison of classical to hybrid model
---
Model | Training Accuracy | Validation Accuracy | Training Loss | Validation Loss | Testing Accuracy | Recall | Precision | F1-Score
Hybrid | 0.975 | 0.951 | 0.148 | 0.380 | 0.965 | 0.980 | 0.951 | 0.966
Classical | 0.915 | 0.892 | 0.316 | 0.557 | 0.900 | 0.91 | 0.892 | 0.901

###  III-B Validation

In the top graph of Figure 9, the hybrid model once again consistently has a
higher accuracy than the classical model, with the difference of accuracy
during convergence being approximately 0.06. Both models drastically improve
in accuracy between epochs 1-4. In the bottom graph of Figure 9, the hybrid
model consistently has less loss than the classical model, with the difference
of loss during convergence being approximately 0.18. Both models initially
start with large values of loss and then drastically decrease in loss between
epochs 1-5, but less drastically than the training losses.

Figure 9: Validation accuracy with respect to epoch number for both hybrid and
classical models

###  III-C Testing

After both models reached convergence, the training accuracy was approximately
0.976 for the hybrid model and 0.915 for the classical model. The
corresponding convergence training losses were 0.148 and 0.316 for the hybrid
and classical models, respectively.

For the hybrid model testing results shown in the top matrix of Figure 10, out
of the 100 normal images tested, 95 were correctly classified as normal and 5
were incorrectly classified as demented. Additionally, out of the 100 demented
images tested, 98 were correctly classified as demented and 2 were incorrectly
classified as normal. In the bottom matrix of Figure 10, which included the
classical model testing results, 89 normal images were correctly classified as
normal and 11 normal images were incorrectly classified as demented. The
classical model also correctly classified 91 demented images and incorrectly
classified 9 demented images as normal.

As shown in Table II, the hybrid model performed better in every metric
calculated: accuracy, loss, recall, precision and F1-score. Since it has a
higher accuracy and F1-score, it is better at correctly predicting the right
label for an image across both classes, demented and non-demented. Since the
hybrid model has a better recall and precision than the classical model, it is
both better at predicting correct labels out of the true positives and has a
better accuracy for predicting positive labels out of the dataset.

Figure 10: Confusion matrix for demented and non-demented labels for the
hybrid and classical model

###  III-D Statistical Analysis

To confirm the validity of the model improvement between the hybrid model and
the classic model, the McNemar’s test of significance was used. The McNemar’s
test is often used in machine learning to compare the performance between two
different binary classification models on the same dataset [34]. The p-value
calculated indicated that the results of the study were significant.

Figure 11: IBM Quantum Composer representation of proposed variational quantum
circuit architecture with a depth of 3

##  IV Conclusion

This study found that by implementing a variational quantum circuit layer to
the ResNet-18 neural network, the training accuracy, validation accuracy,
training loss, validation loss, testing accuracy, recall, precision, and
F1-score is improved significantly. The proposed hybrid model provides better
predictions than the original ResNet-18 model due to its inherent quantum
nature, which displays another example of quantum supremacy, where a quantum
application provides significant speedups or benefits over a classical one
[35]. This framework of a hybrid model can be applied to other binary
classification systems, as ResNet-18 is generally flexible and the variational
quantum circuit can reduce a large number of parameters. A potential extension
of this application could be to connect it to a video stream, and then feed in
the camera frames for a real-time detection of a desired object. This hybrid
model has potential to provide more accurate results for dementia from MRI
diagnosis and improve patient outcomes. In the future, more layers of the
proposed model can be parameterized by a quantum circuit to improve accuracy
and the model can be trained on other datasets for binary classification
tasks. Another improvement that could be made would be to create a separate
model for each group of MRI participant (e.g. race, gender, etc.) in order to
decrease the bias of the whole model.

Additionally, if ran on a remote quantum computer with enough qubits, this
proposed model could potentially run faster and require less feature space
than its classical counterpart. A proposed VQC of variational depth 3 and
fixed angle of rotation $\pi/2$ is shown in Figure 11, assembled using the IBM
Quantum Composer.

##  V Acknowledgment

This research was done independently by the author, who is affiliated with the
Thomas Jefferson High School for Science and Technology. No funding or
mentoring was associated with this research.

## References

  * [1] Soares-Pinto, D. O. (2023). Quantum information science: From foundations to new technologies. Physica B: Condensed Matter, 653, 414510. https://doi.org/10.1016/j.physb.2022.414510
  * [2] Golestan, S., Habibi, M.R., Mousazadeh Mousavi, S.Y., Guerrero, J.M., & Vasquez, J.C. (2023). Quantum computation in power systems: An overview of recent advances. Energy Reports, 9, 584-596. https://doi.org/10.1016/j.egyr.2022.11.185
  * [3] Yan, F., Iliyasu, A., & Jiang, Z. (2014). Quantum computation-based image representation, processing operations and their applications. Entropy, 16(10), 5290-5338. https://doi.org/10.3390/e16105290
  * [4] Preskill, J. (2018). Quantum computing in the NISQ era and beyond. Quantum, 2, 79. https://doi.org/10.22331/q-2018-08-06-79
  * [5] Du, Y., Hsieh, M.-H., Liu, T., & Tao, D. (2020). Expressive power of parametrized quantum circuits. Physical Review Research, 2(3). https://doi.org/10.1103/PhysRevResearch.2.03312
  * [6] Peruzzo, A., McClean, J., Shadbolt, P., Yung, M.-H., Zhou, X.-Q., Love, P. J., Aspuru-Guzik, A., & O’Brien, J. L. (2014). A variational eigenvalue solver on a photonic quantum processor. Nature Communications, 5(1). https://doi.org/10.1038/ncomms5213
  * [7] Tilly, J., Chen, H., Cao, S., Picozzi, D., Setia, K., Li, Y., Grant, E., Wossnig, L., Rungger, I., Booth, G. H., & Tennyson, J. (2022). The variational quantum eigensolver: A review of methods and best practices. Physics Reports, 986, 1-128. https://doi.org/10.1016/j.physrep.2022.08.003
  * [8] Martín-Guerrero, J. D., & Lamata, L. (2022). Quantum machine learning: A tutorial. Neurocomputing, 470, 457-461. https://doi.org/10.1016/j.neucom.2021.02.102
  * [9] Cerezo, M., Sharma, K., Arrasmith, A., & Coles, P. J. (2022). Variational quantum state eigensolver. Npj Quantum Information, 8(1). https://doi.org/10.1038/s41534-022-00611-6
  * [10] Maheshwari, D., Sierra-Sosa, D., & Garcia-Zapirain, B. (2022). Variational quantum classifier for binary classification: Real vs synthetic dataset. IEEE Access, 10, 3705-3715. https://doi.org/10.1109/ACCESS.2021.3139323
  * [11] Jäger, J., & Krems, R. V. (2023). Universal expressiveness of variational quantum classifiers and quantum kernels for support vector machines. Nature Communications, 14(1). https://doi.org/10.1038/s41467-023-36144-5
  * [12] Chen, S. Y.-C., Yang, C.-H. H., Qi, J., Chen, P.-Y., Ma, X., & Goan, H.-S. (2019). Variational quantum circuits for deep reinforcement learning. ArXiv. https://doi.org/10.48550/arXiv.1907.00397
  * [13] Khairy, S., Shaydulin, R., Cincio, L., Alexeev, Y., & Balaprakash, P. (2020). Learning to optimize variational quantum circuits to solve combinatorial problems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03), 2367-2375. https://doi.org/10.1609/aaai.v34i03.5616
  * [14] Zhu, D., Linke, N. M., Benedetti, M., Landsman, K. A., Nguyen, N. H., Alderete, C. H., Perdomo-Ortiz, A., Korda, N., Garfoot, A., Brecque, C., Egan, L., Perdomo, O., & Monroe, C. (2019). Training of quantum circuits on a hybrid quantum computer. Science Advances, 5(10). https://doi.org/10.1126/sciadv.aaw9918
  * [15] Farhi, E., & Neven, H. (2018). Classification with quantum neural networks on near term processors. https://doi.org/10.48550/arXiv.1802.06002
  * [16] Dementia. (2023, March 15). World Health Organization. Retrieved April 7, 2023, from https://www.who.int/news-room/fact-sheets/detail/dementia
  * [17] Farisco, M., & Petrini, C. (2014). Misdiagnosis as an ethical and scientific challenge. Annali Dell’Istituto Superiore Di Sanità, 50(03). https://doi.org/10.4415/ANN_14_03_05
  * [18] Bhuva, A. N., Bai, W., Lau, C., Davies, R. H., Ye, Y., Bulluck, H., McAlindon, E., Culotta, V., Swoboda, P. P., Captur, G., Treibel, T. A., Augusto, J. B., Knott, K. D., Seraphim, A., Cole, G. D., Petersen, S. E., Edwards, N. C., Greenwood, J. P., Bucciarelli-ducci, C., . . . Manisty, C. H. (2019). A multicenter, scan-rescan, human and machine learning CMR study to test generalizability and precision in imaging biomarker analysis. Circulation: Cardiovascular Imaging, 12(10). https://doi.org/10.1161/CIRCIMAGING.119.009214
  * [19] El-Sappagh, S., Alonso, J. M., Islam, S. M. R., Sultan, A. M., & Kwak, K. S. (2021). A multilayer multimodal detection and prediction model based on explainable artificial intelligence for alzheimer’s disease. Scientific Reports, 11(1). https://doi.org/10.1038/s41598-021-82098-3
  * [20] Dubey, S. (n.d.). Alzheimer’s Dataset ( 4 class of Images). Kaggle. Retrieved April 7, 2023, from https://www.kaggle.com/datasets/tourist55/alzheimers-dataset-4-class- of-images
  * [21] Carmichael, O. T., Kuller, L. H., Lopez, O. L., Thompson, P. M., Dutton, R. A., Lu, A., Lee, S. E., Lee, J. Y., Aizenstein, H. J., Meltzer, C. C., Liu, Y., Toga, A. W., & Becker, J. T. (2007). Ventricular volume and dementia progression in the cardiovascular health study. Neurobiology of Aging, 28(3), 389-397. https://doi.org/10.1016%2Fj.neurobiolaging.2006.01.006
  * [22] Studholme, C., Kroenke, C. D., & Dighe, M. (2020). Motion corrected MRI differentiates male and female human brain growth trajectories from mid-gestation. Nature Communications, 11(1). https://doi.org/10.1038/s41467-020-16763-y
  * [23] Liu, G., Allen, B., Lopez, O., Aizenstein, H., Boudreau, R., Newman, A., Yaffe, K., Kritchevsky, S., Launer, L., Satterfield, S., Simonsick, E., & Rosano, C. (2015). Racial differences in gray matter integrity by diffusion tensor in black and white octogenarians. Current Alzheimer Research, 12(7), 648-654. https://doi.org/10.2174
  * [24] Alberto, I. R. I., Alberto, N. R. I., Altinel, Y., Blacker, S., Binotti, W. W., Celi, L. A., Chua, T., Fiske, A., Griffin, M., Karaca, G., Mokolo, N., Naawu, D. K. N., Patscheider, J., Petushkov, A., Quion, J., Senteio, C., Taisbak, S., Tırnova, İ., Tokashiki, H., . . . Yap, K. (2023). Who does the fairness in health AI community represent? medRxiv. https://doi.org/10.1101/2023.03.20.23287471
  * [25] Mari, A., Bromley, T. R., Izaac, J., Schuld, M., & Killoran, N. (2020). Transfer learning in hybrid classical-quantum neural networks. Quantum, 4, 340. https://doi.org/10.22331/q-2020-10-09-340
  * [26] Montanaro, A. (2016). Quantum algorithms: An overview. Npj Quantum Information, 2(1). https://doi.org/10.1038/npjqi.2015.23
  * [27] Schuld, M., & Killoran, N. (2019). Quantum machine learning in feature hilbert spaces. Physical Review Letters, 122(4). https://doi.org/10.1103/PhysRevLett.122.040504
  * [28] Yu, T., & Zhu, H. (2020). Hyper-Parameter Optimization: A Review of Algorithms and Applications. ArXiv. https://doi.org/10.48550/arXiv.2003.05689
  * [29] Yang, L., & Shami, A. (2020). On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415, 295-316. https://doi.org/10.1016/j.neucom.2020.07.061
  * [30] Dong, X.-M., & Chen, D.-R. (2009). Learning rates of gradient descent algorithm for classification. Journal of Computational and Applied Mathematics, 224(1), 182-192. https://doi.org/10.1016/j.cam.2008.04.022
  * [31] Nakamura, K., Soatto, S., & Hong, B.-W. (2022). Stochastic batch size for adaptive regularization in deep network optimization. Pattern Recognition, 129, 108776. https://doi.org/10.1016/j.patcog.2022.108776
  * [32] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. ArXiv. https://doi.org/10.48550/arXiv.1412.6980
  * [33] Classification: Precision and recall. (n.d.). Retrieved April 7, 2023, from https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall
  * [34] Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895-1923. https://doi.org/10.1162/089976698300017197
  * [35] Harrow, A. W., & Montanaro, A. (2017). Quantum computational supremacy. Nature, 549(7671), 203-209. https://doi.org/10.1038/nature23458


# Zero-Shot Text-Guided Object Generation with Dream Fields

Ajay Jain1,2∗ Ben Mildenhall2 Jonathan T. Barron2 Pieter Abbeel1 Ben Poole2

###### Abstract

We combine neural rendering with multi-modal image and text representations to
synthesize diverse 3D objects solely from natural language descriptions. Our
method, Dream Fields, can generate the geometry and color of a wide range of
objects without 3D supervision. Due to the scarcity of diverse, captioned 3D
data, prior methods only generate objects from a handful of categories, such
as ShapeNet. Instead, we guide generation with image-text models pre-trained
on large datasets of captioned images from the web. Our method optimizes a
Neural Radiance Field from many camera views so that rendered images score
highly with a target caption according to a pre-trained CLIP model. To improve
fidelity and visual quality, we introduce simple geometric priors, including
sparsity-inducing transmittance regularization, scene bounds, and new MLP
architectures. In experiments, Dream Fields produce realistic, multi-view
consistent object geometry and color from a variety of natural language
captions. ††1UC Berkeley, 2Google Research. ††∗Work done at Google.
††Correspondence to<EMAIL_ADDRESS>††Project website and code:
https://ajayj.com/dreamfields

Figure 1: Given a caption, we learn a Dream Field, a continuous volumetric
representation of an object’s geometry and appearance learned with guidance
from a pre-trained model. We optimize the Dream Field by rendering images of
the object from random camera poses that are scored with frozen pre-trained
image and  text encoders trained on web images and alt-text. 2D views share
the same underlying radiance field for consistent geometry. Figure 2: Example
Dream Fields rendered from four perspectives. On the right, we show
transmittance from the final perspective. We create diverse outputs using the
compositionality of language; these captions from MSCOCO describe three flower
arrangements with different properties like context and color.

##  1 Introduction

Detailed 3D object models bring multimedia experiences to life. Games, virtual
reality applications and films are each populated with thousands of object
models, each designed and textured by hand with digital software. While expert
artists can author high-fidelity assets, the process is painstakingly slow and
expensive. Prior work leverages 3D datasets to synthesize shapes in the form
of point clouds, voxel grids, triangle meshes, and implicit functions using
generative models like GANs [57, 4, 21, 65]. These approaches only support a
few object categories due to small labeled 3D shape datasets. But multimedia
applications require a wide variety of content, and need both 3D geometry and
texture.

In this work, we propose Dream Fields, a method to automatically generate
open-set 3D models from natural language prompts. Unlike prior work, our
method does not require any 3D training data, and uses natural language
prompts that are easy to author with an expressive interface for specifying
desired object properties. We demonstrate that the compositionality of
language allows for flexible creative control over shapes, colors and styles.

A Dream Field is a Neural Radiance Field (NeRF) trained to maximize a deep
perceptual metric with respect to both the geometry and color of a scene. NeRF
and other neural 3D representations have recently been successfully applied to
novel view synthesis tasks where ground-truth RGB photos are available. NeRF
is trained to reconstruct images from multiple viewpoints. As the learned
radiance field is shared across viewpoints, NeRF can interpolate between
viewpoints smoothly and consistently. Due to its neural representation, NeRF
can be sampled at high spatial resolutions unlike voxel representations and
point clouds, and are easy to optimize unlike explicit geometric
representations like meshes as it is topology-free.

However, existing photographs are not available when creating novel objects
from descriptions alone. Instead of learning to reconstruct known input
photos, we learn a radiance field such that its renderings have high semantic
similarity with a given text prompt. We extract these semantics with pre-
trained neural image-text retrieval models like CLIP [46], learned from
hundreds of millions of captioned images. As NeRF’s volumetric rendering and
CLIP’s image-text representations are differentiable, we can optimize Dream
Fields end-to-end for each prompt. Figure Zero-Shot Text-Guided Object
Generation with Dream Fields illustrates our method.

In experiments, Dream Fields learn significant artifacts if we naively
optimize the NeRF scene representation with textual supervision without adding
additional geometric constraints (Figure 3). We propose general-purpose priors
and demonstrate that they greatly improve the realism of results. Finally, we
quantitatively evaluate open-set generation performance using a dataset of
diverse object-centric prompts.

Our contributions include:

  * •

Using aligned image and text models to optimize NeRF without 3D shape or
multi-view data,

  * •

Dream Fields, a simple, constrained 3D representation with neural guidance
that supports diverse 3D object generation from captions in zero-shot, and

  * •

Simple geometric priors including transmittance regularization, scene bounds,
and an MLP architecture that together improve fidelity.

Figure 3: Challenges of text-to-3D synthesis: (a) Poor generalization from
limited 3D datasets: Most 3D generative models are learned on datasets of
specific object categories like ShapeNet [7], and won’t generalize to novel
concepts zero-shot. (b) Neural Radiance Fields are too flexible without multi-
view supervision: NeRF learns to represent geometry and texture from scene-
specific multi-view data, so it does not require a diverse dataset of objects.
Yet, when only a source caption is available instead of multi-view images,
NeRF produces significant artifacts (_e.g_., near field occlusions). (c) Dream
Fields: We introduce general geometric priors that retain much of NeRF’s
flexibility while improving realism.

##  2 Related Work

Our work is primarily inspired by DeepDream [34] and other methods for
visualizing the preferred inputs and features of neural networks by optimizing
in image space [39, 37, 36]. These methods enable the generation of
interesting images from a pre-trained neural network without the additional
training of a generative model. Closest to our work is [35], which studies
differentiable image parameterizations in the context of style transfer. Our
work replaces the style and content-based losses from that era with an image-
text loss enabled by progress in contrastive representation learning on image-
text datasets [12, 46, 26, 60]. The use of image-text models enables easy and
flexible control over the style and content of generated imagery through
textual prompt design. We optimize both geometry and color using the
differentiable volumetric rendering and scene representation provided by NeRF,
whereas [35] was restricted to fixed geometry and only optimized texture.
Together these advances enable a fundamentally new capability: open-ended
text-guided generation of object geometry and texture.

Concurrently to Dream Fields, a few early works have used CLIP [46] to
synthesize or manipulate 3D object representations. CLIP-Forge [49] generates
multiple object geometries from text prompts using a CLIP embedding-
conditioned normalizing flow model and geometry-only decoder trained on
ShapeNet categories. Still, CLIP-Forge generalizes poorly outside of ShapeNet
categories and requires ground-truth multi-view images and voxel data.
Text2Shape [9] learns a text-conditional Wasserstein GAN [1, 19] to synthesize
novel voxelized objects, but only supports finite resolution generation of
individual ShapeNet categories. In [10], object geometry is optimized
evolutionarily for high CLIP score from a single view then manually colored.
ClipMatrix [25] edits the vertices and textures of human SMPL models [31] to
create stylized, deformable humanoid meshes. [44] creates an interactive
interface to edit signed-distance fields in localized regions, though they do
not optimize texture or synthesize new shapes. Text-based manipulation of
existing objects is complementary to us.

For images, there has been an explosion of work that leverages CLIP to guide
image generation. Digital artist Ryan Murdock (@advadnoun) used CLIP to guide
learning of the weights of a SIREN network [52], similar to NeRF but without
volume rendering and focused on image generation. Katherine Crowson
(@rivershavewings) combined CLIP with optimization of VQ-GAN codes [16] and
used diffusion models as an image prior [14]. Recent work from Mario
Klingemann (@quasimondo) and [43] have shown how CLIP can be used to guide GAN
models like StyleGAN [27]. Some works have optimized parameters of vector
graphics, suggesting CLIP guidance is highly general [23, 17, 50]. These
methods highlighted the surprising capacity of what image-text models have
learned and their utility for guiding 2D generative processes. Direct text to
image synthesis with generative models has also improved tremendously in
recent years [48, 61], but requires training large generative models on large-
scale datasets, making such methods challenging to directly apply to text to
3D where no such datasets exist.

There is also growing progress on generative models with NeRF-based generators
trained solely from 2D imagery. However, these models are category-specific
and trained on large datasets of mostly forward-facing scenes [5, 38, 51, 20,
66], lacking the flexibility of open-set text-conditional models. Shape-
agnostic priors have been used for 3D reconstruction [2, 58, 64].

##  3 Background

Our method combines Neural Radiance Fields (NeRF) [33] with an image-text loss
from [46]. We begin by discussing these existing methods, and then detail our
improved approach and methodology that enables high quality text to object
generation.

###  3.1 Neural Radiance Fields

NeRF [33] parameterizes a scene’s density and color using a multi-layer
perceptron (MLP) with parameters $\theta$ trained with a photometric loss
relying on multi-view photographs of a scene. In our simplified model, the
NeRF network takes in a 3D position $\mathbf{x}$ and outputs parameters for an
emission-absorption volume rendering model: density
$\sigma_{\theta}({\mathbf{x}})$ and color $\mathbf{c}_{\theta}({\mathbf{x}})$.
Images can be rendered from desired viewpoints by integrating color along an
appropriate ray, $\mathbf{r}(t)$, for each pixel according to the volume
rendering equation:

$\displaystyle\mathbf{C}(r,\theta)=\int_{t_{n}}^{t_{f}}T(\mathbf{r},t)\sigma_{\theta}(\mathbf{r}(t))\mathbf{c}_{\theta}(\mathbf{r}(t))dt,$
(1) $\displaystyle\text{where
}T(\mathbf{r},\theta,t)=\exp\left(-\int_{t_{n}}^{t}\sigma_{\theta}(\mathbf{r}(s))ds\right)\,.$
(2)

The integral $T(\mathbf{r},\theta,t)$ is known as “transmittance” and
describes the probability that light along the ray will not be absorbed when
traveling from $t_{n}$ (the near scene bound) to $t$. In practice [33], these
two integrals are approximated by breaking up the ray into smaller segments
$[t_{i-1},t_{i})$ within which $\sigma$ and $\mathbf{c}$ are assumed to be
roughly constant:

$\displaystyle\mathbf{C}(\mathbf{r},\theta)\approx\sum_{i}T_{i}(1-\exp(-\sigma_{\theta}(\mathbf{r}(t_{i}))\delta_{i}))\mathbf{c}_{\theta}(\mathbf{r}(t_{i}))$
(3) $\displaystyle
T_{i}=\exp\left(-\textstyle\sum_{j<i}\sigma_{\theta}(\mathbf{r}(t_{j}))\delta_{j}\right),\quad\delta_{i}=t_{i}-t_{i-1}\,.$
(4)

For a given setting of MLP parameters $\theta$ and pose $\mathbf{p}{}$, we
determine the appropriate ray for each pixel, compute rendered colors
$C(\mathbf{r},\theta)$ and transmittances, and gather the results to form the
rendered image, $I(\theta,\mathbf{p}{})$ and transmittance
$T(\theta,\mathbf{p}{})$.

In order for the MLP to learn high frequency details more quickly [55], the
input $\mathbf{x}$ is preprocessed by a sinusoidal positional encoding
$\gamma$ before being passed into the network:

$\displaystyle\gamma(\mathbf{x})=\left[\cos(2^{l}\mathbf{x}),\sin(2^{l}\mathbf{x})\right]_{l=0}^{L-1}\,,$
(5)

where $L$ is referred to as the number of “levels” of positional encoding. In
our implementation, we specifically apply the integrated positional encoding
(IPE) proposed in mip-NeRF to combat aliasing artifacts [3] combined with a
random Fourier positional encoding basis [55] with frequency components
sampled according to

$\displaystyle\mathbf{\omega}=2^{u}\mathbf{d},\quad\text{where
}u\sim\mathcal{U}[0,L],\,\,\mathbf{d}\sim\mathcal{U}(\mathbb{S}^{2})\,.$ (6)

###  3.2 Image-text models

Large-scale datasets of images paired with associated text have enabled
training large-scale models that can accurately score whether an image and an
associated caption are likely to correspond [46, 26, 12]. These models consist
of an image encoder $\mathbf{g}$, and text encoder $\mathbf{h}$, that map
images and text into a shared embedding space. Given a sentence $y$ and an
image $\mathbf{I}{}$, these image-text models produce a scalar score:
$\mathbf{g}(\mathbf{I}{})^{\mathrm{T}}\mathbf{h}(\mathbf{y})$ that is high
when the text is a good description of the image, and low when the the image
and text are mismatched. Note that the embeddings $\mathbf{g}(\mathbf{I}{})$
and $\mathbf{h}(\mathbf{y})$ are often normalized, i.e.
$\|\mathbf{g}(\mathbf{I}{})\|=\|\mathbf{h}(\mathbf{y})\|=1$. Training is
typically performed with a symmetric version of the InfoNCE loss [40, 45] that
aims to maximize a variational lower bound on the mutual information between
images and text. Prior work has shown that once trained, the image and text
encoders are useful for a number of downstream tasks [46, 60]. In [48], the
image and text encoders are used to score the correspondence of outputs of a
generative image model to a target caption [48]. We build on this work by
optimizing a volume to produce a high-scoring image, not just reranking.

##  4 Method

In this section, we develop Dream Fields: a zero-shot object synthesis method
given only a natural language caption.

###  4.1 Object representation

Building on the NeRF scene representation (Section 3.1), a Dream Field
optimizes an MLP with parameters $\theta$ that produces outputs
$\sigma_{\theta}(\mathbf{x})$ and $\mathbf{c}_{\theta}(\mathbf{x})$
representing the differential volume density and color of a scene at every 3D
point $\mathbf{x}$. This field expresses object geometry via the density
network. Our object representation is only dependent on 3D coordinates and not
the camera’s viewing direction, as we did not find it beneficial. Given a
camera pose $\mathbf{p}{}$, we can render an image
$\mathbf{I}{}(\theta,\mathbf{p}{})$ and compute the transmittance
$T(\theta,\mathbf{p}{})$ using $N$ segments via (4). Segments are spaced at
roughly equal intervals with random jittering along the ray. The number of
segments, $N$, determines the fidelity of the rendering. In practice, we fix
it to 192 during optimization.

###  4.2 Objective

How can we train a Dream Field to represent a given caption? If we assume that
an object can be described similarly when observed from any perspective, we
can randomly sample poses and try to enforce that the rendered image matches
the caption at all poses. We can implement this idea by using a CLIP network
to measure the match between a caption and image given parameters $\theta$ and
pose $\mathbf{p}{}$:

$\mathcal{L}_{\mathrm{CLIP}}{}(\theta,\text{pose}~{}\mathbf{p}{},\text{caption}~{}y)=-\mathbf{g}(\mathbf{I}{}(\theta,\mathbf{p}{}))^{\mathrm{T}}\mathbf{h}(\mathbf{y})$
(7)

where $\mathbf{g}(\cdot)$ and $\mathbf{h}(\cdot)$ are aligned representations
of image and text semantics, and $\mathbf{I}{}(\theta,\mathbf{p}{})$ is a
rendered image of the scene from camera pose $\mathbf{p}{}$. Each iteration of
training, we sample a pose $\mathbf{p}{}$ from a prior distribution, render
$\mathbf{I}{}$, and minimize $\mathcal{L}_{\mathrm{CLIP}}{}$ with respect to
the parameters of the Dream Field MLP, $\theta$. Equation (7) measures the
similarity of an image and the provided caption in feature space.

We primarily use image and text encoders from CLIP [46], which has a Vision
Transformer image encoder $\mathbf{g}(\cdot)$ [15] and masked transformer text
encoder $\mathbf{h}(\cdot)$ [56] trained contrastively on a large dataset of
400M captioned 2242 images. We also use a baseline Locked Image-Text Tuning
(LiT) ViT B/32 model from [60] trained via the same procedure as CLIP on a
larger dataset of billions of higher-resolution (2882) captioned images. The
LiT training set was collected following a simplified version of the ALIGN web
alt-text dataset collection process [26] and includes noisy captions.

Figure Zero-Shot Text-Guided Object Generation with Dream Fields shows a high-
level overview of our method. DietNeRF [24] proposed a related semantic
consistency regularizer for NeRF based on the idea that “a bulldozer is a
bulldozer from any perspective”. The method computed the similarity of a
rendered and a real image. In contrast, (7) compares rendered images and a
_caption_ , allowing it to be used in zero-shot settings when there are no
object photos.

###  4.3 Challenges with CLIP guidance

Due to their flexibility, Neural Radiance Fields are capable of high-fidelity
novel view synthesis on a tremendous diversity of real-world scenes when
supervised with multi-view consistent images. Their reconstruction loss will
typically learn to remove artifacts like spurious density when sufficiently
many input images are available. However, we find that the NeRF scene
representation is too unconstrained when trained solely with
$\mathcal{L}_{\mathrm{CLIP}}$ (7) alone from a discrete set of viewpoints,
resulting in severe artifacts that satisfy $\mathcal{L}_{\mathrm{CLIP}}$ but
are not visually compatible according to humans (see Figure 3b). NeRF learns
high-frequency and near-field [62] artifacts like partially-transparent
“floating“ regions of density. It also fills the entire camera viewport rather
than generating individual objects. Geometry is unrealistic, though textures
reflect the caption, reminiscent of the artifacts in Deep Dream feature
visualizations [34, 39].

###  4.4 Pose sampling

Image data augmentations such as random crops are commonly used to improve and
regularize image generation in DeepDream[34] and related work. Image
augmentations can only use in-plane 2D transformations. Dream Fields support
3D data augmentations by sampling different camera pose extrinsics at each
training iteration. We uniformly sample camera azimuth in 360∘ around the
scene, so each training iteration sees a different orientation of the object.
As the underlying scene representation is shared, this improves the realism of
object geometry. For example, sampling azimuth in a narrow interval tended to
create flat, billboard geometry.

The camera elevation, focal length and distance from the subject can also be
augmented, but we did not find this necessary. Instead, we use a fixed camera
focal length during optimization that is scaled by $m_{\mathrm{focal}}=1.2$ to
enlarge the object 20%. Rendering cost is constant in the focal length.

Figure 4: To encourage coherent foreground objects, Dream Fields train with 3
types of background augmentations: blurred Gaussian noise, textures and
checkerboards. At test time, we render with a white background. Prompt: “A
sculpture of a rooster.”

###  4.5 Encouraging coherent objects through sparsity

To remove near-field artifacts and spurious density, we regularize the opacity
of Dream Field renderings. Our best results maximize the average transmittance
of rays passing through the volume up to a target constant. Transmittance is
the probability that light along ray $r$ is not absorbed by participating
media when passing between point $t$ along the ray and the near plane at
$t_{n}$ (2). We approximate the total transmittance along the ray as the joint
probability of light passing through $N$ discrete segments of the ray
according to Eq. (4). Then, we define the following transmittance loss:

$\displaystyle\mathcal{L}_{T}$
$\displaystyle=-\min\\!\left(\tau,\operatorname{mean}\\!\left(T(\theta,\mathbf{p}{})\right)\right)$
(8) $\displaystyle\mathcal{L}_{\text{total}}$
$\displaystyle=\mathcal{L}_{\mathrm{CLIP}}+\lambda\mathcal{L}_{T}$ (9)

This encourages a Dream Field to increase average transmittance up to a target
transparency $\tau$. We use $\tau=88\%$ in experiments. $\tau$ is annealed in
from $\tau=40\%$ over 500 iterations to smoothly introduce transparency, which
improves scene geometry and is essential to prevent completely transparent
scenes. Scaling $1-\tau\propto f^{2}/d^{2}$ preserves object cross sectional
area for different focal and object distances.

When the rendering is alpha-composited with a simple white or black background
during training, we find that the average transmittance approaches $\tau$, but
the scene is diffuse as the optimization populates the background. Augmenting
the scene with random background images leads to coherent objects. Dream
Fields use Gaussian noise, checkerboard patterns and the random Fourier
textures from [35] as backgrounds. These are smoothed with a Gaussian blur
with randomly sampled standard deviation. Background augmentations and a
rendering during training are shown in Figure 4.

We qualitatively compare (9) to baseline sparsity regularizers in Figure 5.
Our loss is inspired by the multiplicative opacity gating used by [35].
However, the gated loss has optimization challenges in practice due in part to
its non-convexity. The simplified additive loss is more stable, and both are
significantly sharper than prior approaches for sparsifying Neural Radiance
Fields.

###  4.6 Localizing objects and bounding scene

When Neural Radiance Fields are trained to reconstruct images, scene contents
will align with observations in a consistent fashion, such as the center of
the scene in NeRF’s Realistic Synthetic dataset [33]. Dream Fields can place
density away from the center of the scene while still satisfying the CLIP loss
as natural images in CLIP’s training data will not always be centered. During
training, we maintain an estimate of the 3D object’s origin and shift rays
accordingly. The origin is tracked via an exponential moving average of the
center of mass of rendered density. To prevent objects from drifting too far,
we bound the scene inside a cube by masking the density $\sigma_{\theta}$.

Figure 5: Our transmittance losses and background augmentations are
complementary. Top: Without background augmentations, priors on transmittance
(right three columns) do not remove low-density structures. NeRF’s density
perturbations improve coherence, but cloudy artifacts remain. Bottom: When the
object is alpha composited with random backgrounds during training, CLIP fills
the scene with opaque material to conceal the background. However, gated and
our simplified additive transmittance regularizers both limit the opacity of
the volume successfully and lead to a sharper object. Inset panels depict
transmittance. Prompt: “an illustration of a pumpkin on the vine.”

###  4.7 Neural scene representation architecture

The NeRF network architecture proposed in [33] parameterizes scene density
with a simple 8-layer MLP of constant width, and radiance with an additional
two layers. We use a residual MLP architecture instead that introduces
residual connections around every two dense layers. Within a residual block,
we find it beneficial to introduce Layer Normalization at the beginning and
increase the feature dimension in a bottleneck fashion. Layer Normalization
improves optimization on challenging prompts. To mitigate vanishing gradient
issues in highly transparent scenes, we replace ReLU activations with Swish
[47] and rectify the predicted density $\sigma_{\theta}$ with a softplus
function. Our MLP architecture uses 280K parameters per scene, while NeRF uses
494K parameters.

##  5 Evaluation

We evaluate the consistency of generated objects with their captions and the
importance of scene representation, then show qualitative results and test
whether Dream Fields can generalize compositionally. Ablations analyze
regularizers, CLIP and camera poses. Finally, supplementary materials have
further examples and videos.

###  5.1 Experimental setup

3D reconstruction methods are evaluated by comparing the learned geometry with
a ground-truth reference model, e.g. with Chamfer Distance. Novel view
synthesis techniques like LLFF [32] and NeRF do not have ground truth models,
but compare renderings to pixel-aligned ground truth images from held-out
poses with PSNR or LPIPS, a deep perceptual metric [63].

As we do not have access to diverse captioned 3D models or captioned multi-
view data, Dream Fields are challenging to evaluate with geometric and image
reference-based metrics. Instead, we use the CLIP R-Precision metric [41] from
the text-to-image generation literature to measure how well rendered images
align with the true caption. In the context of text-to-image synthesis,
R-Precision measures the fraction of generated images that a retrieval model
associates with the caption used to generate it. We use a different CLIP model
for learning the Dream Field and computing the evaluation metric. As with NeRF
evaluation, the image is rendered from a held-out pose. Dream Fields are
optimized with cameras at a $30^{\circ}$ angle of elevation and evaluated at
$45^{\circ}$ elevation. For quantitative metrics, we render at resolution 1682
during training as in [24]. For figures, we train with a 50$\%$ higher
resolution of 2522.

We collect an object-centric caption dataset with 153 captions as a subset of
the Common Objects in Context (COCO) dataset [28] (see supplement for
details). Object centric examples are those that have a single bounding box
annotation and are filtered to exclude those captioned with certain phrases
like “extreme close up”. COCO includes 5 captions per image, but only one is
used for generation. Hyperparameters were manually tuned for perceptual
quality on a set of 20-74 distinct captions from the evaluation set, and are
shared across all other scenes. Additional dataset details and hyperparameters
are included in the supplement.

###  5.2 Analyzing retrieval metrics

In the absence of 3D training data, Dream Fields use geometric priors to
constrain generation. To evaluate each proposed technique, we start from a
simplified baseline Neural Radiance Field largely following [33] and introduce
the priors one-by-one. We generate two objects per COCO caption using
different seeds, for a total of 306 objects. Objects are synthesized with 10K
iterations of CLIP ViT B/16 guided optimization of 168$\times$168 rendered
images, bilinearly upsampled to the contrastive model’s input resolution for
computational efficiency. R-Precision is computed with CLIP ViT B/32 [46] and
LiT${}_{\text{uu}}$ B/32 [60] to measure the alignment of generations with the
source caption.

Table 5.2 reports results. The most significant improvements come from
sparsity, scene bounds and architecture. As an oracle, the ground truth images
associated with object-centric COCO captions have high R-Precision. The NeRF
representation converges poorly and introduces aliasing and banding artifacts,
in part from its use of axis-aligned positional encodings.

We instead combine mip-NeRF’s integrated positional encodings with random
Fourier features, which improves qualitative results and removes a bias toward
axis-aligned structures. However, the effect on precision is neutral or
negative. The transmittance loss $\mathcal{L}_{T}$ in combination with
background augmentations significantly improves retrieval precision $+$18% and
$+$15.6%, while the transmittance loss is not sufficient on its own. This is
qualitatively shown in Figure 5. Our MLP architecture with residual
connections, normalization, bottleneck-style feature dimensions and smooth
nonlinearities further improves the R-Precision $+$8% and $+$2%. Bounding the
scene to a cube improves retrieval $+$13% and $+$11%. The additional bounds
explicitly mask density $\sigma$ and concentrate samples along each ray.

We also scale up Dream Fields by optimizing with an image-text model trained
on a larger captioned dataset of 3.6B images from [60]. We use a ViT B/32
model with image and text encoders trained from scratch. This corresponds to
the uu configuration from [60], following the CLIP training procedure to learn
both encoders contrastively. The LiT${}_{\text{uu}}$ ViT encoder used in our
experiments takes higher resolution 2882 images while CLIP is trained with
2242 inputs. Still, LiT${}_{\text{uu}}$ B/32 is more compute-efficient than
CLIP B/16 due to the larger patch size in the first layer.

LiT${}_{\text{uu}}$ does not significantly help R-Precision when optimizing
Dream Fields with low resolution renderings, perhaps because the CLIP B/32
model used for evaluation is trained on the same dataset as the CLIP B/16
model in earlier rows. Optimizing for longer with higher resolution 2522
renderings closes the gap. LiT${}_{\text{uu}}$ improves visual quality and
sharpness (Appendix A), suggesting that improvements in multimodal image-text
models transfer to 3D generation.

| Method | R-Precision $\uparrow$
---|---|---
| CLIP B/32 |  | LiT${}_{\text{uu}}$ B/32
Baseline | COCO GT images | 77.1$\pm$3.4 |  |  75.2$\pm$3.5
Simplified NeRF | 31.4$\pm$2.7 |  | 10.8$\pm$1.8
Positional encoding | \+ mip-NeRF IPE |  29.7$\pm$2.6 |  |  12.4$\pm$1.9
\+ Higher freq. Fourier features |  24.2$\pm$2.5 |  |  10.5$\pm$1.8
Sparsity, augment | \+ random crops |  25.8$\pm$2.5 |  |  10.5$\pm$1.8
\+ transmittance loss |  23.7$\pm$2.4 |  |  7.6$\pm$1.5
\+ background aug. |  44.1$\pm$2.8 |  |  26.1$\pm$2.5
Scene param. | \+ MLP architecture |  52.0$\pm$2.9 |  |  27.8$\pm$2.6
\+ scene bounds |  65.4$\pm$2.7 |  |  38.9$\pm$2.8
\+ track origin |  59.8$\pm$2.8 |  |  34.6$\pm$2.7
Scaling | \+ LiT${}_{\text{uu}}$ ViT B/32 |  59.5$\pm$2.8 |  | –
\+ 20K iterations, 2522 renders |  68.3$\pm$2.7 |  | –

Table 1: When used together, geometric priors improve caption retrieval
precision. We start with a simplified version of the NeRF scene representation
and add in one prior at a time until all are used in conjunction. Captions are
retrieved from rendered images of the generated objects at held-out camera
poses using CLIP’s ViT B/32. Objects are generated with
$\mathcal{L}_{\mathrm{CLIP}}$ guidance from the pre-trained CLIP ViT B/16
except in scaling experiments where we experiment with the higher-resolution
LiT${}_{\text{uu}}$ B/32 model.

Figure 6: Compositional object generation. Dream Fields allow users to express
specific artistic styles via detailed captions. Top two rows: Similar to text-
to-image experiments in [48], we generate objects with the caption “armchair
in the shape of an avocado. armchair imitating avocado.” Bottom: Generations
vary the texture of a single snail. Captions follow the template “a snail made
of baguette. a snail with the texture of baguette” Results are not cherry-
picked.

###  5.3 Compositional generation

In Figure 6, we show non-cherrypicked generations that test the compositional
generalization of Dream Fields to fine-grained variations in captions taken
from the website of [48]. We independently vary the object generated and
stylistic descriptors like shape and materials. DALL-E [48] also had a
remarkable ability to combine concepts in prompts out of distribution, but was
limited to 2D image synthesis. Dream Fields produces compositions of concepts
in 3D, and supports fine-grained variations in prompts across several
categories of objects. Some geometric details are not realistic, however. For
example, generated snails have eye stalks attached to their shell rather than
body, and the generated green vase is blurry.

###  5.4 Model ablations

Method | Loss or parameterization | R-Prec.
---|---|---
No regularizer |  $\mathcal{L}_{\mathrm{CLIP}}$ (7) | 35.3
Perturb $\sigma$ [33] | $\sigma=\operatorname{softplus}(f_{\theta}(\mathbf{x})+\epsilon)$ | 47.7
Beta prior [30] | (10) | 50.3
Gated $T$ [35] | $-\operatorname{mean}(T(\theta,\mathbf{p}{}))\cdot\mathcal{L}_{\mathrm{CLIP}}$ | 34.6
Clipped gated $T$ |  $-\mathcal{L}_{T}\cdot\mathcal{L}_{\mathrm{CLIP}}$ (11) | 62.1
Clipped additive $T$ |  $\mathcal{L}_{\mathrm{CLIP}}+\lambda\mathcal{L}_{T}$ (9) | 62.1
Table 2: Ablating sparsity regularizers. Optimization is done for 10K
iterations at 1682 resolution with LiT${}_{\text{uu}}$ ViT B/32 and background
augmentation, and retrieval uses CLIP ViT B/32. For the purposes of ablation,
we run one seed per caption (153 runs).

Ablating sparsity regularizers While we regularize the mean transmittance,
other sparsity losses are possible. We compare unregularized Dream Fields,
perturbations to the density $\sigma$ [33], regularization with a beta prior
on transmittance [30], multiplicative gating versions of $\mathcal{L}_{T}$ and
our additive $\mathcal{L}_{T}$ regularizer in Figure 5. On real-world scenes,
NeRF added Gaussian noise to network predictions of the density prior to
rectification as a regularizer. This can encourage sharper boundary
definitions as small densities will often be zeroed by the perturbation. The
beta prior from Neural Volumes [30] encourages rays to either pass through the
volume or be completely occluded:

$\mathcal{L}_{\mathrm{total}}^{\text{beta}}=\mathcal{L}_{\mathrm{CLIP}}+\lambda\cdot\operatorname{mean}\\!\left(\log
T(\theta,\mathbf{p}{})+\log(1-T(\theta,\mathbf{p}{}))\right)$ (10)

The multiplicative loss is inspired by the opacity scaling of [35] for feature
visualization. We scale the CLIP loss by a clipped mean transmittance:

$\mathcal{L}_{\text{total}}=\min(\tau,\;\operatorname{mean}(T(\theta,\mathbf{p}{})))\cdot\mathcal{L}_{\mathrm{CLIP}}$
(11)

Table 2 compares the regularizers, showing that density perturbations and the
beta prior improve R-Precision $+$12.4% and $+$15%, respectively. Scenes with
clipped mean transmittance regularization best align with their captions,
$+$26.8% over the baseline. The beta prior can fill scenes with opaque
material even without background augmentations as it encourages both high and
low transmittance. Multiplicative gating works well when clipped to a target
and with background augmentations, but is also non-convex and sensitive to
hyperparameters. Figure 7 shows the effect of varying the target transmittance
$\tau$ with an additive loss.

Figure 7: The target transmittance $\tau$ affects the size of generated
objects. Inset panels depict transmittance. Prompt from Object Centric COCO:
“A cake toped [sic] with white frosting flowers with chocolate centers.”

#### Varying the image-text model

We compare different image and text representations $h(\cdot),g(\cdot)$ used
in $\mathcal{L}_{\mathrm{CLIP}}$ (7) and for retrieval metrics. Table 3 shows
the results. CLIP B/32, B/16 and LiT${}_{\text{uu}}$ B/32 all have high
retrieval precision, indicating they can synthesize objects generally aligned
with the provided captions. CLIP B/32 performs the best, outperforming the
more compute intensive CLIP B/16 model. The architectures differ in the number
of pixels encoded in each token supplied to the Transformer backbone, i.e. the
ViT patch size. A larger patch size may be sufficient due to the low
resolution of renders: 1682 cropped to 1542, then upsampled to CLIP’s input
size of 2242. Qualitatively, training with LiT${}_{\text{uu}}$ B/32 produced
the most detailed geometry and textures, suggesting that open-set evaluation
is challenging.

| Retrieval model R-Precision
---|---
Optimized model | CLIP B/32 | CLIP B/16 | LiT${}_{\text{uu}}$ B/32
COCO GT |  77.1$\pm$3.4 |  79.1$\pm$3.3 |  75.2$\pm$3.5
CLIP B/32 [46] | (86.6$\pm$2.0) |  74.2$\pm$2.5 |  42.8$\pm$2.8
CLIP B/16 [46] |  59.8$\pm$2.8 | (93.5$\pm$1.4) |  35.6$\pm$2.7
LiT${}_{\text{uu}}$ B/32 |  59.5$\pm$2.8 |  66.7$\pm$2.7 | (88.9$\pm$1.8)
Table 3: The aligned image-text representation used to optimize Dream Fields
influences their quantitative validation R-Precision according to a held-out
retrieval model. All contrastive models produce high retrieval precision,
though qualitatively CLIP B/32 produced overly smooth and simplified objects.
We optimize for 10K iterations at 1682 resolution. (Italicized) metrics use
the optimized model at a held-out pose and indicate Dream Fields overfit.

Varying optimized camera poses Each training iteration, Dream Fields samples a
camera pose $\mathbf{p}$ to render the scene. In experiments, we used a full
360∘ sampling range for the camera’s azimuth, and fixed the elevation. Figure
8 shows multiple views of a bird when optimizing with smaller azimuth ranges.
In the left-most column, a view from the central azimuth (frontal) is shown,
and is realistic for all training configurations. Views from more extreme
angles (right, left, rear view columns) have artifacts when the Dream Field is
optimized with narrow azimuth ranges. Training with diverse cameras is
important for viewpoint generalization.

Figure 8: Training with diversely sampled camera poses improves generalization
across views. In the top row, we sample camera azimuth from a single
viewpoint. The rendered view from the same perspective (left column) is
realistic, but the object structure is poor as seen from other angles.
Qualitative results improve with larger sampling intervals, with the best
results from 360∘ sampling.

##  6 Discussion and limitations

There are a number of limitations in Dream Fields. Generation requires
iterative optimization, which can be expensive. 2K-20K iterations are
sufficient for most objects, but more detail emerges when optimizing longer.
Meta-learning [54] or amortization [42] could speed up synthesis.

We use the same prompt at all perspectives. This can lead to repeated patterns
on multiple sides of an object. The target caption could be varied across
different camera poses. Many of the prompts we tested involve multiple
subjects, but we do not target complex scene generation [11, 8, 6, 13] partly
because CLIP poorly encodes spatial relations [29, 53]. Scene layout could be
handled in a post-processing step.

The image-text models we use to score renderings are not perfect even on
ground truth training images, so improvements in image-text models may
transfer to 3D generation. Our reliance on pre-trained models inherits their
harmful biases. Identifying methods that can detect and remove these biases is
an important direction if these methods are to be useful for larger-scale
asset generation.

##  7 Conclusion

Our work has begun to tackle the difficult problem of object generation from
text. By combining scalable multi-modal image-text models and multi-view
consistent differentiable neural rendering with simple object priors, we are
able to synthesize both geometry and color of 3D objects across a large
variety of real-world text prompts. The language interface allows users to
control the style and shape of the results, including materials and categories
of objects, with easy-to-author prompts. We hope these methods will enable
rapid asset creation for artists and multimedia applications.

## Acknowledgements

We thank Xiaohua Zhai, Lucas Beyer and Andreas Steiner for providing pre-
trained models on the LiT 3.6B dataset, Paras Jain, Kevin Murphy, Matthew
Tancik and Alireza Fathi for useful discussions and feedback on our work, and
many colleagues at Google for building and supporting key infrastructure. Ajay
Jain is supported in part by the NSF GRFP under Grant Number DGE 1752814.

## References

  * [1] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  ICML, 2017.
  * [2] Jonathan T. Barron and Jitendra Malik.  Shape, illumination, and reflectance from shading.  TPAMI, 2015.
  * [3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields.  ICCV, 2021.
  * [4] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan.  Learning gradient fields for shape generation.  ECCV, 2020.
  * [5] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  CVPR, 2021.
  * [6] Angel Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D. Manning.  Text to 3d scene generation with rich lexical grounding.  ACL-IJCNLP, 2015.
  * [7] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [8] Angel X Chang, Manolis Savva, and Christopher D Manning.  Learning spatial knowledge for text to 3D scene generation.  EMNLP, 2014.
  * [9] Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas A. Funkhouser, and Silvio Savarese.  Text2shape: Generating shapes from natural language by learning joint embeddings.  CoRR, abs/1803.08495, 2018.
  * [10] Eric Chu.  Evolving evocative 2d views of generated 3d objects.  NeurIPS Creativity and Design Workshop, 2021.
  * [11] Bob Coyne and Richard Sproat.  Wordseye: an automatic text-to-scene conversion system.  Computer graphics and interactive techniques, 2001.
  * [12] Karan Desai and Justin Johnson.  VirTex: Learning Visual Representations from Textual Annotations.  CVPR, 2021.
  * [13] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind.  Unconstrained scene generation with locally conditioned radiance fields.  arXiv, 2021.
  * [14] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  arXiv:2105.05233, 2021.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [16] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  CVPR, 2021.
  * [17] Kevin Frans, Lisa B. Soros, and Olaf Witkowski.  Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.  CoRR, 2021.
  * [18] Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah.  Multimodal neurons in artificial neural networks.  Distill, 2021.  https://distill.pub/2021/multimodal-neurons.
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NeurIPS, 2014.
  * [20] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis, 2021.
  * [21] Kunal Gupta and Manmohan Chandraker.  Neural mesh flow: 3d manifold mesh generationvia diffeomorphic flows, 2020\.
  * [22] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee.  Flax: A neural network library and ecosystem for JAX, 2020.
  * [23] Ajay Jain.  VectorAscent: Generate vector graphics from a textual description, 2021\.
  * [24] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  ICCV, 2021.
  * [25] Nikolay Jetchev.  Clipmatrix: Text-controlled creation of 3d textured meshes, 2021.
  * [26] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  ICML, 2021.
  * [27] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  CVPR, 2020.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  ECCV, 2014.
  * [29] Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, and Antonio Torralba.  Learning to compose visual relations.  In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  * [30] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  ACM Trans. Graph., 2019.
  * [31] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  SIGGRAPH Asia, 2015.
  * [32] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Transactions on Graphics (TOG), 2019.
  * [33] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  ECCV, 2020.
  * [34] Alexander Mordvintsev, Christopher Olah, and Mike Tyka.  Inceptionism: Going deeper into neural networks.  2015\.
  * [35] Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah.  Differentiable image parameterizations.  Distill, 2018.  https://distill.pub/2018/differentiable-parameterizations.
  * [36] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  CVPR, 2017.
  * [37] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  NeurIPS, 2016.
  * [38] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  CVPR, 2021.
  * [39] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2017.  https://distill.pub/2017/feature-visualization.
  * [40] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv:1807.03748, 2018.
  * [41] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach.  Benchmark for compositional text-to-image synthesis.  NeurIPS, 2021.
  * [42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  CVPR, 2019.
  * [43] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  ICCV, 2021.
  * [44] Victor Perez, Joel Simon, and Tal Shiri.  Sculpting with words.  2021\.
  * [45] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker.  On variational bounds of mutual information.  ICML, 2019.
  * [46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  CoRR, abs/2103.00020, 2021.
  * [47] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions.  CoRR, abs/1710.05941, 2017.
  * [48] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  CoRR, abs/2102.12092, 2021.
  * [49] Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero.  Clip-forge: Towards zero-shot text-to-shape generation, 2021.
  * [50] Peter Schaldenbrand, Zhixuan Liu, and Jean Oh.  Styleclipdraw: Coupling content and style in text-to-drawing synthesis, 2021.
  * [51] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  NeurIPS, 2020.
  * [52] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  NeurIPS, 2020.
  * [53] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach.  Reclip: A strong zero-shot baseline for referring expression comprehension.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [54] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng.  Learned initializations for optimizing coordinate-based neural representations.  CVPR, 2021.
  * [55] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, pages 5998–6008, 2017.
  * [57] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In Advances in Neural Information Processing Systems, pages 82–90, 2016.
  * [58] Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum.  Learning 3D Shape Priors for Shape Completion and Reconstruction.  In European Conference on Computer Vision (ECCV), 2018.
  * [59] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  arXiv:2106.12052, 2021.
  * [60] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning, 2021.
  * [61] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  CVPR, 2021.
  * [62] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv:2010.07492, 2020.
  * [63] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  CVPR, 2018.
  * [64] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu.  Learning to Reconstruct Shapes From Unseen Classes.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [65] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  ICCV, 2021.
  * [66] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian.  CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.  2021\.

Supplementary Material

Figure 9: Varying the image-text model used for Dream Field optimization.

##  Appendix A Qualitative results and ablations

An explanatory video with more qualitative results, code, an interactive Colab
notebook, and object-centric prompts are available at
https://ajayj.com/dreamfields. The video includes 360∘ renderings where the
camera orbits the object, as well as associated depth maps.

Figure 10: Object diversity: Objects vary with different seeds, effecting
NeRF’s weight initialization, camera sampling and augmentations.

#### Changing the image-text model

Figure 9 qualitatively compares generations using guidance from three
different contrastive image-text models. Dream Fields generated with
LiT${}_{\text{uu}}$ B/32 are generally sharper than those with CLIP models,
but all three can produce objects reflecting some aspects of the prompts.

#### Diversity of synthesized objects

In creative applications, users often want to select between multiple
synthesized results. Dream Fields can synthesize multiple objects from the
same prompt by changing the random seed before optimization. The seed changes
the initialization of the NeRF weights, the camera pose sampled each iteration
and the random background and crop augmentations. Figure 10 shows the effect
of changing the seed for four object-centric COCO prompts. Changing the seed
changes scene shape and layout. For example, the bus on the left is compressed
into a cubical shape, while the bus on the right is elongated. Colors and
textures are often similar across the seeds, though can also vary. Changing
the seed is another dimension of control in addition to prompt engineering.

##  Appendix B Object Centric COCO captions dataset

Our Object Centric COCO dataset includes 153 test set prompts and 74
development set prompts. Several additional prompts are used for qualitative
results, and are included in the main paper alongside figures. Captions are
included along with code on the project website.

##  Appendix C Hyperparameters and training setup

#### Positional encoding

Our Fourier feature positional encodings use $L=8$ frequency levels, while
novel view synthesis applications with image supervision commonly use $L=10$
to fit high-frequency details in photographs. Low-frequency ablations in Table
1 use $L=6$, which can improve convergence in the absence of our other
geometric priors.

#### Rendering

Scenes are bounded to a cube with side length 2. The camera is sampled at a
fixed radius of 4 units from the center of the cubical scene bounds and an
elevation of 30∘ above the equator. Near and far planes are set at
$4\pm\sqrt{3}$ units from the camera based on the minimum and maximum possible
distance to the corners of the cube. During training, we sample 192 points
along each ray, spaced uniformly and jittered with uniform noise. Rendered
1682 views are cropped to 1542 and upsampled to CLIP’s input resolution for
scenes where we compute qualitative metrics or 2522 views are cropped to 2242
for certain higher-quality visualizations. Crop sizes are selected to cover
about 80% of the image area. At test time, we sample 512 points along the rays
and render at a higher resolution equal to CLIP’s input size of 2242 or LiT’s
input size of 2882 for computing R-Precision and 4002 for visualizations.

Figure 11: Total loss with different sparsity regularizers and constant
$\mathcal{L}_{\mathrm{CLIP}}\in[0.1,-0.3]$. Upper and lower bounds of
$\mathcal{L}_{\mathrm{CLIP}}$ are shown with dashed lines. The additive loss
is convex, and is always minimized by increasing transmittance.

#### Optimization

MLP parameters are initialized with the Flax [22] defaults: LeCun normal
weights and zero bias for linear layers, and unit scaling and zero bias for
layer normalization. The MLP is optimized with Adam with $\epsilon=10^{-5}$.
Learning rate warms up exponentially from $10^{-5}$ to $10^{-4}$ over 1500
iterations, then is held constant. The camera origin is separately tracked
with an exponential moving average with decay rate $0.999$ of the center of
mass of rendered density. Figure 11 visualizes different forms of the
transmittance loss.

Figure 12: Optimizing a Neural Radiance Field scene representation (bottom)
leads to fewer artifacts than optimizing an explicit single-view 2D image
(top) or 3D voxel grid (middle), even when the explicit representations are
regularized. Our MLP has 16$\times$ fewer parameters than the voxel grid,
which may contribute to smoother, less noisy objects. We use CLIP B/16 for
this experiment.

#### Hyperparameter selection

Hyperparameters are manually tuned for visual quality on a development set of
74 object-centric COCO captions distinct from the test set reported in the
paper. Most tuning is done on a smaller subset of 20 of the 74 captions, and
hyperparameters are shared across all scenes.

#### Hardware

Optimization is done on 8 preemptible TPU cores, and 10K iterations takes
approximately 1 hour 12 minutes. This means each Dream Field costs
approximately $3-4 to generate on Google Cloud, which is economical for
applications. Training is bottlenecked by MLP inference and backpropagation
during volumetric rendering, not CLIP.

##  Appendix D Pixel and voxel baselines

We implemented 2D image optimization with a total variation loss and
generative prior (CLIP Guided Diffusion), as well as a 3D voxel baselines to
replace NeRF in Fig. 12. All results for this ablation optimize CLIP ViT B/16.

The 2D image is an RGB$\alpha$ pixel grid, composited with random backgrounds
during optimization similar to Dream Fields. Optimizing a single 2D
RGB$\alpha$ image does not produce a multi-view consistent 3D object, so other
viewpoints cannot be rendered. Even with transmittance and TV regularization,
the resulting image is noisy.

The voxel grid stores 1283 RGB and alpha values, interpolated trilinearly at
ray sample points and composited without a neural network using the PyTorch3D
library. Despite the transmittance loss, data augmentations and scene bounds,
the voxel grid also has significant low-density artifacts. The voxel baseline
has CLIP B/32 R-Precision 37.0%$\pm$3.9, while NeRF has 59.8%$\pm$2.8 (Tables
5.2, 3) with 16$\times$ fewer parameters, showing that the neural
representation improves consistency with the input caption in a generalizable
way. Using a hybrid representation with an explicit voxel grid followed by a
smaller MLP head might improve computational efficiency of Dream Fields
without degrading quality.

##  Appendix E Signed distance field parameterization

In early experiments, we learned scene density $\sigma$ with the VolSDF
parameterization [59]
$\sigma(\mathbf{x})=\alpha\Phi_{\beta}(-d_{\Omega}(\mathbf{x}))$ where
$d_{\Omega}(\mathbf{x})$ is a signed distance function implicitly defining the
object surface and $\Phi$ is the CDF of the Laplace distribution. This allows
normal vector prediction with autodifferentiation and could improve the
quality of the surface extracted from the radiance field. Dream Fields
successfully train with this alternate parameterization and produce visually
compelling objects. The SDF Eikonal loss introduces an additional loss weight
hyperparameter, which benefits from some tuning. Alternate 3D representations
are an interesting avenue for future work.

##  Appendix F Impact of optimization time

Figure 13: Long-run training and validation curves averaged over 79 hand-
written prompts. Transmittance remains close to the target $\tau$ throughout
training. Dream Fields overfit to the image-text representations used for
optimization, so in quantitative experiments, we stop training at 10K
iterations. The standard error of the mean is shaded.

Dream Fields can overfit to the aligned image-text representation used for
optimization. Figure 13 shows the training losses,
$\mathcal{L}_{\mathrm{CLIP}}$ and mean transmittance
$\mathrm{mean}(T(\theta,\mathbf{p}))$, as well as the validation R-Precision.
Objects are generated with LiT${}_{\text{uu}}$ ViT B/32 guidance, and
R-Precision is computed with a different contrastive image-text model, CLIP
ViT B/32. Validation renderings are also done at a held-out elevation angle.
Training loss continues to improve over long optimization trajectories, up to
10$\times$ longer than reported in the main paper. However, validation
retrieval accuracy declines after 5-10K iterations. The metrics are averaged
over 79 different hand-written captions that test fine-grained variations in
wording and prompt engineering.

Qualitatively, additional details and hyper-realistic effects are added over
the course of long runs, shown in our supplementary video. Some details are
not realistic, like floating text related to the typographic attacks
identified in [18].

More augmentations may help further regularize the optimization. These include
more aggressive 2D image augmentations such as smaller random crops, and more
3D data augmentations including varying focal length, varying distance from
the subject and varying elevation. 3D data augmentations are supported by our
approach.