# Viewpoint Textual Inversion: Unleashing Novel View Synthesis with Pretrained
2D Diffusion Models

James Burgess  Kuan-Chieh Wang  Serena Yeung
Stanford University
<EMAIL_ADDRESS>
jmhb0.github.io/viewneti/

###### Abstract

Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.

ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts. Code is available at our project
website.

Figure 1: Text-to-image diffusion models like Stable Diffusion encode 3D
understanding in their latent space. Left: our method, Viewpoint Neural
Textual Inversion (ViewNeTI), takes camera viewpoint parameters,
$\mathbf{R}_{i}$, and a scene-specific token, $S_{o}$, to predict a latent in
the CLIP output text space. We then sample from a frozen Stable Diffusion [41]
model conditioned on this text latent to produce an image of the $S_{o}$
object from camera view $\mathbf{R}_{i}$. We learn how to predict this text
condition using textual inversion (TI) [1, 13]. Right: our viewpoint control
mechanism enables leveraging pretrained 2D diffusion models for novel view
synthesis from only a single view; we can learn from more views if available.
The generations are photorealistic, have good semantic details, and generate
reasonable completions under ambiguity.

##  1 Introduction

Text-to-image diffusion models trained on web-scale datasets have shown
impressive capabilities in reasoning about objects, the composition of
multiple objects, and 2D spatial layout [41, 50, 18, 52, 44]. Despite being
trained on 2D image data, these models seem able to do 3D reasoning: in a
simple experiment, we ask a Stable Diffusion model [41] to infill the
background around a car, and find that it generates 3D-consistent shadows and
reflections (Fig. 2). In this work, we investigate how to extract 3D knowledge
from a pretrained diffusion model’s latent space, and how to leverage that
knowledge for 3D vision tasks. Utilizing pretrained diffusion models is
appealing for two reasons. First, because large and unposed 2D datasets [46,
36] are cheaper to procure than 3D datasets [10, 38] or multi-view datasets
[21, 9, 33], diffusion models cover a larger distribution of concepts; this
has motivated recent work on lifting 2D knowledge from diffusion models to 3D
[49, 64, 54, 22, 35]. Second, since diffusion models are generative, they are
well-suited for modeling the ambiguity that naturally arises in 3D vision due
to incomplete input information, for example in sparse-view novel view
synthesis [11, 32, 27].

Figure 2: Diffusion infilling models do 3d reasoning. Left: input masked
object. Middle: the generated shadows are consistent with the shadows on the
car and the shadows in the sand. Right: the ground has object reflections.

Our key insight is that the diffusion model text encoder space embeds
learnable concepts for 3D reasoning. Our method, Viewpoint Neural Textual
Inversion (ViewNeTI), controls the 3D viewpoint of objects via the text
encoder latent space, while keeping the object details consistent.
Specifically, we train a small neural mapper that takes camera parameters and
predicts a text encoding; the text encoding then conditions the diffusion
process to produce images with the desired camera view ( Fig. 1). The ViewNeTI
mapper weights are optimized using Textual Inversion (TI), an approach common
in personalization and content creation [43, 13, 24, 1]. TI adds novel
concepts to the diffusion model vocabulary, such as content (e.g. a user’s
particular dog) or styles (e.g. a particular artist’s visual aesthetic) by
optimizing new word embeddings to condition the diffusion model image
generation. Here, we apply textual inversion to add viewpoint control words to
the diffusion model vocabulary. To our knowledge, this is the first
application of TI for doing semantically-consistent image transformations.

We leverage ViewNeTI in a new formulation of novel view synthesis (NVS) by
conditioning diffusion model generation on camera parameters. We first
optimize ViewNeTI on a single scene using a small set of multi-view images.
Without access to any prior 3D data, the mapper generalizes to novel views, as
long as those views are ‘interpolations’ of the training views. Seeking a more
generic control mechanism that does viewpoint extrapolation, we pretrain
ViewNeTI on multiple scenes with a shared coordinate system. The pretrained
ViewNeTI mapper generalizes to novel scenes, enabling synthesis of novel views
far from the input views with little data; it can even do NVS from a single
image. Compared to existing single-image NVS methods, ViewNeTI has several
advantages, especially in single-view NVS. It produces views with
photorealistic details for real-world objects that are in the massive 2D
training distribution of 2D diffusion models like Stable Diffusion. Once
trained, it can generate diverse predictions under uncertainty in close to
real time. The pre-training multi-view dataset is small (less than 50 scenes)
and is cheap to train (less than one day with one GPU). Moreover, the view-
control generalizes to new object scenes under distribution shift of object
semantics. Finally, we show that ViewNeTI has the potential for more diverse
applications, such as content creation: by adding ViewNeTI conditioning to new
text prompts, we can control the 3D viewpoint around objects in generated
images.

Figure 3:  Our method for solving sparse-view novel view synthesis using
ViewNeTI. Left: we have a small multi-view dataset, $\mathcal{D}_{MV}$
containing images, $\mathbf{x}_{i}$, and known camera poses, $\mathbf{R}_{i}$.
We create a caption for each image. The caption contains a camera-specific
token $S_{\mathbf{R}_{i}}$ that is different for each view, and an object
token, $S_{o}$, which is common across views. Right: textual inversion
training of our neural mappers, $\mathcal{M}_{v}$ and $\mathcal{M}_{o}$. The
mappers predict the word embeddings for $S_{\mathbf{R}_{i}}$ and $S_{o}$
respectively. These are conditioned on diffusion timestep $t$ and UNet layer
$\ell$; the view mapper, $\mathcal{M}_{v}$, is additionally conditioned on
camera parameters, $\mathbf{R}_{i}$. All parameters are encoded by a Fourier
feature mapper, $\gamma$ [55]. The remaining tokens take their regular word
embeddings. The prompt is passed through the CLIP text encoder [36]. The post-
CLIP encoding corresponding to $S_{\mathbf{R}_{i}}$ and $S_{o}$ are perturbed
by a vector that is also predicted by $\mathcal{M}_{v}$ and $\mathcal{M}_{o}$.
This encoding conditions the diffusion model, $\epsilon_{\theta}$. We do
diffusion model training on this dataset while optimizing only
$\mathcal{M}_{v}$ and $\mathcal{M}_{o}$ (this is textual inversion training
[13, 1]). More details in Sec. 4.2.

##  2 Related work

Textual inversion of diffusion models Personalization aims to inject novel
concepts - concepts like objects and styles - into the diffusion model
vocabulary using a few image examples of that concept. Textual inversion (TI)
is a popular approach that optimizes a word embedding for each new concept
[13]. Compared to alternatives that fine-tune the model weights [43, 25],
learned concepts are editable, have low storage cost, and are more portable to
other copies of the same model [13, 8, 1]. Recent extensions train an encoder
to map images to concept vectors [80, 60, 14, 67]. Another direction improves
the quality and editability of the learned concepts by training different text
embeddings depending on the noising timestep and UNet layer [62, 1, 79]. These
are combined in the recent NeTI model [15], which is the current state of the
art in Textual Inversion. Our work utilizes many of the architectural ideas
used in NeTI.

Our work is, to our knowledge, the first to use textual inversion for
controlling 3D viewpoint in images and the first to use it for any 3D control
of generated objects. We propose an architecture for predicting text
embeddings that control camera viewpoint, and also contribute learning
strategies for adapting TI to do novel view synthesis and view-controlled
image generation.

Sparse-view novel view synthesis One set of techniques for sparse-view novel
view synthesis (NVS) train a NeRF [65] as an explicit 3d representation. To
address the challenge of sparse input views, these approaches add
regularization on novel views [34, 20, 71, 63, 12, 40, 47, 77, 69], modify the
training process [73, 48, 59], or condition the neural field on image features
that are derived from pretraining on multi-view datasets [75, 26, 6, 7]. Most
of these works that do not have pretraining do not attempt single-view NVS. On
the other hand, a recent series of models do tackle single-view NVS using
NeRFs with novel view regularization [11, 27, 32, 72] by leveraging diffusion
models as a data prior over 2D renders (via the Score Distillation Loss [64,
35]). Unlike our approach, the NeRF-based methods are not generative: once
trained, they cannot be easily sampled from, and thus cannot easily model
ambiguity. The sparse NeRF methods also tend to produce blurriness and image
artifacts compared to our method.

The other set of techniques for sparse-view NVS are ‘geometry free’: they do
not train any explicit 3D representation of the scene. Rather than doing per-
scene optimization, they train an inductive model. One approach for single-
view NVS is to formulate an image-to-image translation problem [53, 57], and
recent works have leveraged diffusion models [27, 66]. Many works design the
architecture and losses to encode 3D priors [45, 68, 19], and recent
approaches also use diffusion models [28, 58, 54, 81, 22, 2, 4]. These
geometry-free methods require pretraining on multi-view datasets [10, 5, 38],
and they most commonly test on images that are in the same class and covariate
distribution as the training set. Our approach differs in that we do not
directly train or fine-tune an image generation model. We instead learn to
predict text embeddings to condition image generation in large pretrained
models like StableDiffusion. Since we control the 3D view of objects in the
very large StableDiffusion vocabulary, we can do novel view synthesis on
scenes with semantics outside the pretraining dataset.

A concurrent work on ArXiv, DreamSparse [74], does sparse-view NVS by learning
to map multi-view inputs to 3D features, which then condition a perturbation
on the UNet predictions (as in ControlNet [78]). This has a similar motivation
to our work: by using pretrained frozen diffusion models, they can do NVS for
classes outside the distribution of the multi-view pretraining set: they are
‘open-set’. However, their guidance approach is different, and it does not do
NVS from one input view, which is our primary application.

##  3 Background

###  3.1 Text-to-image latent diffusion models

We apply viewpoint textual inversion to text-to-image Stable-Diffusion (SD).
SD’s are Latent Diffusion Models (LDMs) for image generation [41] and are
typically trained on web-scale datasets of text-image pairs
$(\mathbf{x},y)\\!\sim\\!\mathcal{D}$ [46]. There are two components. First, a
variational autoencoder (VAE) [23, 39] with encoder $\mathcal{E}$ and decoder
$\mathcal{D}$ compresses RGB images, $\mathbf{x}\in\mathbb{R}^{H\times W\times
3}$ to a lower-dimensional latent
$\mathbf{z}_{0}\\!=\\!\mathcal{E}(\mathbf{x})\in\mathbb{R}^{h\times w\times
c}$. Second, a conditional diffusion model [50, 18, 52, 17] is trained to
generate this distribution of latents with respect to the text prompt, $y$, as
$p(\mathbf{z}|y)$.

LDMs model a diffusion process: noise is incrementally added to the the latent
over $T$ steps; the intermediate latents are $\mathbf{z}_{t}$ with
$t\in[0,T]$. We learn a neural network $\epsilon_{\theta}$ that reverses each
step by predicting the applied noise. To train this network, we simulate
$\mathbf{z}_{t}$ by sampling isotropic Gaussian noise, $\mathbf{\epsilon}$,
scaling it according to a parameter $t\\!\sim\mathcal{U}[0,T]$, and adding it
to $\mathbf{z}$. The training objective is for $\epsilon_{\theta}$ to predict
$\mathbf{\epsilon}$ conditioned on noising step, $t$ and the text, $y$:

$L_{LDM}:=\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D},\mathbf{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),t}\Bigl{[}\|\mathbf{\epsilon}-\epsilon_{\theta}(\mathbf{z}_{t},y,t)\|\Bigl{]}$
(1)

One can sample from $p(\mathbf{z}|y)$ with
$\mathbf{z}_{T}\\!\sim\\!\mathcal{N}(0,\mathbf{I})$, and using
$\epsilon_{\theta}$ to run the reverse process over $T$ steps [50], . This
gives a latent sample $\tilde{z}$ which is decoded to an image,
$\tilde{\mathbf{x}}\\!=\\!\mathcal{D}(\tilde{\mathbf{z}})$. The
$\epsilon_{\theta}$ architecture is a conditional UNet [42]. $\mathbf{z}_{t}$
is passed through the main Unet stream. The text is passed through a
pretrained CLIP [36] text encoder, giving a $d$-dim conditioning vector for
each token, $\mathbf{c}(y)\in\mathbb{R}^{d\times 77}$, which is mixed with
each Unet layer via cross-attention [41].

###  3.2 Textual inversion

In textual inversion (TI) [13], we learn a new text embedding,
$\mathbf{v}_{S_{o}}$, for pseudo-word $S_{o}$, which represents a new concept
from a small dataset. The dataset, $(\mathbf{x},y)\\!\sim\\!\mathcal{D}_{TI}$
contains images, $\mathbf{x}$, of that concept; and paired captions, $y$, that
are “A photo of $S_{o}$”. To learn the word embedding $\mathbf{v}_{S_{o}}$, we
use the LDM loss as in Eq. 1, but replacing $\mathcal{D}$ with
$\mathcal{D}_{TI}$, and optimizing only with respect to $\mathbf{v}_{S_{o}}$.
Importantly, the diffusion model weights are frozen.

A recent work proposes the NeTI model [1], which includes many recent advances
in textual inversion [80, 60, 14, 67]. Instead of learning a single embedding
for $S_{o}$, it learns an embedding for each UNet cross-attention layer,
$\ell$ [62] and for each noise step, $t$ [15]; this representation space is
denoted $\mathcal{P}^{*}$ [1]. NeTI is implemented as a small neural network
mapper, $\mathcal{M}$ conditioned on $t$ and $l$. The optimization of Eq. 1 is
with respect to the weights of $\mathcal{M}$ [1]. Our work, ViewNeTI, uses the
NeTI architecture.

##  4 Method

Viewpoint Neural Textual Inversion (ViewNeTI) controls the viewpoint of
objects in images generated by diffusion models. In Sec. 4.1 we describe the
core architectural component, which is a small neural network - the view-
mapper or $\mathcal{M}_{v}$ \- that takes camera parameters and predicts a
text encoding. This text encoding then conditions the diffusion generation
process to control viewpoint in images.

The ViewNeTI view mapper is trained to do novel view synthesis from few
inputs. Illustrated in Fig. 3, we use textual inversion (TI) to optimize the
$\mathcal{M}_{v}$ parameters jointly with a mapper for the objects semantics
$\mathcal{M}_{o}$. In Sec. 4.2 we describe how to train $\mathcal{M}_{v}$ on a
single scene with no prior multi-view supervision, which enables interpolation
of novel views. In Sec. 4.3, we propose pretraining $\mathcal{M}_{v}$ on a
multi-view dataset, which enables extrapolation of novel views far from the
input views, and under semantic distribution shift. Here, we highlight the
especially challenging application of NVS from a single input view. Finally in
Sec. 4.4, we show that the pretrained $\mathcal{M}_{v}$ can be used in text-
to-image generation to control the viewpoint around objects.

###  4.1 ViewNeTI mapper architecture and inference

The ViewNeTI mappers control the text encoding for the diffusion model. As
shown in Fig. 3 we have a view-mapper, $\mathcal{M}_{v}$, and an object
mapper, $\mathcal{M}_{o}$, to predict the text encoding of the view and object
tokens respectively. They are both conditioned on the diffusion timestep, $t$,
and UNet conditioning layet, $\ell$, while the view-mapper is also conditioned
on camera parameters, $\mathbf{R}$. They predict a word embedding input to the
text encoder. They also predict a ‘textual bypass’, which is a perturbation to
the vectors that are output from the text encoder. We now explain Fig. 3 in
detail.

3D-view/pose representation One input parameter to the view-mapper,
$\mathcal{M}_{v}$, is the camera parameters, $\mathbf{R}$, which can be any
vector representation of the camera extrinsics and intrinsics. In our
experiments, we use the camera-to-world project matrix, and we normalize each
matrix entry independently to $[-1,1]$. The normalization range is fixed
according to the dataset parameters for experiments in Sec. 5. Our method is
agnostic to the camera parameterization: we verify that our method also works
with spherical coordinates in Appendix G.

We apply Fourier-feature encoding [55, 37] with bandwidth $\sigma=2$. This
enables the neural mappers to learn a predictor of word-embedding space that
is sufficiently sensitive to small changes in camera parameters; that is, it
can represent high frequency changes in word embedding space. The $\sigma$
parameter is fixed across our experiments: it is big enough to model a diverse
viewpoint range, but small enough to interpolate viewpoints (see Sec. 5.4
ablations).

Architecture Both neural mappers, $(\mathcal{M}_{v},\mathcal{M}_{o})$ are
conditioned on the the denoising timestep and the diffusion model UNet layer,
$(t,\ell)$. This improves textual inversion reconstruction quality and
optimization convergence because different timesteps and UNet layers control
different image features [1, 62]; for example, small $t$ denoising steps
control finer texture details rather than layout and shape. We concatenate
these parameters with the camera parameters into a vector,
$[t,\ell,\mathbf{R}]$. Let the Fourier feature encoding function [37, 55] be
$\gamma(\cdot)$, and the encoding be $\mathbf{c}_{\gamma}$. Then the encoding
is $\mathbf{c}_{\gamma}=\gamma([t,\ell,\mathbf{R}])$, and we choose its
dimension as 64.

We define the function parameterized by our view-mapper as:

$(\mathbf{v}_{\mathbf{R}},\mathbf{v}_{\mathbf{R},p})=\mathcal{M}_{v}(\mathbf{c}_{\gamma})$
(2)

The vectors $(\mathbf{v}_{R},\mathbf{v}_{\mathbf{R},p})$ have the same
dimension as the text encoder, which is 768 in Stable Diffusion 2 (SD2) [41].
We parameterize $\mathcal{M}_{v}$ as a 2-layer MLP with 64 dimensions,
LayerNorm [3], and LeakyRelu [70]. The view-mapper has 140,000 parameters. The
object-mapper, $\mathcal{M}_{o}$ is defined in the same way, except without
conditioning on camera parameters. So, $\mathbf{c}_{\gamma}=\gamma([t,\ell])$
and:

$(\mathbf{v}_{o},\mathbf{v}_{o,p})=\mathcal{M}_{o}(\mathbf{c}_{\gamma})$

In the next subsection we describe how the two output vectors are used for
controlling the 3D viewpoint of Stable Diffusion generations.

Inference for view-controlled generation We pass the text encoder a prompt
like, ‘$S_{\mathbf{R}}$. A photo of a $S_{o}$’, where $S_{\mathbf{R}}$ has
word embedding $\mathbf{v}_{\mathbf{R}}$ and $S_{o}$ has word embedding
$\mathbf{v}_{o}$. The embedding for $S_{\mathbf{R}}$ controls the viewpoint
for the image that is been generated. The embeddings for $S_{o}$ are the same
for all novel views generated in one scene, so it captures properties that are
invariant across all images such as object semantics and image style.

We scale the embeddings, $(\mathbf{v}_{\mathbf{R}},\mathbf{v}_{o})$ to have
the same $L_{2}$ norm as the word embedding for ‘object’ (the choice of this
reference word is not important in practice). We then pass this prompt through
the CLIP text encoder with padding, which gives 77 encoding vectors. If $k$ is
the index for token $S_{\mathbf{R}}$, then call its embedding vector
$\mathbf{e}_{k}$. We perturb $\mathbf{e}_{k}$ in the direction of the
$\mathbf{v}_{\mathbf{R},p}$ vector (which was predicted by $\mathcal{M}_{v}$
as in Eq. 2):

$\mathbf{e}_{k}^{\prime}:=\mathbf{e}_{k}+\alpha\|\mathbf{e}_{k}\|\cdot\frac{\mathbf{v}_{\mathbf{R},p}}{\|\mathbf{v}_{\mathbf{R},p}\|}$
(3)

This is called textual bypass [1], and it enables more flexibility for the
view-mapper control, as well as faster convergence. $\alpha$ controls the
perturbation magnitude, and prior textual inversion methods set it to $0.2$.
This constrains the bypass mechanism to only refine object details, and thus
preserve the compositionality of the word embedding with other concepts in
word space. We set $\alpha=0.2$ only for experiments in text-to-image
generation. But for NVS, we do not care about compositionality, so we are free
to increase $\alpha$ to 5, which we find has similar view-control performance,
and better reconstructs object details (see Sec. 5.4 ablations). The final
view-mapper output, $\mathbf{e}_{k}^{\prime}$, is the $k$th conditioning
vector that is passed to the UNet generator. We apply this exact same textual
bypass approach to the object token, where we perturb using the predicted
$\mathbf{v}_{o,p}$.

To generate the final image in viewpoint $\mathbf{R}$, we run diffusion model
generation with DPMSolver [30, 31, 51] for 50 steps to get the final image.

Data augmentations We apply simple augmentations to images, similar to [32].
This helps avoid overfitting when the training dataset is small, as we show in
Sec. 5.4 ablations. We also do text prompt augmentations that are standard in
textual inversion [13]. See Appendix C for details.

###  4.2 Single-Scene Novel View Synthesis

In single-scene NVS, we have a small ($<10$) dataset called $\mathcal{D}_{MV}$
with multi-view images, $\mathbf{x}_{i}$, with known camera pose parameters,
$\mathbf{R}_{i}$. We do not have prior 3D knowledge from other multi-view
datasets. As in Fig. 3, we generate captions of the form
$y(S_{\mathbf{R}_{i}},S_{o})=$“$S_{\mathbf{R}_{i}}$. a photo of a $S_{o}$”. As
in the previous Sec. 4.1, $S_{\mathbf{R}_{i}}$ corresponds to viewpoint and
its embeddings are predicted by $\mathcal{M}_{v}$. Likewise, $S_{o}$
corresponds to object semantics and its embeddings are predicted by
$\mathcal{M}_{o}$. The $S_{o}$ embeddings are the same for all images. Now,
let the modified multi-view dataset be the set of image-caption pairs,
$\left(\mathbf{x}_{i},y(S_{\mathbf{R}_{i}},S_{o})\right)\sim\mathcal{D}_{MV^{\prime}}$.

We optimize the weights of $\mathcal{M}_{v}$ and $\mathcal{M}_{o}$ using the
loss in Eq. 1, except we replace $\mathcal{D}$ with
$\mathcal{D}_{MV^{\prime}}$. Intuitively, we are learning text space latents
that, when conditioning diffusion model generation, reproduce the training set
images with the correct camera views.

###  4.3 Novel view synthesis with pretraining on multi-view data

The single-scene optimization case cannot extrapolate to views far from the
input views (see Sec. 5.1 and Fig. 4). It also cannot do one-shot novel view
synthesis. To tackle these challenging settings, we propose pretraining on a
multi-view dataset with multiple scenes.

Pre-training We now have access to images, $\mathbf{x}_{ij}$ with known camera
poses, $\mathbf{R}_{ij}$, for views $i$ and scenes $j$. The camera poses are
in the same reference frame: their notion of ‘up’ and ‘down’ are the same;
however there is no requirement for the scene’s objects to have any particular
pose within that space. The multi-view datasets should have dense coverage of
the view space, which we visualize in Appendix D.

The goal is to train a view-mapper, $\mathcal{M}_{v}$, that is shared across
all the scenes, and is generalizable to new scenes. The training is the same
as the single scene case in Sec. 4.2, with one change: in the captions,
“$S_{\mathbf{R}_{i}}$, a photo of a $S_{o,j}$”, each scene has its own object
token $S_{o,j}$ and object mapper, $\mathcal{M}_{o,j}$ (but again, they share
view tokens, $S_{\mathbf{R}_{i}}$ and view-mapper, $\mathcal{M}_{v}$).

Novel view synthesis from sparse input views Given a sparse dataset of muti-
view images and camera parameters,
$(\mathbf{x}_{i},\mathbf{R}_{i})\sim\mathcal{D}_{MV}$, we do the exact
training procedure for single-scene NVS described in Sec. 4.2, except we use
the pretrained view mapper, $\mathcal{M}_{v}$ and freeze it’s weights. The
inference procedure for generating novel views is also the same.

Novel-view synthesis from a single input view Under this framework, single-
view NVS is solved with the same approach as sparse-view NVS.

###  4.4 Generalizing Viewpoint Control to New Prompts

Content creation via text-to-image diffusion is a popular application, and one
research direction seeks to control certain properties in generated images
[16, 78]. ViewNeTI can be easily extended to control the camera viewpoint
around objects that are generated from user-defined captions, $y$. We take the
pretrained view mapper, $\mathcal{M}_{v}$, described in Sec. 4.3, and we
prepend its corresponding token to $y$. An example text prompt is
“$\mathbf{R}_{0}$. a brown teddy bear”. Then, the inference procedure is the
same as Sec. 4.1.

Figure 4: Novel view synthesis trained on a single scene (without
pretraining), described Sec. 5.2.2. Top: the camera positions for the DTU
buddha scene [21] visualized with Nerfstudio [56] and SDFStudio [76]. We show
input views from the 6-view split (blue), inference views that are
‘interpolated’ from the input (green), and inference views that are
‘extrapolated’ (red). Bottom left: inference on interpolated views has good
semantics and pose. Bottom right: inference on extrapolated views does not
work. The semantics are good but the poses are wrong. This limitation
motivates our pretraining strategy, see Sec. 4.3.  Figure 5: Qualitative
comparison of single-view novel view synthesis on DTU [21]. Our method has
good photorealism and semantics compared to baselines (see Sec. 5.2.2).

##  5 Results

In Sec. 5.1, we use ViewNeTI for sparse-view novel view synthesis (NVS) on a
single scene, without access to any multi-view or 3D datasets. We show the
limitations of the single-scene approach, which motivate our pretraining and
fine-tuning method. This leads to our primary result in Sec. 5.2.2: novel view
synthesis from a single view. In Sec. 5.3, we show an extra application of
ViewNeTI to view-controlled text-to-image generation. All results use the same
hyperparameters, and we ablate these design choices in 5.4.

NVS data setting We evaluate on DTU-MVS [21], a multi-view dataset of real-
world objects with challenging details. We use the train-test splits for
camera views used in sparse-view snythesis from prior works [75, 11] (training
set sizes 1, 3, 6, and 9). As visualized in Appendix E, DTU images can be
roughly grouped by their camera pitch angle into ‘top-view’, ‘side-view’, and
‘neutral-view’. The 9-view training set has images from all groups, so novel
test views are ‘interpolations’. The 6-view dataset has images from the side
and neutral views, so test views from top-view are ‘extrapolation’ (this is
also visualized in Fig. 5. The 3-view dataset uses neutral-view images, and so
top and side views are extrapolation. For the 1-view set, all test views are
extrapolation.

###  5.1 Single-scene novel view synthesis from sparse views

We start by establishing that the text latent space can be manipulated for
viewpoint control. Using the method described in Sec. 4.2, we optimize a view-
mapper and object-mapper from scratch on a scene with 6-views. The training
time is 2.5hrs with one Titan RTX.

In Fig. 5 we visualize the training and inference camera positions, and the
NVS predictions. Despite being supervised on few images, we can interpolate to
novel views. We retain good object details, and avoid imaging artifacts like
floaters that appear in NeRF [34]. However, in this setting we fail to
extrapolate. The failure mode is interesting and different to other NVS
methods: the object semantics approximately match the training views, but the
poses are incorrect. This motivated us to pretrain the view-mapper, as we
review in the next section

###  5.2 Novel view synthesis from one input view

####  5.2.1 View-mapper pretraining

We pretrain a view-mapper, $\mathcal{M}_{v}$ as described in Sec. 4.3. We take
88 training scenes from DTU chosen by [75] to ensure that object classes are
different between the train and test scenes. The pretraining takes 48hours on
one Titan RTX. We validate the approach with generations in Appendix F: the
view-mapper and object-mapper correctly reconstruct the training set of views,
where the view-mapper is shared across all scenes.

####  5.2.2 Single-view NVS with a pretrained view-mapper

We use the frozen view-mapper, $\mathcal{M}_{v}$, and fine-tune a new object-
mapper, $\mathcal{M}_{o}$ on a new scene from a different semantic
distribution, as described in Sec. 4.3, which takes one hour on one Titan RTX.
We show single-view NVS predictions for selected scenes in Fig. 1, and all DTU
test scenes in Fig. 7. In Fig. 5, we compare NVS predictions against baselines
using the same challenging evaluation scenes and views chosen by [11]. We
compare against zero-1-to-3 in Appendix H; this is separate because this model
has a different coordinate system that does not allow translation.

The first 3 methods learn an explicit 3D scene representation - a NeRF - which
has the benefit of consistent scene geometry. But they have blurriness and
artifacts that commonly appear under sparse supervision [65, 75, 11] and they
have errors in the semantic details. They also have the drawback that one
cannot easily sample novel views from a high-variance distribution, which is
desirable when the prediction problem is ambiguous. The zero-1-to-3 model has
a fast runtime, but it is not suitable for NVS on images with photorealisitc
object details (as shown by the toy pig scene), or for images where the
borders crop the object (as shown by the brick scene), becahsethey are outside
the training distribution of 3D assets [10].

Our ViewNeTI predictions, while sometimes hallucinating inaccurate object
details, do generate photorealistic images with plausible semantics and little
blurriness. Once trained, ViewNeTI can generate diverse samples in close to
real time, and thus can be useful for modeling uncertainty. For completeness,
we also compare the LPIPS, SSIM, and PSNR against baselines in Tab. 1, which
is standard in NVS. We agree with the remarks in NeRDi [11] that these
reconstruction-based metrics are not suitable for the single-view setting
because there is high uncertainty in the 3D prediction. This metric will
reward methods that perform some ‘averaging’ over diverse views, rather than
those methods that predict diverse but plausible views. Having said that, the
’perceptual’ metrics LPIPS and SSIM may not be as problematic as PSNR. Our
approach is state-of-the-art for LPIPS, and near state-of-the art for SSIM.
Our PSNR is poor which, as we argue in Sec. 6, is caused by issues with
localizing the object in the image. That is, while the object details and pose
are good, the image is mis-translated with respect to the ground truth. This
is an area in which NeRFs are naturally at an advantage over diffusion models.

Table 1: Single-image novel view synthesis metrics on DTU. The best score is in bold, and second best is underlined. We argue in Sec. 5.2.2 that these metrics are imperfect for settings with ambiguity like this one. Method |  LPIPS $\downarrow$ |  SSIM $\uparrow$ |  PSNR $\uparrow$
---|---|---|---
NeRF [65] | 0.703 | 0.286 | 8.000
pixelNeRF [75] | 0.515 | 0.564 | 16.048
SinNeRF [71] | 0.525 | 0.560 | 16.520
NerDi [11] | 0.421 | 0.465 | 14.472
ViewNeTI (ours) | 0.378 | 0.516 | 10.947

###  5.3 View-controlled text-to-image generation

In Fig. 6, we show examples of using ViewNeTI to control the viewpoint of
images in text-to-image generation. The object semantics are consistent across
views, and using our conditioning adds only negligible time to the generation
procedure. Moreover, the example images are outside the semantic distribution
of the DTU training dataset.

###  5.4 Ablations

We show the effectiveness of our design choices by showing qualitative results
on ablated versions of ViewNeTI for single-view NVS in Appendix B. The key
architectural choices are the frequency of the positional encoding, the text
embedding norm scaling, the flexibility permitted in the textual bypass, and
the image augmentations. Other important choices include the freezing of the
pretrained view-mapper, and the choice of image dimension in object sampling.
Note that the results in Fig. 5 already established that no-pretraining leads
to a failure to extrapolate away from the training viewpoints. Note that our
application differs from standard textual inversion works [13, 1] in that we
are not interested in compositionality of our word embeddings with existing
concepts. As such, we do not ablate against this.

Figure 6:  ViewNeTI controls viewpoint in text-to-image generation by
composing the view-mapper text encoder, represented by $R_{i}$, with novel
text prompts. Top: reference poses from a scene in the DTU [21] dataset.
Middle & bottom: text generations conditioned on ViewNeTI view control tokens.

##  6 Limitations

The major limitation for ViewNeTI in novel view synthesis (NVS) is
localization: it is prone to generating objects that are mis-translated
slightly from the ground truth, and this drives the degradation in PSNR scores
seen in Tab. 1. This is a product of the training procedure: unlike in NeRFs,
we do not penalize the MSE loss with an image. However in return for this
drawback, we gain the ability to generate realistic samples (thanks to the
pretraining on a web-scale datasets) and diverse samples (thanks to the
generative nature of the model).

ViewNeTI also struggles to generate precise details in objects, such as small
textures in the building scene ( Fig. 7, row 2). Reconstruction quality is an
active area of research in textual inversion [1] and advances there should be
transferable to ViewNeTI.

Although ViewNeTI scene optimization has similar training time to NeRF-based
methods for sparse NVS (about 1 hour, which is similar to [73, 63]), it is far
from the realtime speeds of recent methods that pose NVS as image-to-image
translation [27]. The key bottleneck in our approach is optimizing new object
tokens. But here, too, there is much work among textual inversion researchers
for accelerating these predictions, [80, 60, 14, 67] and we expect these
advances will be applicable to ViewNeTI.

##  7 Conclusions

In this study, we investigated the capabilities of diffusion models to
represent and reason about the 3D world. Remarkably, despite having only
unposed 2D image supervision during training, we found strong evidence that
the diffusion model latent space does encode 3D knowledge. We proposed
Viewpoint Neural Textual Inversion (ViewNeTI), a framework for taking
advantage of this prior knowledge for 3D vision tasks. Our method controls the
viewpoint of images generated by a diffusion model by predicting appropriate
conditioning vectors in the text latent space using a neural view-mapper.

Our method naturally addresses novel view synthesis by fitting the neural
view-mapper on multi-view image datasets. We fit the view mapper on a single
scene with very sparse viewpoints ( $<$ 10). Despite this small dataset being
the only 3D supervision ever seen by the image diffusion model, the learned
view-mapper was able to generalize to novel camera viewpoints, as long as
those views were near the training views. Next we showed that the ViewNeTI
could learn a more generic control mechanism for viewpoint control. We
pretrained the view mapper on a multi-scene dataset that shared a common
reference frame. The view mapper was able to learn this reference frame, while
the per-scene object mappers learned the pose of objects within that scene. We
highlighted impressive results for single-view novel view synthesis, an
extremely challenging task. Here, our results showed excellent photorealism
and good object semantics. This is most likely the result of the diffusion
model leveraging prior knowledge from its massive training distribution, since
the NeRF methods - without the benefit of large-scale pretraining datasets -
rendered views with more blur and artifacts. Notably our multi-view
pretraining dataset was small in scale compared to datasets underpinning other
advances in vision, and the training time was modest at only 2 days on one
GPU. This shows that extracting 3D knowledge from 2D models need not require
unreasonably large 3D datasets.

We have demonstrated strong results for viewpoint manipulation, which is one
important form of 3D control. Looking forward, we hope that our framework can
inspire work on leveraging diffusion model latent spaces for other challenging
3D tasks, such as scene relighting and 2D-to-3D lifting for 3D human pose
estimation.

## References

  * Alaluf et al. [2023] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or.  A neural space-time representation for text-to-image personalization.  _arXiv preprint arXiv:2305.15391_ , 2023.
  * Anciukevičius et al. [2023] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero.  Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12608–12618, 2023.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Chan et al. [2023] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  Generative novel view synthesis with 3d-aware diffusion models.  _arXiv preprint arXiv:2304.02602_ , 2023.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14124–14133, 2021.
  * Chibane et al. [2021] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7911–7920, 2021.
  * Cohen et al. [2022] Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon.  “this is my unicorn, fluffy”: Personalizing frozen vision-language representations.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX_ , pages 558–577, 2022.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13142–13153, 2023.
  * Deng et al. [2023] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al.  Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20637–20647, 2023.
  * Deng et al. [2022] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12882–12891, 2022.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gal et al. [2023a] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _arXiv preprint arXiv:2302.12228_ , 2023a.
  * Gal et al. [2023b] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _arXiv preprint arXiv:2302.12228_ , 2023b.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Jabri et al. [2023] Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi SM Sajjadi, and Thomas Kipf.  Dorsal: Diffusion for object-centric representations of scenes $etal.$.  _arXiv preprint arXiv:2306.08068_ , 2023.
  * Jain et al. [2021] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5885–5894, 2021.
  * Jensen et al. [2014] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 406–413, 2014.
  * Karnewar et al. [2023] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra.  Holodiffusion: Training a 3d diffusion model using 2d images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18423–18433, 2023.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kumari et al. [2023a] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023a.
  * Kumari et al. [2023b] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023b.
  * Lin et al. [2023] Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 806–815, 2023.
  * Liu et al. [2023a] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object, 2023a.
  * Liu et al. [2023b] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  _arXiv preprint arXiv:2309.03453_ , 2023b.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _arXiv preprint arXiv:2206.00927_ , 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8446–8455, 2023.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–14, 2019.
  * Niemeyer et al. [2022] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5480–5490, 2022.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rahimi and Recht [2007] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  _Advances in neural information processing systems_ , 20, 2007.
  * Reizenstein et al. [2021] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10901–10911, 2021.
  * Rezende et al. [2014] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International conference on machine learning_ , pages 1278–1286. PMLR, 2014.
  * Roessle et al. [2022] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12892–12901, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sajjadi et al. [2022] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al.  Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6229–6238, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Seo et al. [2023a] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  _arXiv preprint arXiv:2303.07937_ , 2023a.
  * Seo et al. [2023b] Seunghyeon Seo, Donghoon Han, Yeonjin Chang, and Nojun Kwak.  Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20659–20668, 2023b.
  * Siddiqui et al. [2023] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder.  Panoptic lifting for 3d scene understanding with neural fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9043–9052, 2023.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Sun et al. [2018] Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, and Joseph J Lim.  Multi-view to novel view: Synthesizing novel views with self-learned confidence.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 155–171, 2018.
  * Szymanowicz et al. [2023] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi.  Viewset diffusion:(0-) image-conditioned 3d generative models from 2d data.  _arXiv preprint arXiv:2306.07881_ , 2023.
  * Tancik et al. [2020] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  _Advances in Neural Information Processing Systems_ , 33:7537–7547, 2020.
  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.  Nerfstudio: A modular framework for neural radiance field development.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023.
  * Tatarchenko et al. [2016] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Multi-view 3d models from single images with a convolutional network.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_ , pages 322–337. Springer, 2016.
  * Tewari et al. [2023] Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B Tenenbaum, Frédo Durand, William T Freeman, and Vincent Sitzmann.  Diffusion with forward models: Solving stochastic inverse problems without direct supervision.  _arXiv preprint arXiv:2306.11719_ , 2023.
  * Truong et al. [2023] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari.  Sparf: Neural radiance fields from sparse and noisy poses.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4190–4200, 2023.
  * Valevski et al. [2023] Dani Valevski, Danny Wasserman, Yossi Matias, and Yaniv Leviathan.  Face0: Instantaneously conditioning a text-to-image model on a face.  _arXiv preprint arXiv:2306.06638_ , 2023.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models, 2022.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  _arXiv preprint arXiv:2303.09522_ , 2023.
  * Wang et al. [2023a] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu.  Sparsenerf: Distilling depth ranking for few-shot novel view synthesis.  _arXiv preprint arXiv:2303.16196_ , 2023a.
  * Wang et al. [2023b] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12619–12629, 2023b.
  * Wang et al. [2021] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_ , 2021.
  * Watson et al. [2022] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  _arXiv preprint arXiv:2210.04628_ , 2022.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Wiles et al. [2020] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7467–7477, 2020.
  * Wynn and Turmukhambetov [2023] Jamie Wynn and Daniyar Turmukhambetov.  Diffusionerf: Regularizing neural radiance fields with denoising diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4180–4189, 2023.
  * Xu et al. [2015] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  _arXiv preprint arXiv:1505.00853_ , 2015.
  * Xu et al. [2022a] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang.  Sinnerf: Training neural radiance fields on complex scenes from a single image.  In _European Conference on Computer Vision_ , pages 736–753. Springer, 2022a.
  * Xu et al. [2022b] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang.  Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360° views.  _arXiv e-prints_ , pages arXiv–2211, 2022b.
  * Yang et al. [2023] Jiawei Yang, Marco Pavone, and Yue Wang.  Freenerf: Improving few-shot neural rendering with free frequency regularization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8254–8263, 2023.
  * Yoo et al. [2023] Paul Yoo, Jiaxian Guo, Yutaka Matsuo, and Shixiang Shane Gu.  Dreamsparse: Escaping from plato’s cave with 2d frozen diffusion model given sparse views.  _arXiv preprint arXiv:2306.03414_ , 2023.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Yu et al. [2022a] Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger.  Sdfstudio: A unified framework for surface reconstruction, 2022a.
  * Yu et al. [2022b] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  _Advances in neural information processing systems_ , 35:25018–25032, 2022b.
  * Zhang and Agrawala [2023] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  _arXiv preprint arXiv:2302.05543_ , 2023.
  * Zhang et al. [2023a] Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong-Yee Lee, Oliver Deussen, and Changsheng Xu.  Prospect: Expanded conditioning for the personalization of attribute-aware image generation.  _arXiv preprint arXiv:2305.16225_ , 2023a.
  * Zhang et al. [2023b] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu.  Inversion-based style transfer with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10146–10156, 2023b.
  * Zhou and Tulsiani [2023] Zhizhuo Zhou and Shubham Tulsiani.  Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12588–12597, 2023.

Supplementary Material

## Supplementary overview

All supplementary figures are after the supplementary text. Especially
important are Fig. 8 and Fig. 7, which are 3-view and single-view novel view
synthesis predictions for every scene in the DTU test set.

The supplementary sections and their figures are:

  * •

A: diffusion model outfilling tests as evidence that diffusion models do 3D
reasoning. Fig. 9.

  * •

B: qualitative ablations of ViewNeTI design decisions for single-view NVS.
Fig. 11, Fig. 10.

  * •

C: image and text prompt augmentation details. Fig. 14.

  * •

D: visualizations of the camera positions in DTU pretraining data. Fig. 15.

  * •

E: visualizations of the camera positions in DTU sparse view datasets (1, 3,
and 6 views). Fig. 15.

  * •

F: validation of ViewNeTI pretraining of the view-mapper, and disentanglement
of the object-mapper. Fig. 16, Fig. 17.

  * •

G: validation that ViewNeTI works with spherical coordinate camera
parameterization. Fig. 13.

  * •

H: comparison to Zero-1-to-3 [27], a baseline for single-view NVS. Fig. 12.

  * •

I: implementation details for ViewNeTI.

  * •

J: implementation details for evaluation on DTU.

##  Appendix A Evidence for 3D capabilities in diffusion models with image
outfilling

As discussed in Sec. 1, our work is motivated by the observation that 2D image
diffusion models seem capable of reasoning about 3D phenomena. In Fig. 9 we
ask a diffusion model to do outfilling around a real car. In the figure
caption, we discuss the evidence for 3D reasoning. The model is a Stable
Diffusion, [41] stabilityai/stable-diffusion-2-inpainting checkpoint, run for
50 denoising steps. The car and the mask are from Common Objects in 3D [38],
car id 106_12650_23736.

##  Appendix B Ablations

We do ablations for our design decisions for two scenes: one object-based
scene in Fig. 10, and one multiple-object scene in Fig. 11. We now elaborate
on the significance of some of the ablations. The $\alpha$ parameter is from
Eq. 3: a larger value gives more flexibility to predict the text encoding
directly, rather than just the word embedding. “No geometric augmentations”
means we remove the rotation and random cropping, but keep random color jitter
and random Gaussian blur. The view-mapper is pretrained and frozen (except for
the pretraining ablations), and so the architecture changes are applied to the
object-mapper only.

##  Appendix C Data augmentation

As in Sec. 4.2, we apply the image augmentations to help learn a robust object
token. The augmentations are similar to [32] with some changes: no
grayscaling, because that leads to some gray generations; higher probability
Gaussian blur; and no horizontal flips, since it would prevent learning the
view-mapper. The ablations in Appendix B demonstrate the necessity of these
augmentations. For 3-view NVS, we found that such strong augmentations were
not necessary, so we reduced the size parameter of the RandomResizedCrop to
(0.950, 1.05).

We also do text prompt augmentations. As described in Methods, the text
encoder input is the prompt “$S_{R_{i}}$. A photo of an $S_{o}$”, where
$S_{R_{i}}$ and $S_{o}$ are controlled by the view and object mappers
respectively. This is the same as regular textual inversion, but with the view
token [13]. Following that work, we use different text templates, for example
“$S_{R_{i}}$. a rendition of the $S_{o}$.” and “$S_{R_{i}}$. a photo of a
small $S_{o}$.” [13, 36]. We use the same templates as [1], which are also
available in our code.

##  Appendix D Visualizing the pretraining camera positions

As discussed in Sec. 4.3, we pretrain a view-mapper on a set of views that are
consistent across scenes. We visualize this distribution in Fig. 15 (the
figure is annotated with the sparse-view splits, but for pretraining we use
all the visible cameras). This visualization was generated by running the
introductory example for SDFStudio [76] at their github repo, which does
visualization with the interactive nerfstudio viewer [56]. The two images are
different views of the same 3D scene, but to get a better understanding of the
3D distribution, we recommend running the example SDFStudio code and
interacting with the viewer directly. The cameras are equidistant from an
origin that has the object (they live on a sphere), and they approximately
point at the origin. They cover an azimuthal range around the origin of about
160∘, with a polar range of about 75∘. Note that although the cameras may be
focused on one point, the object is not necessarily placed at that point, so
the object will be not centered in all images (for example, refer to the
ground truth images in Fig. 8).

##  Appendix E Visualizing the camera positions for DTU sparse dataset

In Sec. 5.1 and Fig. 4, we claimed that training on the 6-view split requires
generalizing to some views that are ‘interpolations’ and others that are
‘extrapolations’. To support this, we annotate the 6-view split that was used
in Fig. 15 (we explain how we generated this visualization in Appendix D, and
the explanation of training splits is in Appendix J). All other visualized
camera views are test views. Compared to the full set of images, the 6-view
split covers most of the range in the azimuthal angle, $\varphi$ around the
object, but only covers about half the range in the polar angle, $\theta$. It
is intuitive that the test views outside the polar angle range are a more
challenging generalization problem, and this is what we refer to as
‘extrapolation’. More concretely, the extrapolated views are those outside the
convex hull of the train views in the angle space $(\varphi,\theta)$.

##  Appendix F Pretraining results

In Sec. 5.2, we describe the model pretraining of dense views from multiple
scenes from the DTU training dataset [21]. We use caption of the form
“$S_{R_{i}}$. A photo of an $S_{*j}$”. The token for viewpoint, $S_{R_{i}}$,
is controlled by the view-mapper and is shared across all scenes. The token
for the $j$th object, $S_{*j}$, is controlled by an object mapper and is the
same for all views within a scene, but different across scenes. We verify that
the view-mapper has learned the training views by checking reconstructions
from sample scenes. In Fig. 16, we reconstruct every view for the training
scenes 9 and 33 after 100k steps of training

The captions above are constructed so that the object token captures all the
view-invariant object semantics; that is, we want the viewpoint and semantics
to be disentangled. One way to check this is to try generating images using
only the object token, e.g. “ A photo of an $S_{*j}$”. We do this in Fig. 17
for a sample of the object tokens at different points in training. This
suggests the object tokens approximately embed the object semantics, though
with varying degrees of success. Firstly, the generations are different and
far more diverse than the original. Secondly, they have a different ‘style’ to
the DTU images; especially for the toy houses, they become more realistic, and
arguably closer to the diffusion model training distribution. Third, for some
scenes, the disentanglement become worse with more training time; in the first
row, the generations have some similar attributes after 10k steps, but after
100k steps they are only a circular pattern.

In prior literature on textual inversion and personalization, evaluations are
done on ‘reconstruction’ ability or ‘fidelity’ to the original images [43, 13,
1]. This refers to the correctness of object semantics and details in the
generated images, and it tends to be traded off against ‘editability’, which
is the ability to compose the new concepts with existing concepts in the
vocabulary [1] by mixing the token with existing tokens in the prompt. How do
these ideas relate to our setting? The reconstructions in the NVS experiments
are good in most cases, and they are much more faithful to the images in the
‘disentanglement’ test. We propose two explanations. First, the view-mapper
could be embedding information not captured by the object token; we would
expect the view-mapper to capture attributes that are common across the
scenes, and this might include common backgrounds (the table in DTU images is
always white or brown, and there is a black background); or it could capture
common ‘style’ in the object semantics (e.g. there are many statues, many food
items, and many toy houses). Second, it could be that generating images with
only the object token in the caption does not properly test disentanglement,
but we do not have any further insights about this.

##  Appendix G Validation of spherical coordinate parameterization

Our method is agnostic to the camera parameterization. Our main results use
the camera-to-world projection matrix given in the DTU-MVS dataset [21], but
in Fig. 13, we show NVS results where the camera is parameterized by spherical
coordinates. We assume a central object is fixed at the origin, and that the
camera is at a fixed radius with variable polar and azimuth angles,
$(\theta,\varphi)$; we assume the camera is pointed at the origin. We do
single-scene optimization of ViewNeTI on a rendering of a ShapeNet car [5]. To
encourage the image to be close to the diffusion model training distribution,
we generated an augmented training set by outfilling the background around the
car, similar to in Appendix A. The left and right columns are reconstructions
of camera poses from the multiview train set. The middle columns are NVS
predictions from interpolating the polar and azimuth angles.

##  Appendix H Qualitative comparison to Zero-1-to-3

We add another qualitative basleine, Zero-1-to-3, [27], which treats novel
view synthesis as image-to-image translation with a diffusion model,
conditioned on the change in camera parameter. It is trained on renderings
from Objaverse, a large dataset of 3D assets available online [10]. By
training on a large dataset, Zero-1-to-3 is intended to ‘zero-shot’ generalize
to new scenes without any extra finetuning, similar to zero-shot
classification in CLIP [36]. This has the advantage that NVS predictions are
generated quickly - the time taken to generate a sample with Stable Diffusion
model. On the other hand, this poses a very difficult challenge to generalize
beyond the Objaverse data distribution. Unlike in CLIP, the training
distribution of Zero-1-to-3 does not yet cover the full distribution of test
scenes of interest for 3D perception applications [21, 33, 9], and this is
because enormous multiview datasets are harder to collect that 2D image
datasets.

In Fig. 12, we compare with Zero-1-to-3 using the same views as Fig. 5 (views
1, 45, and 22). We also show predictions for ‘no camera change’, which is the
train view reconstruction. We do the standard image preprocessing by masking
the central objects, centering them, and padding them to have aspect ratio 1.
Zero-1-to-3 does good reconstructions, and generates a good prediction for the
brick scene in view 45, which is the closest view to the input. The other
predictions are worse. This is likely due to the distribution shift in
objects:

  * •

DTU has real images with realistic details, while Objaverse (the Zero-1-to-3
training distribution) is mostly rendered objects from artist-generated 3D
models.

  * •

DTU images are also ‘cropped’: they do not contain the entire object. The
Zero-1-to-3 renders contain the entire object.

  * •

DTU images have some multi-object scenes, while Objaverse is mostly single
objects. Although the Zero-1-to-3 paper showed some generalization to multi-
object scenes, the object details were not good.

Finally, note that the failure modes of Zero-1-to-3 on DTU are distinct from
the NeRF-based models in Fig. 5, which all have imaging artifacts. Similar to
ViewNeTI, the predictions do not have such artifacts, probably because both
methods use diffusion models that are trained to generate images from a
certain distribution of real images.

##  Appendix I ViewNeTI implementation detais

The frozen diffusion model is Stable Diffusion [16], version
stabilityai/stable-diffusion-2-1 accessed from the diffusers library [61]. The
weights are all frozen. We did not test our method on Stable Diffusion1.

The inputs for camera parameters, timestep, and Unet layer are embedded to a
64-dim random Fourier feature vector. Specifically, for each of the 12 camera
params, one timestep, and one unet layer, we sample 64 random frequencies from
a standard normal, $\mathcal{N}(0,\sigma^{2})$ where $\sigma$ is 0.5, 0.03,
and 2 respectively. The encoding is computed as in [55], and as shown in our
code. For the object-mapper, the encoding is the same, but without the camera
parameters.

Following the base architecture of [1], the encoding is passed through an MLP
with two blocks. Each block has a linear layer, LayerNorm [3], and LeakyRelu
[70]. Finally, they are projected to 1536 dimensions, which is twice the word-
embedding for Stable Diffusion 2. This gives 140,000 parameters, which is the
same for the view-mappers and object-mappers. The first half of the output is
the word embedding and the second half is the textual bypass, as described in
Sec. 4.1.

The word embedding input is scaled to have the same norm as a particular
placeholder word, for example ‘statue’ for the buddha statue scene (again,
like in [1]). We did one experiment on varying this word on one scene, and
this showed that while norm scaling was important, the exact choice of word
for the reference token was not, so we just used ‘statue’ for every scene in
all experiments.

We use an effective batch size of 9 (batch size 3 with 3 gradient accumulation
steps), a constant learning rate of 0.09, with the AdamW optimizer [29]
(again, like in [1]). In training, DTU images were resized to (512, 384),
which has the same aspect ratio as the originals. At inference, we found that
image quality was better (fewer artifacts) if sampling at a higher resolution,
(768, 576). Since Stable Diffusion 2 was trained with square images at
(768,768), we experimented with padding DTU images to have the same aspect
ratio, but we found these results to be worse.

##  Appendix J DTU evaluation details

The DTU [21] splits are the same as [75]. The test set scenes are (8, 21, 30,
31, 34, 38, 40, 41, 45, 55, 63, 82, 103, 110, 114), which are the ones
visualized in Fig. 8 and Fig. 7, and used for quantitative results in Tab. 1.
For pretraining, the train scenes are every non-test scene except (1, 2, 7,
25, 26, 27, 29, 39, 51, 54, 56, 57, 58, 73, 83, 111, 112, 113,115, 116, 117).
They are excluded because they have too much overlap with the train set; e.g
115-117 have the same obejct as scene 114, but with a different pose. This
ensures that there is a domain shift between train and test with respect to
object semantics.

The splits for views also follow prior work [75, 34, 73, 11]. The index names
for views are 0-indexed, while the DTU filenames are 1-indexed. The standard
9-view splits (which we do not experiment with here) use train views (25, 22,
28, 40, 44, 48, 0, 8, 13). The 6-view, 3-view, and 1-view splits choose the
first 6, 3, and 1 views from that list respectively. The test views are all
indexes from 0-49 except a set that were excluded due to image quality which
are views (3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 36, 37, 38, 39). We use all
the train and test views for pretraining, and unlike in PixelNerf [75], we do
not include the excluded views for pretraining.

In figures Fig. 7 and Fig. 8 we evaluate novel views in (1, 8, 12, 15, 24, 27,
29, 33, 40, 43, 48). We chose them as a representative subset of the test set
views. In Fig. 5, we evaluate the novel views (1, 45, 22), and they were
chosen to match with evaluation in prior work [11].

Figure 7: ViewNeTI Novel view synthesis predictions for every DTU test set
scene from one input view (the rows are alternating ground truth and
prediction). Higher resolution images available at our project website. All
scenes have the same training hyperparameters and random seed for training and
generation. The hyperparameters are also the same as the three-view case in
Fig. 8, except training steps are reduced from 3k to 1.5k, and the image
augmentations are slightly changed as described in Appendix C. In almost all
cases, the rendered views are photorealistic, and the object semantics are
close to the ground truth, though semantics are worse for more complex scenes.
The failure modes are incorrect object details, and misaligned camera poses.
By contrast, NeRF-based methods will have consistent object semantics across
views due to the explicit 3D representation, but have much worse image quality
(see NeRF baseline comparison in Fig. 5 for these comparisons). Different to
the three-view case, another failure mode is for novel view predictions to be
too close to the input view (overfitting). We mitigate this by reducing the
training steps, to 1.5k. The views chosen for input are standard from previous
work, and the novel viewpoints are chosen to cover the full sequence of views
in DTU (see Appendix J).

Figure 8: ViewNeTI novel view synthesis predictions for every DTU test set
scene from three input views (the rows are alternating ground truth and
prediction). Higher resolution images available at our project website. All
scenes have the same training hyperparameters and random seed for training and
generation. In almost all cases, the rendered views are photorealistic, and
the object semantics are close to the ground truth, though semantics are worse
for more complex scenes. The failure modes are incorrect object details, and
misaligned camera poses. By contrast, NeRF-based methods have consistent
object semantics across views due to the explicit 3D representation, but much
worse image quality (see NeRF baseline comparison in Fig. 5 for these
comparisons). The views chosen for input are standard from previous work, and
the novel viewpoints are chosen to cover the full sequence of views in DTU
(see Appendix J).

Figure 9: Evidence that image diffusion models can reason in 3D. We take the
car on the top left, and the mask in the bottom left [38], and run a Stable
Diffusion [41] inpainting model. The model only sees the car object itself.
The shadows inferred for the ‘beach’ prompt are the same as the shadow in the
real image; the model has inferred the shadow placement from the lighting
pattern on the car. For the ‘shiny surface’ prompt, the model has reflected
the car object. We also see reflections for the right-side beach scene.

Figure 10: Ablation of design parameters for single-view NVS for DTU scan 114.
The test views are the same as Fig. 7 and Fig. 8, and are listed in Appendix
J.

Figure 11: Ablation of design parameters for single-view NVS for DTU scan 31.
The test views are the same as Fig. 7 and Fig. 8, and are listed in Appendix
J.

Figure 12: Baseline comparison of ViewNeTI with Zero-1-to-3 [27], a zero shot
novel view synthesis model. We use the same views as Fig. 5.

Figure 13: Examples of novel view synthesis using ViewNeTI where the input
camera parameters are in spherical coordinate system. We do single-scene NVS
using an input dataset of nine multiview images of a ShapeNet car [5] with
random forward-facing poses. The left and right columns are views in the
multiview train set. The middle columns are NVS predictions from interpolating
the polar and azimuth angles between the two edge images. The interpolation of
the car poses are smooth and the object semantic consistency is good..

⬇

transform = T.Compose([

T.RandomApply([T.RandomRotation(degrees=10, fill=1)], p=0.75),

T.RandomResizedCrop(image_size, scale=(0.70, 1.3)),

T.RandomApply([T.ColorJitter(0.04, 0.04, 0.04, 0.04)], p=0.75),

T.RandomApply([T.GaussianBlur(5, (0.1, 2))], p=0.20),

])

Figure 14: PyTorch code for the image augmentations in single-view novel view
synthesis. The 3-view NVS augmentations are slighty changed, as described in
Appendix C.

Figure 15: Visualization of the camera view positions in a sample DTU scene
[21]. The camera positions are the same for all scenes, and the left and right
images are the same 3D scene from different angles. Single-view training uses
view 25. Three-view training uses the same view 25, plus views 22 and 28
(yellow). Six-view training uses those three views, plus views 40, 44, and 48
(green). Visualization by running SDFStudio [76], and visualized with
nerfstudio [56].

Figure 16: Reconstructions of every training view in scenes 9 and 33 during
pretraining for 100k steps. The scenes have the same view-mapper, but
different object-mappers. The correct reconstructions validates the view-
mapper, which is shared across scenes, has learned the relative mapping
between the camera viewpoints.

Figure 17: Disentanglement results: for 10 random scenes in the pretraining
dataset, we generate images for the object-mapper without the view-mapper
using prompts “A photo of a $S_{o}$” for special token $S_{o}$. The left
column is view 25 from the original data. The other columns take two samples
at different points in training.

(minitoc(hints)) Package minitoc(hints) Warning: W0024Some hints have been
written in the main.log file